Parse and analyze robots.txt rules to understand what can and cannot be scraped.
Powered by Scraper.bot