EN There are many things that the robots.txt can do to affect the site's crawlers, but preventing access is the most common. Other options include the crawl-delay option. You can use this option to make the bot delay between crawling the pages on the site.