Robots.txt

Robots.txt: A file with specific instructions for different search engine spiders about which parts of your website should and shouldn’t be crawled.