Crawler directives

Crawler directives: These are rules to legitimate bots on what content to read on your site and what not to read. This can help to keep from exceeding your crawl budget thereby having your site fully indexed faster.