What is robots.txt?
robots.txt is a text file in the site root telling crawlers which paths they may crawl. It manages crawling; it does not remove a page from the index.
Contents
Why does it matter?
One wrong Disallow line can close the whole site to crawling.
Example
Example: the lines "User-agent: *" and "Disallow: /cart/" ask that the cart folder not be crawled while product pages stay open.
What should you do?
After a change, confirm with a robots.txt tester that important URLs are open.
Related indexing errors
- Blocked by robots.txt
- Indexed, though blocked by robots.txt
- Blocked due to unauthorized request (401) / access forbidden (403)
Guides that use this term
- How Google indexing works: crawling, rendering and indexing
- How to build a sitemap and submit it to Search Console
- Google indexing checklist: 15 items before you publish
Related terms
Frequently asked questions
What is robots.txt?
robots.txt is a text file in the site root telling crawlers which paths they may crawl. It manages crawling; it does not remove a page from the index.
Why does robots.txt matter?
One wrong Disallow line can close the whole site to crawling.
What should I do about robots.txt?
After a change, confirm with a robots.txt tester that important URLs are open.