Glossary

What is robots.txt?

robots.txt is a text file in the site root telling crawlers which paths they may crawl. It manages crawling; it does not remove a page from the index.

Updated: 05.11.2026 · 1 min read · Türkçe

Contents
  1. Why does it matter?
  2. Example
  3. What should you do?
  4. Related indexing errors
  5. Guides that use this term
  6. Related terms

Why does it matter?

One wrong Disallow line can close the whole site to crawling.

Example

Example: the lines "User-agent: *" and "Disallow: /cart/" ask that the cart folder not be crawled while product pages stay open.

What should you do?

After a change, confirm with a robots.txt tester that important URLs are open.

Guides that use this term

Frequently asked questions

What is robots.txt?

robots.txt is a text file in the site root telling crawlers which paths they may crawl. It manages crawling; it does not remove a page from the index.

Why does robots.txt matter?

One wrong Disallow line can close the whole site to crawling.

What should I do about robots.txt?

After a change, confirm with a robots.txt tester that important URLs are open.

Related pages