robots.txt generator
Write the rules and the file is created instantly. Copy it or download it as robots.txt.
How to use it
- Choose between allowing the whole site and blocking all of it.
- Write the paths you want to block (one per line) and, if needed, the paths to allow as exceptions.
- Add your sitemap address.
- Download the file and upload it to the root of your domain as
/robots.txt. - Once it is live, test it with the robots.txt tester.
Important warnings
- The line "Disallow: /" closes the whole site to crawling. Be sure before you go live: Blocked by robots.txt
- robots.txt manages crawling; it does not remove a page from the index. Use noindex for pages you do not want in the index.
- Blocking AI crawlers does not affect your place in Google Search. Google-Extended only controls use for generative AI such as Gemini; Googlebot is separate.
- Make sure sitemap addresses are not blocked by robots.txt: How to build a sitemap and submit it to Search Console
For the term itself, see robots.txt.
Frequently asked questions
Should I block AI bots?
It is a choice. If you block them your content is not used to train those companies' models, but your chance of appearing in those services' answers may also drop.
Where does the robots.txt file go?
In the root of the domain, so it opens at https://example.com/robots.txt. Files in subfolders are ignored.
Is my data sent to a server?
No. The file is created in your browser and downloaded directly.