How to fix “Blocked by robots.txt” on React SPA
If Search Console shows “Blocked by robots.txt” on your React SPA site, you are in the right place. robots.txt stops Googlebot from fetching the URL. This page covers the causes and a step-by-step fix for React SPA. A React app that runs only in the browser (Create React App, Vite). Content is drawn with JavaScript, so you depend on Google's rendering queue; prerendering or server-side rendering is recommended.
Contents
What does this status mean?
A Disallow rule in your robots.txt forbids crawling this URL. Google cannot read the content; if other sites link to it, it can still appear in the index without content.
Likely causes on React SPA
- A "Disallow: /" rule added during development was forgotten live.
- A broad pattern (for example /*?) also catches pages you want.
- The CMS or a plugin rewrote robots.txt on its own.
- Sitemap URLs sit in folders closed by robots.txt.
- React SPA note: Add public/robots.txt; without it many server setups return index.html in place of robots.txt. Do not block /static or JS files; Google needs them to render the page.
Step-by-step fix on React SPA
- In your React SPA admin go to: public/index.html, react-helmet or similar head management, public/robots.txt and server redirect settings.
- Add public/robots.txt; without it many server setups return index.html in place of robots.txt. Do not block /static or JS files; Google needs them to render the page.
- Use the Search Console robots.txt report or a robots tester to find which rule closes the URL.
- Narrow or delete the rule; close only folders that really should not be crawled.
- Verify that sitemap URLs do not conflict with robots.txt.
- After fixing, give Google a day to refresh its robots.txt cache or request a recrawl in the report.
- Send the URL for indexing and follow progress with URL inspection.
Extra checks for React SPA
- speed and crawling: A large JS bundle delays rendering. Use code splitting, lazy loading and caching, and prerender critical pages on the server where you can.
- noindex and visibility: A forgotten noindex tag in index.html affects every route. Code that adds or removes noindex on the client can behave inconsistently even after rendering; keep noindex static in the HTML.
After fixing
In Search Console start "Validate fix" for this status and resubmit the page with URL inspection. First confirm the blocker is gone with the free Free indexability checker tool; for many URLs follow Bulk URL indexing: how to send thousands of pages to Google. Google makes the indexing decision itself and timing ranges from days to weeks.
More information
Frequently asked questions
Does robots.txt remove a page from the index?
No. It blocks crawling, it does not remove from the index. To remove, allow crawling and use noindex.
How do I test robots.txt?
Enter the file and a URL in our free robots.txt tester to see the Googlebot allow status.
Where are these settings on React SPA?
public/index.html, react-helmet or similar head management, public/robots.txt and server redirect settings.