How to fix “Blocked due to unauthorized request (401) / access forbidden (403)” on Magento
If Search Console shows “Blocked due to unauthorized request (401) / access forbidden (403)” on your Magento site, you are in the right place. Googlebot could not read the page because it asked for login or access was denied. This page covers the causes and a step-by-step fix for Magento. A powerful e-commerce platform (Adobe Commerce) for large catalogs. Layered navigation filters create many parametrised URLs.
Contents
What does this status mean?
The server answered Googlebot with 401 (login required) or 403 (forbidden). Googlebot does not log in, so the page cannot be indexed.
Likely causes on Magento
- The page is members-only or password-protected.
- A firewall, WAF or bot protection blocks Googlebot IPs.
- A country or IP restriction covers Google IPs.
- A hotlink or user-agent rule rejects Googlebot.
- Magento note: Edit robots.txt on the same screen. Disallowing filter parameters, search results and cart URLs protects crawl budget.
Step-by-step fix on Magento
- In your Magento admin go to: Stores > Configuration > Catalog > Catalog > Search Engine Optimization and Marketing > SEO & Search.
- Edit robots.txt on the same screen. Disallowing filter parameters, search results and cart URLs protects crawl budget.
- Enable full page cache and Varnish and turn on JS and CSS merge/minify. Slow catalog pages reduce crawl frequency.
- If the page should be public, remove the protection; if it should stay private this line is expected.
- Allow-list Googlebot as a verified crawler in the WAF and firewall.
- Allow Googlebot by verifying with reverse DNS rather than trusting the user agent.
- Confirm the response code with a live test in URL inspection.
- After fixing, start validation and send the URL.
Extra checks for Magento
- noindex and visibility: Under Stores > Configuration > General > Design > Search Engine Robots the default should be INDEX,FOLLOW. A NOINDEX,NOFOLLOW value left over from development is a common live mistake.
- canonical and duplicate URLs: In Catalog > SEO turn on "Use Canonical Link Meta Tag" for categories and products. Dropping the category path from product URLs reduces duplicates.
After fixing
In Search Console start "Validate fix" for this status and resubmit the page with URL inspection. First confirm the blocker is gone with the free Free indexability checker tool; for many URLs follow Bulk URL indexing: how to send thousands of pages to Google. Google makes the indexing decision itself and timing ranges from days to weeks.
More information
Frequently asked questions
Can Google see members-only content?
No. Previews need structured data and paywalled-content markup.
How do I tell if I am blocking Googlebot?
Look for 403 entries for Googlebot requests in your firewall logs.
Where are these settings on Magento?
Stores > Configuration > Catalog > Catalog > Search Engine Optimization and Marketing > SEO & Search.
Related pages
- Magento: URL is unknown to GoogleFixes
- Magento: Indexed, not submitted in sitemapFixes
- Magento: Crawled - currently not indexedFixes
- OpenCart: Blocked due to unauthorized request (401) / access forbidden (403)Fixes
- PrestaShop: Blocked due to unauthorized request (401) / access forbidden (403)Fixes
- Ticimax: Blocked due to unauthorized request (401) / access forbidden (403)Fixes