What is Crawl budget?
Crawl budget is the number of pages Google is willing and able to crawl on a site in a given time. Server capacity and perceived page value determine it.
Why does it matter?
On very large sites, low-value URLs eat the budget and delay discovery of important pages.
Example
Example: a store with 200,000 products serves every filter combination as a separate URL. Googlebot spends its daily allowance on them and new products go undiscovered for weeks.
What should you do?
Close filter and parameter URLs, keep the sitemap clean and speed up the server.
Related indexing errors
- Crawled - currently not indexed
- Discovered - currently not indexed
- Soft 404
- Server error (5xx)
- Blocked due to unauthorized request (401) / access forbidden (403)
Related terms
Frequently asked questions
What is Crawl budget?
Crawl budget is the number of pages Google is willing and able to crawl on a site in a given time. Server capacity and perceived page value determine it.
Why does Crawl budget matter?
On very large sites, low-value URLs eat the budget and delay discovery of important pages.
What should I do about Crawl budget?
Close filter and parameter URLs, keep the sitemap clean and speed up the server.