Crawl budget is the number of pages a search engine will fetch from your site in a given period, set by how much load your server can take and how much Google thinks your pages are worth revisiting. For sites under a few thousand pages it is almost never the limiting factor. It becomes a real constraint on large sites, on sites generating endless URL variations through filters and parameters, and on slow servers.
What it actually consists of
Two separate things get bundled under one name:
| Crawl capacity | How much Google can fetch without degrading your site. Driven by server response time and error rates |
|---|---|
| Crawl demand | How much Google wants to fetch. Driven by how popular and how fresh your pages appear to be |
A fast server with unimportant pages gets crawled lightly. A slow server with important pages gets crawled carefully. Both look like a budget problem and have different fixes.
How to tell whether it applies to you
- Your site has tens of thousands of URLs or more
- Faceted navigation or filters generate URL combinations without limit
- Search Console shows a large number of pages as discovered but not indexed
- New pages take weeks to appear even when linked from prominent places
- Your crawl stats report shows average response times measured in seconds
If none of those describe you, the thing you are troubleshooting has another cause. Look at internal linking and at whether the pages are worth indexing before you look here.
Where the budget gets wasted
On sites where it does matter, the waste is almost always the same handful of things.
- Parameter and filter URLs. One product listing with five filters can generate thousands of crawlable combinations, all nearly identical.
- Long redirect chains. Each hop is a fetch. A chain of four costs four times what it should.
- Soft 404s. Pages returning a 200 status with nothing on them get crawled repeatedly, because nothing tells Google to stop.
- Internal links to redirected URLs. Common after a migration, and it quietly doubles the cost of crawling your own navigation.
- Session IDs and tracking parameters in internal links. Every variant is a separate URL as far as a crawler is concerned.
How to check, in about ten minutes
Open the Crawl Stats report in Search Console. Three things are worth reading:
- Total crawl requests over time. A flat line while your site grows suggests a capacity ceiling.
- Average response time. Consistently above a second or so is worth raising with whoever runs the server.
- By response and by file type. A large share of requests going to redirects, errors or non-HTML files is budget going nowhere.
Crawl budget is one of the most discussed and least relevant topics in technical SEO. It matters enormously for a handful of large sites and not at all for most. Confirm which you are before spending a quarter on it.