Crawl budget: does it matter for your website?

Crawl budget is the number of URLs a search engine is able and willing to crawl on your site in a given time. Google says it mainly matters for very large sites or sites that change very often; most sites with a few thousand pages don’t need to worry. If you do have a problem, it usually comes from wasted crawling: endless parameter URLs, redirect chains and server errors.

What crawl budget is made of

Google describes crawl budget as two things combined:

• Crawl capacity limit. How much Googlebot can crawl without overloading your server. If your site responds quickly and reliably, the limit goes up. If it slows down or returns errors, Googlebot backs off.
• Crawl demand. How much Google wants to crawl. Popular pages and pages that change often are crawled more; stale or low-value URLs less.

Your effective budget is what Google crawls within both limits. Bing works on similar principles, though the details differ.

Does it matter for you?

Google’s own guide on managing crawl budget is aimed at large sites, giving rough examples like sites with around a million or more unique pages that change weekly, or tens of thousands of pages that change daily. It also points to sites with many URLs stuck as “Discovered – currently not indexed”.

If you run a blog, a small shop or a company site with a few hundred or a few thousand pages, crawl budget is very unlikely to be why a page isn’t indexed. Look first at quality, duplication and technical blocks; the guide why isn’t my page on Google? walks through those.

Signs you might have a problem

• New or updated pages take weeks to be crawled, even though they’re linked and in your sitemap.
• Search Console’s Page indexing report shows a large and growing number of “Discovered – currently not indexed” URLs.
• The Crawl stats report (Settings → Crawl stats) shows Googlebot spending most of its requests on parameter URLs, redirects or errors.
• Your server logs show crawlers hitting thousands of near-identical URLs you didn’t know existed.

Common causes of wasted crawling

Faceted navigation and filters

Shops and listing sites often let visitors combine filters: colour, size, price, sort order. Each combination can create a new URL:

/shoes?colour=red
/shoes?colour=red&size=9
/shoes?colour=red&size=9&sort=price_asc
/shoes?size=9&colour=red

Multiply that across a catalogue and you get millions of URLs with almost the same content. Crawlers can spend their time here instead of on your real pages.

URL parameters

Session IDs, tracking tags and calendar pages that go on forever all create crawlable URLs. Keep tracking parameters out of internal links, and avoid generating links to infinite spaces such as “next month” on every calendar page.

Redirect chains

A link that goes A → B → C → D costs several requests to reach one page. Update internal links to point straight at the final URL, and keep redirects to a single hop where you can.

Server errors and slow responses

Frequent 5xx errors, timeouts or 429 (too many requests) responses tell crawlers to slow down. Google reduces its crawl rate when your server struggles. Fixing reliability often fixes crawling.

Soft 404s and thin duplicates

Empty search results pages, “no products found” pages that return 200 OK, and near-duplicate pages all take crawl time without adding anything.

How to fix it

1. Block crawl traps in robots.txt. For filter and sort combinations you never want in search, a Disallow rule stops crawling altogether. For example, Disallow: /*?sort=. Test carefully; see robots.txt mistakes that hurt SEO.
2. Don’t rely on noindex to save crawl budget. A crawler has to fetch a page to see its noindex tag, so noindex controls indexing, not crawling.
3. Use canonical tags for duplicates you need to keep, so search engines consolidate them. They still get crawled, but less often over time. See canonical tags.
4. Return real status codes. Gone pages should return 404 or 410, not 200 with an error message.
5. Keep your sitemap clean. List only canonical, indexable URLs that return 200, with accurate lastmod dates. See the XML sitemap guide.
6. Link to important pages internally, so crawlers find them quickly and know they matter.
7. Speed up the server. Faster responses let crawlers fetch more without hurting your visitors.

Note that Google ignores the crawl-delay rule in robots.txt. Bing does support it.

AI crawlers add load too

Search engines aren’t the only crawlers now. AI bots such as GPTBot, ClaudeBot and PerplexityBot also fetch pages, and on some sites they make a noticeable share of requests. They don’t use Google’s crawl budget, but they do use your server’s capacity, and a slow, overloaded server can make Googlebot back off.

Before you block anything, find out who is actually visiting and whether the visits are real; plenty of “AI bots” are fakes. Our guide on whether to block AI crawlers weighs the trade-offs, and the free bot checker verifies a single visit in seconds.

In short

• Crawl budget mostly matters for very large or fast-changing sites.
• Small sites with indexing problems should look at quality and technical blocks first.
• Check Search Console’s Crawl stats and Page indexing reports for signs of waste.
• Tame faceted navigation and parameters, shorten redirect chains, fix server errors.
• Remember noindex doesn’t stop crawling; robots.txt does.
• Keep sitemaps clean and servers fast.
• Watch AI crawler load as part of overall server health.