Beta Position Eleven is live and still being built.
Home › Writing › SEO

Crawl budget probably is not your problem

Crawl budget is real. Google allocates finite crawling to each site, based roughly on how fast the site responds and how much it seems worth crawling. Exceed it and new pages take longer to be discovered.

It is also the most over-applied diagnosis in technical SEO.

The threshold

Google has been fairly direct: sites under a few thousand URLs generally do not need to think about crawl budget. A site with 400 pages is not being held back by crawl allocation. If those 400 pages are not indexed, the reason is something else — quality, duplication, or an instruction telling Google not to.

If you have 400 pages and a crawl budget theory, the theory is wrong.

When it genuinely bites

Faceted navigation. A shop with 12 filters, each combinable, generates a combinatorial explosion of URLs that are all technically distinct and nearly all worthless. This is the classic case and it is genuinely serious — millions of URLs, each crawled, none worth having.

Calendars and paginated archives. Any URL pattern that generates infinite valid pages. A date-based archive with no lower bound will be crawled into the last century.

Session identifiers in URLs. Every visitor generating a fresh crawlable URL for the same content.

Slow responses. Crawl rate adapts to server speed. A site that takes 3 seconds per response gets crawled less, and this one is often the real story behind a "crawl budget" complaint.

What to look at first

The crawl stats report in Search Console tells you what Google actually spent its time on. The useful question is not "how many pages did it crawl" but "how many of the pages it crawled were worth crawling".

If the answer is that Google spent 60% of its requests on filtered URLs that should never have existed, you have a crawl budget problem and also an information architecture problem, and fixing the second fixes the first.

If the answer is that Google crawled your 400 real pages regularly and indexed 120 of them, you have a quality or duplication problem, and no amount of robots.txt tuning will change it.

The fix, when it applies

Do not block with robots.txt first. Blocked URLs still accumulate as known-but-not-crawled, and Google cannot see any canonical or noindex directive on a page it is not allowed to fetch.

Prefer, in order: do not generate the URL; make it non-linkable; canonical it to the version worth having; and only then consider disallowing patterns you are confident are worthless.

Read next

Reading about it is slower than looking

Run the free audit on your own site. Public data, about ten seconds, no account.

Audit my site

Stop reporting what happened.

Start telling clients what to change — and prove it moved.

Get started
Beta · help shape it

What would make this worth paying for?

Everything on this site works today. It is early, so the roadmap is still genuinely movable — tell us what is missing and we will build it. Testers who give us something useful get a lifetime deal on whatever plan they end up on.

No account needed. Email is optional.