Most sites do not have a crawl budget problem. Google's guide says it is aimed mainly at sites with 1 million+ unique pages that change about weekly, sites with 10,000+ unique pages that change daily, or sites with many URLs marked "Discovered - currently not indexed" in Search Console [1]. If you are below those sizes and new pages get crawled and indexed quickly, your time is better spent elsewhere.
If you are above them, or you have that "Discovered" symptom, the work is mostly about removing waste: faceted URLs, soft 404s and redirect chains.
What crawl budget is
Google defines crawl budget as the limits on time and resources it can devote to crawling a site. It is driven by two things: crawl capacity limit and crawl demand [1]. Capacity is how much your server can handle. Demand is how much Google wants to crawl.
Pages that are not crawled cannot be indexed. That is the only reason the topic matters.
Step 1: Decide whether it applies to you
Start with Google's own thresholds [1]:
| Your site | Crawl budget a real concern? |
|---|---|
| 1 million+ unique pages, content changes about weekly | Yes, per Google |
| 10,000+ unique pages, content changes daily | Yes, per Google |
| Many URLs in "Discovered - currently not indexed" | Worth investigating |
| Fewer pages, or new pages crawled the day they are published | Usually no |
For the last row, Google says keeping your sitemap current and checking the Page Indexing report is usually enough [1]. See the XML sitemap guide for that part.
Two caveats. First, "Discovered - currently not indexed" does not always mean a budget problem. Google's troubleshooting guidance lists several possible causes when important pages are not crawled in a reasonable time: Google not knowing about the pages, blocking, maximum serving capacity, or running out of crawl budget [3]. Rule out the first two before blaming the last.
Second, the thresholds are Google's guidance, not a hard cutoff. Treat them as a prior, then check your own data.
Step 2: Read the Crawl Stats report
Crawl Stats in Search Console shows Google's crawling history, including request counts, server response and availability issues. Google describes it as aimed at advanced users and says sites with fewer than 1,000 pages usually do not need it [2]. It is only available for root-level properties, such as a domain property or a root URL-prefix property [2]. If you only verified a subfolder, you will not see it.
What to look for:
- Availability problems. Repeated server errors or timeouts mean Google may be limited by your server. Fix these first.
- Slow responses. Slow server responses are a capacity problem, not an indexing-tactics problem.
- Request patterns. If most requests go to URLs you do not care about (parameters, filters, old redirects), that is the waste to remove.
- Trend changes. A sudden drop or spike after a deploy usually points to a specific change, such as a new filter component or a changed redirect rule.
The report tells you where Google spends requests. It does not tell you what Google intends. Match it against your server logs if you have them.
Step 3: Fix the waste that matters
Three causes account for most real-world crawl waste. Google's documentation names each.
Faceted and filter URLs
Filter URLs can multiply into huge numbers of near-duplicate pages. Google says they can waste crawl resources and recommends preventing crawling of faceted URLs when they do not need to be indexed, and returning 404 for no-result combinations [5].
A typical robots.txt pattern for filter parameters:
User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=
Test it against real URLs before shipping. A pattern that is too broad can block category pages you want indexed. Keep the clean category URL crawlable and linked. For paginated archives, see pagination SEO in 2026. To handle duplicates that must stay crawlable, see the canonical tags guide.
Soft 404s
A soft 404 is a page that returns 200 OK but is effectively an error page. Google says pages whose content is gone should return 404 or 410 [3]. Common sources are empty search results, out-of-stock templates with no content, and "not found" pages served with a 200.
Check a suspect URL:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/this-page-does-not-exist
If that prints 200, your server is producing soft 404s. The HTTP status codes guide covers what to return and when.
Redirect chains
Google says long redirect chains hurt crawling. In its site-move documentation it says Googlebot can follow up to 10 hops, but recommends redirecting directly to the final destination [4].
Trace a chain:
curl -sIL -o /dev/null -w "%{num_redirects} redirects, final: %{url_effective}\n" http://example.com/old-page
Anything above 1 is worth collapsing. Update internal links to point at the final URL, and change old redirect rules so each source goes straight to the destination. Chains usually build up from migrations stacked on top of each other (HTTP to HTTPS, then www, then a new URL structure).
What is cargo cult
Much crawl budget advice is more ritual than evidence.
- Auditing a small site for crawl budget. If you have a few hundred pages and they are indexed promptly, there is nothing to optimize [1][2].
- Using robots.txt to move budget around. Google says removing low-value URLs from crawling does not automatically shift budget elsewhere unless Google is already hitting serving limits. It says not to use robots.txt as a temporary reallocation trick [3]. Block URLs because they should not be crawled, not as a lever.
- Deleting content to "save" budget. Pruning can make sense for quality or maintenance reasons. Doing it on a site far below the thresholds, purely for crawl reasons, has no support in the documentation above.
- Resubmitting sitemaps repeatedly. An accurate, current sitemap is enough [1]. Resubmitting does not add capacity.
- Treating a lower crawl count as a problem. Fewer requests after you remove waste is the intended result. What matters is whether important pages are crawled promptly.
Documentation changes
Google's changelog records that the crawl-budget and related crawling docs were migrated, polished and clarified in late 2025 and July 2026 [6]. The core guidance cited here is consistent across those pages, but check the current version before relying on specific numbers.
What to do next
- Compare your site size and change rate with Google's thresholds [1].
- Check the Page Indexing report for "Discovered - currently not indexed" and note how many URLs are affected.
- If you qualify and have a root-level property, open Crawl Stats and look for availability errors, slow responses and request waste [2].
- Fix in this order: server errors and speed, soft 404s, redirect chains, then faceted URL crawling.
- Re-check Crawl Stats and the Page Indexing report after a few weeks. Change one thing at a time so you can tell what helped.