Crawl budget is the number of URLs Google can and wants to crawl on your site within a given timeframe. Google determines it from two factors: crawl capacity limit, which is how much crawling your server can handle without slowing down, and crawl demand, which is how much Google's systems want to crawl your site based on its popularity, freshness, and quality. If your important pages are getting crawled and indexed promptly, your crawl budget is fine. If new pages sit undiscovered for weeks while Googlebot burns requests on faceted filters and duplicate URLs, your budget is being wasted — and this guide shows how to fix it.
Crawl budget is one of the most misunderstood concepts in SEO. It is real, Google documents it, and it genuinely matters — but only for a specific kind of site. Most blogs and small business websites will never need to think about it. The sites that do need it are large, fast-changing, or drowning in low-value URLs. The first skill in crawl budget work is knowing which group you are in. The second is spending your effort on the fixes that actually move the needle, instead of the myths that sound technical and accomplish nothing.
What crawl budget actually is
Google's own crawl budget documentation defines crawl budget as the set of URLs Googlebot can crawl and wants to crawl. Those two halves matter equally, and they respond to different fixes.
Crawl capacity limit is the technical half. It is the maximum rate at which Googlebot will fetch pages from your site without overloading your server. Google raises it when your server responds quickly and reliably, and lowers it when your server slows down, times out, or throws errors. A site on a struggling shared host with two-second response times will get crawled more cautiously than the same site on a fast server. This half is entirely within your control: server speed, stability, and error rates.
Crawl demand is the motivation half. It is how much Google's systems want to crawl your site. Demand rises with popularity, freshness, uniqueness, and overall user value. A news site publishing fresh stories every hour generates high demand. A static archive that has not changed in three years generates little. A site full of thin, duplicate, or near-duplicate pages generates low demand per URL, because Google has learned that crawling it rarely surfaces anything worth indexing. This half is earned, not configured.
Your effective crawl budget is the combination of the two. A fast server with low-demand content gets light crawling. A slow server with high-demand content gets throttled by its own infrastructure. A fast server with high-demand content gets crawled aggressively. Most crawl budget problems are a mismatch: the site has the capacity but wastes it on URLs nobody needs, or it has the demand but chokes it with a slow server.
One more fact that surprises people: Google treats each unique hostname as a separate site for crawl budget purposes. Your main domain and your subdomains get separate budgets. A problem on blog.yoursite.com does not drain the budget of www.yoursite.com, and fixing one does not help the other.
Who actually needs to worry about crawl budget
Google is explicit about this: if your site does not have a large number of pages that change rapidly, or if your pages are crawled the same day they are published, you do not need to read its crawl budget guide. The guidance applies mainly to three kinds of sites:
- Very large sites: more than 1 million unique pages, with content that changes about once a week. Think major e-commerce catalogs, large marketplaces, and huge publishers.
- Medium or larger sites that change daily: more than 10,000 unique pages with content that changes every day. Think job boards, real-estate listings, classifieds, and fast-moving catalogs.
- Sites with a large backlog of "Discovered — currently not indexed" URLs in Search Console. This is the clearest symptom: Google knows your URLs exist but has not gotten around to crawling them.
Google describes those numbers as rough estimates, not exact thresholds. The practical test is simpler than any number: publish a new page, then check how long it takes Google to crawl it. If it is crawled within a day or two, your crawl budget is fine, and your indexing problems — if you have any — come from something else, such as content quality or weak internal linking. If new pages take weeks to be crawled while your site has thousands of pages, crawl budget work is worth your time.
Small sites should read this as background knowledge, not as a to-do list. A 200-page blog does not have a crawl budget problem. It has a content, links, or technical-health problem wearing a crawl budget costume. Fixing the real problem beats optimizing a budget that was never the constraint.
How to tell if crawl budget is your problem
Diagnosis starts in Google Search Console. Open Settings, then Crawl Stats. This report shows your total crawl requests per day over the last 90 days, the breakdown by file type and response code, and whether crawling is trending up, down, or flat. You are looking for three warning signs.
Warning sign 1: High crawl volume, low indexing. Googlebot is visiting constantly, but your Page Indexing report shows a growing pile of "Discovered — currently not indexed" URLs. That combination means the crawler is busy but not productive. It is spending requests on URLs that do not get indexed, while the URLs that matter wait in line.
Warning sign 2: Errors and slow responses. A rising share of 5xx errors, timeouts, or slow response times in Crawl Stats tells you the capacity half is throttling you. Googlebot backs off when your server struggles. It does not send a warning; it just crawls less.
Warning sign 3: The wrong URLs are being crawled. Check which URL patterns dominate your crawl. If faceted filter combinations, internal search results, calendar archives, tag pages, and parameter variations account for a large share of crawls, your budget is leaking into URLs that will never rank. Your server logs — or your CDN analytics — show this more precisely than Search Console, because they record every request, including requests from other crawlers.
Compare crawl volume to your sitemap size. If your sitemap lists 50,000 URLs and Googlebot crawls roughly 1,000 per day, a full pass takes 50 days — and that assumes every crawl hits a useful URL, which it never does. That gap between sitemap size and daily crawl volume is the shape of a crawl budget problem.
Fix 1: Speed up the server
Server speed is the highest-leverage fix on the capacity side, and it is also the fix that helps everything else. Google has said plainly: if Google can load and render your pages faster, it may be able to read more content from your site.
Start with response time. Measure your server's time to first byte for a typical page. If it is consistently above 600 milliseconds, you have room to improve. Common wins include upgrading inadequate hosting, adding server-level caching, putting a CDN in front of your site, and removing plugins or middleware that add database queries to every request. WordPress sites with dozens of active plugins are a classic case: every plugin adds work to every page load, including loads by Googlebot.
Support HTTP caching properly. If a page has not changed since Google last crawled it, returning a 304 (Not Modified) response tells Google to reuse its cached version. That saves your server bandwidth and Googlebot's time, and it is one of Google's own recommendations for crawl efficiency. Check that your server or CDN sends correct caching headers instead of forcing a full re-download of unchanged pages.
Fix availability problems before anything else. A site that throws 500 errors under load teaches Googlebot to crawl it cautiously. If your server cannot stay up under its current crawl rate, no sitemap trick will compensate. Stabilize first, optimize second.
Fix 2: Cut the low-value URLs that eat your budget
Most crawl budget waste comes from URLs that should never have been crawlable in the first place. Audit your site for these patterns:
- Faceted and filtered URLs: e-commerce combinations of color, size, price, and sort order can generate millions of near-identical pages. Let Google crawl the main category pages, and block the combinatorial explosion in robots.txt or with proper canonical tags.
- Internal search results: your site's own search pages (
/?s=...on WordPress, for example) are infinite, thin, and useless in an index. Block them in robots.txt. - Calendar and date archives: day-by-day and month-by-month archives on blogs create hundreds of thin listing pages. Disable or noindex the ones nobody uses.
- Tag and author pages with no value: a tag with one post, or an author archive that duplicates the blog index, is crawl work with no payoff. Prune them.
- Parameter variations: tracking parameters, session IDs, and print views create duplicate copies of the same page. Canonicalize them to one version, and use robots.txt to block the parameter variants that canonical tags cannot clean up.
Be careful with the tool you choose for each pattern. A robots.txt Disallow stops crawling of those URLs, which frees budget — but it does not remove them from the index, and a blocked URL can still be indexed if other pages link to it. A noindex tag removes a page from the index but still costs a crawl to read it. A canonical tag consolidates signals but still costs crawls. For pure budget waste — URLs you never want crawled at all — robots.txt is the right tool. For URLs you want crawled once and then excluded from the index, use noindex. Build clean rules with a robots.txt generator and test them with a robots.txt tester before deploying.
Fix 3: Fix errors and redirect chains
Every crawl request that returns a 404, a 500, or a hop through a redirect chain is a request that accomplished nothing. A few are normal on any site. A pattern is a budget leak.
Redirect chains are the most common offender. A URL that redirects to another URL that redirects to another URL costs Googlebot three requests to reach one page. Google explicitly recommends avoiding long redirect chains. Audit your redirects and flatten every chain to a single hop: old URL straight to final URL, with no intermediate stops. This is especially common after migrations, where legacy redirects stack on top of newer ones.
404 errors at scale usually mean something is generating dead links: a feed, a sitemap that includes deleted pages, or internal links to retired URLs. Fix the source, not just the symptom. Update your sitemap, fix the internal links, and redirect only the dead URLs that actually have inbound links or traffic. Redirecting thousands of dead URLs one by one is busywork; removing the links that point to them is the cure.
5xx errors are the most urgent of all, because they reduce your crawl capacity directly. A server that errors under crawl load gets crawled less. Treat any spike in 5xx responses in Crawl Stats as an infrastructure incident, not an SEO task.
Fix 4: Sitemaps that tell the truth
Your sitemap is how you tell Google which pages matter and how recently they changed. A dishonest sitemap wastes budget; an honest one focuses it.
Include only indexable URLs. Every URL in your sitemap should return a 200 status, be allowed in robots.txt, have no noindex tag, and be canonical. A sitemap full of blocked, redirected, or duplicate URLs trains Google to distrust your sitemap, which is the opposite of what you want.
Keep lastmod accurate. The lastmod field tells Google when a page last changed. Google uses it to prioritize recrawls — but only when it is accurate. A sitemap where every URL claims to have changed today, every day, is noise, and Google learns to ignore it. Set lastmod to the real modification date, and update it only when the page actually changes. If your CMS stamps every URL with the same date on every publish, fix that; a fake lastmod is worse than none.
Split by priority. On a large site, do not dump every URL into one sitemap. Split it: one sitemap for your most important pages — current products, active listings, recent content — and separate sitemaps for the rest. Google can then prioritize the high-value sitemap. If you run a large site and want to generate clean sitemaps, use a sitemap generator and submit the sitemap index through Search Console.
Watch the "Discovered" backlog. If a large share of your sitemap URLs sit in "Discovered — currently not indexed" for weeks, your sitemap is bigger than your budget. The fix is not to demand more budget; it is to shrink the sitemap to the pages that deserve crawling, and to fix the waste described in Fix 2.
Fix 5: Internal linking that points Googlebot at what matters
Googlebot discovers and prioritizes pages partly through links. A page with no internal links — an orphan — may be in your sitemap and still get crawled rarely, because nothing signals that it matters. A page linked from your homepage, your main category pages, and your best articles gets crawled often, because the link structure says it is important.
Audit for orphans. A crawler tool or a simple comparison of your sitemap against your internal link graph will surface pages that nothing links to. Link to them from relevant hub pages, or accept that they are not important and remove them from the sitemap. Both are better than leaving them orphaned.
Use hub pages deliberately. A category page or a well-organized resource page that links to your best content in one place is both a user convenience and a crawl-efficiency device: one crawl of the hub exposes Googlebot to dozens of important URLs. Make sure your mobile version carries the same links as your desktop version — Google indexes the mobile version, and if the mobile page shows fewer links, discovery of new pages slows down.
Prune aggressively. Every internal link to a low-value page is a small vote for crawling that page. Sites accumulate these links over years: footer links to retired campaigns, sidebar links to dead series, navigation entries for sections nobody maintains. Removing them frees budget for the pages that earn their keep.
Fix 6: Earn more demand (the slow, honest fix)
Capacity fixes work in days. Demand fixes work in months, and there is no shortcut. Google's demand signals are popularity, freshness, uniqueness, and overall user value — which is another way of saying that crawl demand is a consequence of being a site worth crawling.
Publish fresh, original content on a schedule. Sites that update regularly get recrawled regularly. This does not mean churning out thin posts; it means maintaining a living site. Updating your best pages with genuinely new information counts too, and it is often more efficient than publishing new ones.
Remove or consolidate thin and duplicate content. A site where a large share of pages are thin, duplicated, or near-duplicated teaches Google that crawling it is low-yield. Merging duplicates, improving thin pages, and deleting what cannot be saved raises the average value of every crawl.
Earn links and visits. Popularity is a demand signal. Pages that people link to and visit get crawled more. That is not a trick to game; it is the system working as designed. The sites with the largest crawl budgets are the sites people actually use.
Be skeptical of anyone selling a crawl-budget shortcut. There is no setting, plugin, or service that increases Google's demand for your site. Demand is earned by the site itself.
Myths that waste your time
"A bigger crawl budget means higher rankings." No. Crawl budget affects whether pages get crawled and indexed. Indexing is a prerequisite for ranking, but crawling a page more often does not make it rank higher. A page that is crawled daily and a page that is crawled weekly compete on content, links, and relevance — not on crawl frequency.
"Crawl-delay in robots.txt controls Googlebot." Googlebot does not support the crawl-delay directive. Adding it to robots.txt does nothing for Google. If Google is crawling your site too aggressively — a rare problem — the real control is in Search Console's crawl-rate setting, and Google says to use it only in unusual cases.
"Blocking a page in robots.txt removes it from the index." It does not. Robots.txt stops crawling, and a page that is never crawled can still remain indexed if other pages link to it, sometimes with an auto-generated snippet. To remove a page from the index, use noindex (and let Google crawl it to see the tag), a 404 or 410 status, or Search Console's removal tool for urgent cases.
"I should submit every URL to force crawling." Submitting a bloated sitemap does not force Google to crawl everything in it. It just hands Google a longer list to prioritize. A focused sitemap of pages that deserve crawling beats an exhaustive sitemap every time.
FAQ
What is crawl budget in simple terms?
Crawl budget is the number of URLs Google can and wants to crawl on your site within a given timeframe. It comes from two factors: crawl capacity limit (what your server can handle) and crawl demand (how much Google wants to crawl your site). Both must be healthy for your important pages to get crawled promptly.
Does my small blog need to worry about crawl budget?
Almost certainly not. Google says crawl budget guidance applies mainly to sites with more than 1 million pages changing weekly, sites with more than 10,000 pages changing daily, or sites with a large "Discovered — currently not indexed" backlog. If your pages are crawled within a day or two of publishing, your budget is fine.
How do I check my crawl budget in Search Console?
Go to Settings, then Crawl Stats. It shows daily crawl requests, response codes, and file-type breakdowns over 90 days. Pair it with the Page Indexing report: a growing "Discovered — currently not indexed" count alongside high crawl volume is the clearest sign of wasted budget.
Does crawl budget directly affect rankings?
No. Crawl budget affects crawling and indexing, and indexing is required before a page can rank. But crawling a page more often does not raise its ranking. Content quality, links, and relevance decide rankings once a page is indexed.
Can I increase my crawl budget?
You cannot set it directly. You can raise your capacity limit by speeding up your server, fixing errors, and supporting caching. You can raise demand by publishing fresh, original content, removing thin and duplicate pages, and earning links and visits. Both halves take work, and demand takes months.
Should I block low-value pages in robots.txt to save crawl budget?
Yes, for URLs you never want crawled — faceted filter combinations, internal search results, parameter variants. But remember that robots.txt blocking does not remove a page from the index. Use noindex for pages you want crawled once and excluded from results, and canonical tags to consolidate duplicates.
