Crawl-delay is a robots.txt line that asks a crawler to wait a set number of seconds between requests — Crawl-delay: 10 means "pause ten seconds between page fetches." It exists to protect small or fragile servers from aggressive crawlers. But here's the part most guides bury: it is not part of the official robots.txt standard (RFC 9309), Googlebot ignores it completely, Yandex stopped honoring it in 2018, and only a handful of crawlers — Bing's being the most important — actually obey it. A 2026 census found it written into 16.5% of sampled robots.txt files, a directive more written than read. This guide explains exactly who honors it, who ignores it, and when it's still worth using.
What Crawl-delay is (and isn't)
Robots.txt's official vocabulary is small: User-agent, Allow, Disallow, and Sitemap. That's what the IETF standardized in RFC 9309 (2022). Crawl-delay is a nonstandard extension that grew up in the wild: a polite request — "please don't hit my server faster than this" — that each crawler operator is free to honor, reinterpret, or ignore.
It only ever applies inside a User-agent group, and the number means seconds between successive requests from that crawler. It is not a rate limit in the strict sense (requests per minute); it's a spacing rule. Set it too aggressively and you don't just slow a crawler — you can effectively stop it from ever finishing your site.
Our robots.txt generator includes Crawl-delay as an optional setting, and our robots.txt directives guide covers the standard vocabulary. This article is the deep dive on the one directive whose support is a patchwork.
Who honors it, who ignores it (verified)
Googlebot: ignores it entirely
Google's documentation lists the robots.txt fields it supports, and Crawl-delay is not among them. Googlebot determines its own crawl rate adaptively, based on your server's response times and capacity — if your server answers quickly, Google crawls faster; if it slows down, Google backs off automatically. There is no robots.txt way to slow Google's crawl. (An old Search Console crawl-rate control existed for years but was retired — don't go looking for it.) If Googlebot is overloading your server, the fix is server-side: faster responses, or rate limiting at your CDN or host.
Bingbot: honors it
Bing is the big one that respects Crawl-delay, and it interprets the number as time windows: Crawl-delay: 10 creates ten-second windows in which Bingbot fetches at most one page. Do the math before you set it: a 10-second delay caps Bing at roughly 8,640 pages per day. On a 500-page blog that's irrelevant; on a million-URL marketplace it's a throttle that could take months to work through your catalog. Bing also offers its own crawl-rate controls in Bing Webmaster Tools, which override the robots.txt setting — if you use both, the Webmaster Tools setting wins.
Yandex: stopped honoring it in 2018
Yandex documented Crawl-delay support for years, then dropped it: since 2018 it directs site owners to its own crawl-rate controls instead. Old guides still list Yandex as a supporter — they're stale. If you see a "Yandex honors Crawl-delay" claim dated before 2018, it's a fossil.
Anthropic's bots: honor it
Anthropic's crawler documentation says its bots honor Crawl-delay. That makes it one of the few AI-training operators with a documented politeness mechanism — relevant if ClaudeBot's crawl volume ever becomes a load concern on your server. (Blocking ClaudeBot outright is the more common choice; see our ClaudeBot blocking guide.)
Everyone else: assume nothing
For the long tail of crawlers — SEO tools, scrapers, minor AI bots — support is undocumented and inconsistent. The Barkhausen AI census from 2026 pairs the documentation picture with the empirical one: Crawl-delay appears in 228 of 1,381 parsed robots.txt files (16.5%), spread across every sector measured. The authors' dry observation is the takeaway: a directive's presence in robots.txt is a separate fact from whether any crawler acts on it, and the two don't have to track each other. Writing it is not the same as it working.
When Crawl-delay is actually worth using
Given the patchwork, the directive has exactly two legitimate uses:
1. You're on a tiny server and Bing is hammering it. Shared hosting, a Raspberry Pi, a legacy box — if Bingbot's crawl is visibly degrading response times, a modest Crawl-delay: 5 or 10 in a Bingbot group is a reasonable, documented lever. Check Bing Webmaster Tools first; its crawl controls are more precise.
2. You want to be polite to compliant AI crawlers without blocking them. If you'd rather throttle ClaudeBot than ban it, the documented support gives you a middle path. This is niche — most publishers either allow or block AI bots outright — but the option exists.
That's the whole list. If your problem is Googlebot's crawl rate, Crawl-delay does nothing. If your problem is a scraper ignoring robots.txt, Crawl-delay does nothing. If your file has Crawl-delay "just in case" with no specific crawler in mind, delete it — it's dead weight that makes your file harder to read.
How to write it correctly
Three rules, each violated constantly in the wild:
- Put it inside a User-agent group, never floating. A Crawl-delay line outside any group is meaningless. It must sit with the bot it applies to:
Note: this delays Bing only. A second group for another bot needs its own line.User-agent: Bingbot Crawl-delay: 10 - Don't set it under
User-agent: *unless you mean every crawler. A wildcard Crawl-delay asks all compliant bots to slow down, including ones that would otherwise crawl you efficiently. Scope it to the bot causing the problem. - Don't set absurd values.
Crawl-delay: 120doesn't mean "be gentle" — it means a crawler can fetch 720 pages a day, max. On a large site, that's a soft block on every bot that honors it. Values of 1–10 are the sane range; anything higher should be a deliberate, temporary choice.
After adding it, run the file through our robots.txt tester — misplaced directives are exactly the kind of silent error the tester catches — and our pre-publish checklist covers the rest.
Crawl-delay vs crawl budget: don't confuse them
Two different concepts that get tangled. Crawl budget is how many pages a search engine is willing to crawl on your site — a function of your authority, server speed, and content freshness. Crawl-delay is a requested pause between requests. Slowing a crawler doesn't increase your budget; it spends the same budget more slowly. If your real problem is important pages not getting crawled, the answer is budget management — sitemaps, internal linking, pruning thin pages — not delay. Our crawl budget guide explains how Google decides what to crawl and how to win more of it.
A real-world example: the small forum that actually needed it
Consider a hobby forum on a $6/month shared host — 4,000 threads, a single MySQL database, and Bingbot crawling it at full speed every night because a decade of deep thread URLs look "fresh" to the crawler. Page loads spike to six seconds, real users bounce, and the host sends a resource-usage warning. Googlebot isn't the problem (it self-throttles against the slow responses). The fix that actually lands: a User-agent: Bingbot group with Crawl-delay: 5, set in Bing Webmaster Tools as the override if the host's panel supports it. Load normalizes within days, Bing keeps indexing the forum — just more slowly — and nothing else on the site changes. That's the entire legitimate use case in one story: one identified crawler, one overloaded server, one scoped directive. If your situation doesn't look like this story, you probably don't need Crawl-delay.
What to do instead (by crawler)
- Googlebot too aggressive: improve server response times (Google adapts automatically); use CDN caching; fix redirect chains and thin pages that waste its budget. There is no robots.txt lever.
- Bingbot too aggressive: Crawl-delay in a Bingbot group, or the crawl controls in Bing Webmaster Tools (which take precedence).
- AI training bots too aggressive: decide allow vs block first — most publishers choose
Disallow: /per training token (GPTBot, ClaudeBot, Google-Extended, CCBot) rather than throttling. Our AI crawler robots.txt templates have the copy-paste rules. - Unknown scrapers hammering you: robots.txt won't help — they ignore the file. You need CDN bot management or server-side rate limiting; see our scraping protection guide.
FAQ
Does Google support Crawl-delay in robots.txt?
No. Google's documentation lists only user-agent, allow, disallow, and sitemap as supported fields. Googlebot ignores Crawl-delay completely and sets its own crawl rate based on your server's responsiveness. Any guide telling you to slow Googlebot with Crawl-delay is wrong.
What does Crawl-delay: 10 actually mean?
It asks the crawler to wait 10 seconds between successive requests. Bing interprets it as ten-second windows with at most one fetch per window — about 8,640 pages per day maximum. Yandex used to treat it as a minimum gap between requests but stopped honoring it in 2018.
Should I add Crawl-delay to my robots.txt?
Only if a specific compliant crawler is overloading your server and you've identified which one. For most sites the answer is no — it's a nonstandard directive that Google ignores, most AI bots don't document, and that adds noise to your file. Fix server performance or use the crawler's own webmaster controls first.
Can Crawl-delay hurt my SEO?
Indirectly, yes. An aggressive value (say, 60+ seconds) on a bot that honors it can prevent large sections of your site from being crawled in any reasonable timeframe — Bing-indexed pages would go stale. And a wildcard Crawl-delay slows every compliant crawler at once. Keep values small (1–10) and scoped to one bot.
Is Crawl-delay part of the official robots.txt standard?
No. RFC 9309 (2022), the IETF standard for the Robots Exclusion Protocol, does not include it. It's a long-standing informal convention — which is exactly why support is inconsistent and why you should never rely on it for anything critical.
What's the difference between Crawl-delay and a CDN rate limit?
Crawl-delay is a polite request inside robots.txt that only compliant crawlers may honor. A CDN or server rate limit is enforced: it actually refuses or slows excess requests regardless of who's making them. If a crawler ignores your Crawl-delay, the rate limit is the tool that works. They solve different halves of the problem — politeness vs enforcement.
