To block ClaudeBot, add two lines to your robots.txt file: User-agent: ClaudeBot followed by Disallow: /. That's it. Anthropic says its crawlers honor robots.txt, so once ClaudeBot next reads your file, it will stop collecting your site's pages for AI training. But one thing most guides get wrong: blocking ClaudeBot does not block Anthropic's other two crawlers, Claude-User and Claude-SearchBot. Each needs its own rule.
This guide explains exactly what each of Anthropic's three crawlers does, helps you decide which ones to block, and gives you copy-paste robots.txt snippets for every scenario — plus how to verify it worked and what to do if it doesn't.
Meet Anthropic's three crawlers
Anthropic runs three separate web crawlers, and mixing them up is the most common mistake publishers make. They have different jobs, different user-agent tokens, and blocking one does nothing to the others.
- ClaudeBot — collects public web content that may be used to train and improve Anthropic's AI models. If you block it, Anthropic says your site's future content will be excluded from AI training datasets.
- Claude-User — fetches a page when a Claude user asks a question that requires visiting that page. This is user-triggered, on-demand fetching, not bulk collection.
- Claude-SearchBot — crawls content to improve the quality and relevance of Claude's search results. Anthropic says blocking it means your content won't be indexed for its search optimization, which may reduce your visibility and accuracy in Claude-powered search answers.
All three are documented in Anthropic's official crawler documentation, and all three honor robots.txt directives.
What blocking each crawler actually does
Blocking ClaudeBot: opting out of training
This is the rule most publishers want. When ClaudeBot reads a Disallow: / rule under its own user-agent, it stops collecting your pages. Anthropic's documentation states that blocking ClaudeBot keeps your site's future content out of AI training datasets. This is the opt-out the whole conversation is usually about.
The trade-off is minimal: ClaudeBot does not drive search rankings or referral traffic. It is a training-data collector. Blocking it changes nothing about how your site appears in Google, Bing, or even in Claude's answers — those depend on the other two crawlers.
Blocking Claude-User: opting out of live answers
Claude-User only visits your site when a real person asks Claude something that requires your page. Blocking it means Claude can't pull up your content to answer user questions, which Anthropic says may reduce your visibility in user-directed search responses. If you want your content cited and discoverable inside Claude conversations, you should generally leave this one alone.
Blocking Claude-SearchBot: opting out of Claude search
Claude-SearchBot builds the index behind Claude's search features. Blocking it may reduce your visibility and accuracy in Claude-powered search results. Again, leave it running unless you have a specific reason to vanish from Claude's search entirely.
Decide before you block: a three-question framework
Don't blindly block everything. Walk through these three questions first:
- Do you object to your content being used for AI training? If yes, block ClaudeBot. This is a clean, targeted opt-out with almost no visibility cost.
- Do you want your pages to show up in Claude's answers? If yes, keep Claude-User and Claude-SearchBot allowed. These crawlers are how Claude finds and references your site. Blocking them shrinks your presence in AI search — the opposite of what most publishers want.
- Is the problem volume, not use? If AI crawlers are hammering your server and driving up hosting costs, you don't need to block anything. Slow them down with a crawl delay instead. Anthropic supports the non-standard
Crawl-delaydirective for exactly this purpose.
Copy-paste robots.txt rules for every scenario
Every rule below goes in the robots.txt file at the root of your domain — for example, https://example.com/robots.txt. You must add these rules to each subdomain you want to restrict; a rule on example.com does not automatically cover blog.example.com.
Scenario 1: Block ClaudeBot only (recommended for most sites)
This is the classic "block ClaudeBot" setup: opt out of training, keep your Claude visibility intact.
User-agent: ClaudeBot
Disallow: /Scenario 2: Block all three Anthropic crawlers
If you want Anthropic's bots off your site entirely — no training, no live fetching, no search indexing — you must write a separate block for each token. One block does not cover the others.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /Scenario 3: Slow ClaudeBot down instead of blocking it
If the crawler is eating bandwidth but you don't object to the crawling itself, add a delay in seconds between requests. A value of 1 or 2 is a sensible starting point.
User-agent: ClaudeBot
Crawl-delay: 1Note: Crawl-delay is a non-standard extension, not part of the formal robots.txt standard. Anthropic supports it, but other crawlers may ignore it.
Scenario 4: Allow ClaudeBot for maximum visibility
Some publishers actively want their content in AI training corpora and search indexes, reasoning that discoverability inside AI assistants is the next SEO. If that's you, you don't need any Anthropic-specific rules — or you can state the permission explicitly for clarity:
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /Writing this out explicitly also protects you against a catch-all Disallow later in the file accidentally locking them out.
Where to put your robots.txt file
Place the file at the very root of each domain and subdomain you control: https://example.com/robots.txt. Crawlers look nowhere else. A robots.txt buried in a subdirectory, or served from a different domain, will be ignored.
After editing, fetch the URL in your browser to confirm the change is live, then check it parses correctly with a tester — CrawlGuard's free robots.txt tester shows you exactly how a bot reads your rules. If building the file from scratch, the robots.txt generator can produce a valid file in seconds.
Mistakes that make your block fail
Using dead tokens: Claude-Web and anthropic-ai
Old guides and copy-pasted robots.txt templates still include rules aimed at Claude-Web and anthropic-ai. These tokens are deprecated — they no longer correspond to anything Anthropic operates. A rule aimed at them blocks nothing. Use only the three current tokens: ClaudeBot, Claude-User, and Claude-SearchBot.
Blocking by IP address
Anthropic explicitly warns that IP-based blocking is not reliable. It doesn't publish crawler IP ranges, and blocking the IPs its bots come from can interfere with the bot's ability to read your robots.txt file in the first place — which breaks the very opt-out mechanism you're relying on. Robots.txt is the documented, supported control mechanism. Use it.
Treating robots.txt like a firewall
Robots.txt became a formal internet standard as RFC 9309 in 2022, but it has always been a voluntary protocol — a polite request, not an enforced barrier. Reputable operators like Anthropic honor it, and ClaudeBot is documented as doing so. But robots.txt does nothing to stop malicious scrapers, bots that ignore the standard, or anyone who fetches your pages with a plain HTTP client. Never rely on it as security for sensitive content; if something truly must stay private, put it behind authentication.
Assuming one rule blocks everything
Worth repeating: User-agent: ClaudeBot / Disallow: / stops ClaudeBot and only ClaudeBot. Claude-User and Claude-SearchBot keep working. If your goal is a total Anthropic opt-out, you need all three blocks.
Expecting instant results
The block only takes effect when the bot next fetches your robots.txt and reads the new rule. Rules apply going forward. Also, a robots.txt block does not erase content that was already collected — it stops future collection.
How to verify ClaudeBot actually stopped
Don't just publish the file and hope. Verify it in two ways:
- Check the file is reachable. Open
https://yourdomain.com/robots.txtin a browser. If you see your new rules, the file is live. Run it through the robots.txt tester to confirm the syntax parses the way you intend. - Watch your server logs. Search your access logs for the
ClaudeBotuser-agent string. You're looking for the request pattern to change: after your edit, ClaudeBot fetches should stop for disallowed paths. Match on the literal stringClaudeBotin your logs — that's the token Anthropic documents.
For a broader strategy on handling AI crawlers beyond just Anthropic, see our guide on how to block AI crawlers with robots.txt, and if you're also dealing with OpenAI's bots, read the GPTBot vs OAI-SearchBot breakdown — OpenAI uses a similar three-way split.
What if ClaudeBot ignores your robots.txt?
Anthropic documents that its bots respect robots.txt, so persistent crawling after a correct, live rule is a malfunction, not the design. Anthropic provides a contact address for exactly this situation: claudebot@anthropic.com. Two important details:
- Write from an email address associated with your domain — for example, something@yourdomain.com. Anthropic requests this so it can verify you actually control the site.
- Include evidence: the domain, the robots.txt rules you set, when you set them, and log excerpts showing ClaudeBot user-agent requests arriving after the change.
Before writing, double-check the basics: is the rule in the root robots.txt of the exact host being crawled (including subdomains)? Are the token spellings exactly ClaudeBot? Has the bot had time to re-fetch the file? Most "it's ignoring me" cases turn out to be a file-location or timing issue.
Quick recap
- Block training only:
User-agent: ClaudeBot+Disallow: / - Block all Anthropic bots: repeat the block for
Claude-UserandClaude-SearchBot - Too much crawling traffic: use
Crawl-delayinstead ofDisallow - Want Claude visibility: leave the bots allowed — or state it explicitly
- One rule per bot, one file per subdomain — nothing is automatic
FAQ
Will blocking ClaudeBot remove my content from Claude's training data?
Blocking ClaudeBot stops future collection — Anthropic says your site's future content will be excluded from training datasets once the bot honors your rule. What happens to content collected before the block is a separate question; robots.txt is a forward-looking control, not a deletion tool. If you need a definitive answer about already-collected data, that's a question for Anthropic directly.
Will blocking ClaudeBot hurt my Google rankings?
No. ClaudeBot has nothing to do with Googlebot or Google Search. Blocking it changes nothing about how Google crawls, indexes, or ranks your site. The only visibility effects are within Claude's own products, and those come from the other two crawlers — which keep working unless you block them too.
How long until the block takes effect?
The rule applies once ClaudeBot re-fetches your robots.txt and reads the change. There is no fixed schedule published for this, so give it time, then confirm by checking your server logs for a drop in ClaudeBot requests to disallowed paths.
Can I block ClaudeBot from only part of my site?
Yes. Replace Disallow: / with the path you want protected — for example, Disallow: /premium/ keeps ClaudeBot out of that directory while leaving the rest of the site open. Rules match path prefixes, so anything under that path is covered.
What's the difference between ClaudeBot and Claude-User?
ClaudeBot collects web content at scale and that content may be used for AI model training. Claude-User fetches a specific page because a Claude user asked a question that needs it. One is bulk collection for training; the other is on-demand fetching for a live conversation. They are controlled by separate robots.txt tokens.
Should I just block everything to be safe?
Probably not. Blanket-blocking all three Anthropic crawlers also removes you from Claude's search results and prevents Claude from citing your pages in answers — real visibility costs, for zero extra training protection beyond what blocking ClaudeBot alone achieves. The recommended default is: block ClaudeBot, keep Claude-User and Claude-SearchBot allowed.
