Claude is answering more questions every month, and every answer that cites a source sends readers to someone's website. If you want those readers to land on your pages, you need to understand one thing first: Anthropic does not run one crawler. It runs three, and they do three different jobs. Get the split right, and you can stay out of AI training data while still showing up in Claude's answers.
Here is the short version. ClaudeBot collects content that may be used for training. Claude-SearchBot builds the search index that improves Claude's answers. Claude-User fetches pages on the spot when a person asks Claude something. Block ClaudeBot to opt out of training. Leave Claude-SearchBot and Claude-User alone if you want to be cited. One careless wildcard block removes you from all three at once.
Anthropic's three bots, and what each one does
Anthropic documents three agents on its official crawling page, and the distinction between them is the whole story of this guide:
- ClaudeBot collects web content that could contribute to training future models. Anthropic says blocking it signals that your site's future material should be excluded from training datasets.
- Claude-SearchBot navigates the web to improve search result quality. Blocking it prevents Anthropic from indexing your content for search, which Anthropic warns may reduce your visibility in user search results.
- Claude-User fetches pages when an individual asks Claude a question that needs fresh web information. Blocking it prevents Claude from retrieving your content in response to a user query.
All three respect robots.txt, which is a meaningful difference from some competitors. Anthropic also supports the non-standard Crawl-delay directive if you want to slow a bot down rather than block it, and it explicitly discourages blocking by IP address: blocking Anthropic's IPs can stop its bots from reading your robots.txt at all, which breaks the very opt-out you are trying to set. Anthropic does not publish fixed IP ranges, because its bots run on cloud-provider addresses that change.
The robots.txt setup for “no training, yes citations”
If your goal is to be cited by Claude without contributing to training, your robots.txt file is short:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Put this file at the root of your domain, and repeat it on every subdomain you want covered. Anthropic is explicit on this point: a robots.txt file on example.com does not cover blog.example.com. Each hostname is a separate file, and each one needs its own rules.
Two mistakes undo this setup regularly. The first is a wildcard block written before the AI era, such as User-agent: * with Disallow: / on a staging file that was copied to production. Wildcard groups apply to any bot without its own named group, so a blanket block catches Claude-SearchBot and Claude-User too. The second is the opposite: blocking ClaudeBot and assuming the job is done, while a Cloudflare rule or a firewall challenge quietly blocks Claude-User's live fetches. Robots.txt is a request, not a wall, and it only works if the fetch itself can reach your pages.
Use CrawlGuards' free robots.txt generator to build the file with the right bot presets, and test it before publishing so a typo does not silently block the bots you want to keep.
Being eligible is not the same as being cited
Allowing the right bots only buys you eligibility. Claude still has to choose your page, and it chooses the way a careful researcher would: it looks for pages that answer the question directly, show their work, and look trustworthy enough to quote.
No setting, file, or tool guarantees a citation. Anyone promising one is selling something. What you can do is remove every technical obstacle and give Claude the kind of page it can quote with confidence. That breaks down into two halves: access and content.
Access: make your pages fetchable
- Serve real HTML. If your key content only appears after JavaScript runs, a fetch bot may see an empty shell. Server-rendered content, or at least the article body in the initial HTML, is far safer.
- Do not put bot challenges in front of answer bots. A CAPTCHA or aggressive challenge page on your article URLs stops Claude-User from reading the page at all. If you run bot management, make sure the search and user-fetch agents are not challenged. Anthropic states its bots will not try to bypass CAPTCHAs, so a challenge is a dead end, not a delay.
- Keep pages fast and stable. A page that times out or returns errors on a fetch will not be cited, and repeated failures teach a crawler to stop trying. Check your server logs for Claude-User requests and their response codes.
- Do not hide content behind logins. Anything Claude-User cannot open without an account cannot be quoted. Keep your public guides fully public.
- Keep your sitemap current. A sitemap does not force a citation, but it helps crawlers discover new and updated pages quickly. Pair it with a clean robots.txt and sensible internal links.
Content: write pages Claude can quote
Claude cites pages that make quoting easy. That is a writing discipline more than a technical trick:
- Answer the question in the first paragraph. A direct, specific answer in the first 100 words gives the model a quotable sentence. Vague introductions do not get quoted.
- State facts with sources. When you cite a number, name the source and link to it. Pages that show their evidence get cited more readily than pages that assert things. This is the same habit that earns human trust, and it works for the same reason.
- Use structure a machine can follow. Clear headings, short paragraphs, tables for comparisons, and lists for steps all help a model extract the right passage. A 3,000-word wall of text with no headings is hard to quote accurately.
- Be specific, not generic. “Many sites block AI crawlers” is not quotable. “Anthropic documents three crawlers, and blocking ClaudeBot alone leaves search and user fetches working” is quotable, because it is precise and checkable.
- Show dates and updates. Freshness matters for fast-moving topics. A visible publish date and an update note tell both readers and models that the page reflects current rules, not last year's.
- Publish original evidence. Your own tests, logs, screenshots, and measurements are the content no one else can offer. Aggregated summaries compete with a thousand similar pages; original data does not.
- Put a real name on the page. A named author with a short bio, plus a contact page and an about page, signals that a real person stands behind the claims. Anonymous, authorless content is harder to trust and harder to cite.
These points also describe good SEO in general, which is the point. There is no separate “AI SEO” trick here. The pages that get cited are the pages that would already rank: clear, sourced, specific, and maintained.
How to check whether it is working
- Test your robots.txt. Paste your file into a robots.txt tester and check ClaudeBot, Claude-SearchBot, and Claude-User individually. Confirm the first is blocked and the other two are allowed.
- Read your server logs. Look for requests with Claude-User and Claude-SearchBot in the user-agent string, and check the response codes. A run of 403s or challenge pages on Claude-User means your firewall, not your robots.txt, is the blocker.
- Ask Claude directly. Ask a question your page answers well, and see whether your site appears in the citations. Try a few phrasings; one miss proves nothing, but a pattern across questions tells you something.
- Watch your referrals. If Claude links to you, visits typically arrive with a claude.ai referrer in your analytics. A trickle of such visits is the earliest sign your pages are being cited.
- Re-check after every site change. A new firewall rule, a CDN migration, or a robots.txt edit can silently change what Claude can fetch. Re-test the three bots after any infrastructure change.
If you previously blocked all AI crawlers and want to reverse course, the same steps apply in reverse: remove the Claude-SearchBot and Claude-User blocks, keep ClaudeBot blocked if training is still your concern, and give the crawlers time to revisit. Indexing and citation are not instant; they follow the next crawl, not the next save.
When blocking everything is still the right call
Not every site should chase citations. If your content is your product, if licensing is your business model, or if you simply do not want your work feeding any AI system, blocking all three bots is a legitimate choice. In that case, block each bot by name rather than relying on a wildcard, so your intent is explicit and survives a future edit to the wildcard group. Pair the robots.txt block with server-side controls for scrapers that ignore robots.txt, and understand the trade-off honestly: you are choosing control over discovery, and that is a valid business decision.
For most informational sites, though, the split approach is the better deal: training off, citations on. It is the one configuration that respects both your rights as a publisher and your interest in being found. For the blocking side of the same coin, see our guide to blocking ClaudeBot with robots.txt, and for the parallel playbook on the other major answer engine, our guide to getting cited by ChatGPT. Anthropic's own documentation is the primary source for the bot definitions used here: Does Anthropic crawl data from the web, and how can site owners block the crawler?
FAQ
Does blocking ClaudeBot remove me from Claude's answers?
No. ClaudeBot is the training crawler. Search indexing and live fetching are handled by Claude-SearchBot and Claude-User. Blocking only ClaudeBot leaves both of those working, so your pages can still be cited.
Do I need to allow Claude-SearchBot and Claude-User explicitly?
Not if your robots.txt does not block them. An explicit Allow group is useful when your default rule is restrictive, or when you want the intent to be obvious to anyone reading the file. If your wildcard group already allows everything, the bots are already allowed.
Why does my robots.txt need to be on every subdomain?
Robots.txt is per-host, not per-domain. A file on example.com says nothing about blog.example.com. Anthropic's documentation states this explicitly, and it is the most common reason a block “doesn't work” on part of a site.
Will Claude cite my page just because I allow its bots?
No. Allowing the bots makes citation possible, not guaranteed. Claude still chooses sources based on relevance, clarity, and trustworthiness. The content section above is what turns eligibility into actual citations.
Does Crawl-delay help with Claude's bots?
Anthropic supports Crawl-delay as a non-standard extension, so you can slow a bot's request rate without blocking it. It is a good middle step if ClaudeBot's crawl volume bothers you but you are not ready to block it outright.
Should I block Claude's bots by IP address instead?
Anthropic advises against it. Its bots run on changing cloud-provider addresses, and blocking those IPs can prevent them from reading your robots.txt, which breaks your opt-out. Robots.txt is the supported mechanism.
