To block AI crawlers with robots.txt, add one Disallow: / block per bot's documented user-agent token — GPTBot for OpenAI training, ClaudeBot for Anthropic training, and Google-Extended for Google's AI training — while leaving Googlebot, Bingbot, and AI search bots like OAI-SearchBot alone if you want traffic. Not all bots do the same job, and blocking the wrong one costs you visibility instead of protecting your content. Here's the full picture and the exact templates to use.
Why bots deserve different treatment
For twenty years the web ran on a simple deal: search crawlers take your content, search engines send you traffic. AI crawlers broke that deal. Cloudflare's Radar data has shown crawl-to-referral ratios like 14:1 for Google but thousands-to-one for major AI training bots — they consume vastly more content than they ever send back as visitors.
But "block all AI bots" is a blunt answer to a nuanced question. The modern bot landscape splits into four tiers, and each one affects your site differently:
Tier 1: AI training crawlers — the ones that feed models
These bots collect web content that may be used to train large language models. They respect robots.txt, and blocking them is the documented way to opt out of training use:
- GPTBot (OpenAI) — crawls content that may be used to train OpenAI's foundation models. OpenAI's docs confirm that disallowing GPTBot signals your content should not be used for training.
- ClaudeBot (Anthropic) — collects public web content that may be used to train Anthropic's models. Blocking it excludes your future content from Anthropic's training datasets, per Anthropic's crawler documentation.
- CCBot (Common Crawl) — the nonprofit crawl corpus that many AI labs use as training input. Blocking it cuts off a major upstream source.
- Google-Extended (Google) — a standalone product token that controls whether Google uses your content for certain AI training purposes. It is not the same as Googlebot.
- Bytespider (ByteDance) — consistently one of the highest-volume AI training crawlers on the web.
If your content is your competitive advantage — original research, proprietary data, unique creative work — these are the bots to restrict.
Tier 2: AI search bots — the ones that cite you
These bots retrieve content to answer user queries with citations and links back to your site. Blocking them removes you from AI search answers — the opposite of what most publishers want:
- OAI-SearchBot (OpenAI) — powers ChatGPT's search results and citations. Blocking it keeps your site out of ChatGPT answers.
- Claude-SearchBot (Anthropic) — improves Claude's search results; blocking it reduces visibility in Claude-powered answers.
- PerplexityBot (Perplexity) — fetches content for Perplexity's cited answers.
These bots represent a new traffic channel. For most sites, the right move is to allow them.
Tier 3: Traditional search indexers — never block these
Googlebot, Bingbot, and DuckDuckBot drive the organic traffic your business depends on. They also feed Google's AI Overviews and AI Mode through the standard crawl — there is no separate "AI Overviews bot" to block. Blocking Googlebot to escape AI features also erases you from Google Search. Don't do it.
Tier 4: User-triggered fetches — robots.txt can't stop these
Some bots act on behalf of a specific user rather than crawling autonomously: ChatGPT-User (OpenAI), Claude-User (Anthropic), and Perplexity-User fetch a page when a person asks the assistant to read it. OpenAI and Perplexity both document that robots.txt rules may not apply to these fetches — like a browser visit, they're treated as user-initiated. Blocking these tokens in robots.txt is harmless but don't expect it to fully work; server-level controls are the only real enforcement here.
The complete robots.txt template
Below is a practical starting point. Adjust the training-bot section to match your own policy — allow what helps you, block what doesn't.
# AI crawler policy — review quarterly, bot names change
# Keep search engines fully allowed
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Block AI TRAINING crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
# Allow AI SEARCH bots — they cite sources and send traffic
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Global defaults
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /*?
Sitemap: https://yourdomain.com/sitemap.xmlOne important nuance: don't blindly paste giant blocklists from blog posts. Verify each token against the vendor's current documentation — names get renamed, and a stale token in your file is just noise. Generate a clean, current file with a robots.txt generator rather than hand-editing.
The Google-Extended trap
This deserves its own section because it causes the most damage. Adding User-agent: Google-Extended / Disallow: / does not opt you out of Google's AI Overviews or AI Mode. Google-Extended only controls use of your content for specific AI training purposes. AI Overviews are built from the regular Googlebot crawl, and Google's documented way to limit content flowing into AI features is snippet controls like nosnippet or max-snippet:0 — which also weaken your normal search appearance. There is no robots.txt line that keeps you in Google Search but out of AI Overviews.
Verifying your implementation
- Test the syntax. Google retired its own robots.txt tester in Search Console in 2023; use a third-party robots.txt tester instead. Enter your URL, then test with
GPTBot,ClaudeBot, andGooglebotas the user-agent. Your rules should block the first two and allow the third. - Check the file is live. Open
yourdomain.com/robots.txtin a browser. Remember it's per host — checkwwwand subdomains separately. - Watch your server logs. Compliant bots fetch robots.txt and honor new rules within about a day. If GPTBot keeps hammering your server a week later, check that the file returns HTTP 200 with
text/plainand isn't behind a firewall rule that blocks the bot from even reading it. - Recheck quarterly. AI companies add and rename bots regularly. A policy that was complete in January can have gaps by July.
A quick decision framework
Not sure where to start? Answer three questions. First: is my content easily replaceable commodity information, or is it original work with real cost behind it? Original research, proprietary data, and creative work deserve training blocks; commodity content may benefit more from AI search visibility. Second: where does my traffic come from today? If organic search pays the bills, protect Googlebot and Bingbot at all costs and be conservative with everything else. Third: am I seeing actual harm — server strain, scraped content outranking me, zero referrals — or just unease? Measurable harm justifies strict blocks; vague unease calls for monitoring your logs for a month before deciding. Write your answers down, build the file to match, and set a calendar reminder to revisit it.
When robots.txt isn't enough
Robots.txt is a voluntary standard — honest bots comply, dishonest scrapers ignore it. If your logs show bots claiming to be GPTBot from IPs that don't belong to OpenAI, or crawlers that never fetch your robots.txt at all, you need enforcement at the server or CDN level: IP verification against the vendor's published ranges, user-agent firewall rules, rate limiting, or a CDN bot-management service. Robots.txt is the first layer of defense, not the only one.
FAQ
Will blocking GPTBot remove my site from ChatGPT?
No. GPTBot only handles training crawls. ChatGPT's search answers come from OAI-SearchBot, a separate bot with a separate robots.txt token. Blocking GPTBot opts you out of training while leaving ChatGPT search visibility intact — as long as you leave OAI-SearchBot allowed.
Does Google-Extended block AI Overviews?
No. Google-Extended controls certain AI training uses of your content. AI Overviews are generated from the standard Googlebot crawl, and the only documented way to limit content in them is snippet controls like nosnippet, which also affect your regular search snippets.
Can robots.txt block ChatGPT-User or Perplexity-User?
Not reliably. These are user-triggered fetches — the assistant reads a page because a person asked it to — and both OpenAI and Perplexity document that robots.txt may not apply to them. Treat them like browser visits from users, not autonomous crawls.
How fast do bots respect new robots.txt rules?
Compliant bots typically pick up changes within about 24 hours, since they re-fetch robots.txt regularly (Google caches it for up to 24 hours). If a bot keeps crawling a week later, your file may have a syntax error, be unreachable, or the bot may not be compliant at all.
Should I block CCBot?
If your goal is to stay out of AI training data, yes — Common Crawl's corpus feeds many AI labs' datasets, so blocking CCBot cuts off a major upstream source. It has no effect on your search traffic.
Is there a legal benefit to blocking AI bots in robots.txt?
It creates a documented, timestamped record that you explicitly withheld consent for training use. Publisher lawsuits against AI companies have cited robots.txt bypassing, so a clear file strengthens your position — but robots.txt is a technical signal, not a legal contract, and it can't stop a determined scraper by itself.
The takeaway
Blocking AI crawlers works when you match the rule to the actual job each bot does: block the training crawlers if your content is your edge, allow the search bots that cite you with links, and never touch the traditional indexers that pay your bills. Write the rules precisely, verify them with real user-agents, and revisit them every few months — this landscape moves too fast for a set-and-forget file.
