If you have ever Googled "robots.txt for AI crawlers" and pasted a random snippet into your file, you are not alone — and there is a good chance the snippet you pasted does nothing. Most copy-paste templates floating around the web are outdated, block the wrong bots, or rely on robots.txt rules that never worked the way people assume. This guide gives you a working robots.txt template for AI crawlers in 2026, explains exactly what each line does, and shows you how to test it before you trust it.
Why you need a dedicated AI crawler section
robots.txt was designed in 1994 to talk to search engine crawlers. AI companies simply adopted the same file for their own bots, and that created a problem: the classic User-agent: * block now controls two completely different economies — search indexing (Googlebot, Bingbot) and AI training (GPTBot, ClaudeBot, CCBot) — with one set of rules.
If you block everything with one blunt rule, you can accidentally remove yourself from ChatGPT search answers while trying to opt out of model training. If you allow everything, you may be feeding paid content to training crawlers for free. A dedicated AI-crawler section lets you decide each case on purpose.
There is one mechanic you must understand first, because it is the source of most broken templates: a crawler obeys only the most specific group that names it. If your file contains a User-agent: GPTBot group, the User-agent: * group does not apply to GPTBot at all — even if the wildcard group is more permissive. Each bot reads the rules addressed to it and ignores everything else.
Meet the bots: what each user-agent token actually controls
Before writing a single line, know who you are talking to. Companies run separate bots for training and for search answers, and the names are confusingly similar.
- GPTBot — OpenAI's training crawler. Feeds content into future models. Blocking it opts you out of training; it does not remove you from ChatGPT search answers.
- OAI-SearchBot — OpenAI's search indexer. Powers citations in ChatGPT search. Block this and you disappear from ChatGPT answers.
- ChatGPT-User — Fetches one specific page when a ChatGPT user asks about it. Not a bulk crawler; OpenAI says robots.txt rules may not fully apply to it since a human triggered the request.
- ClaudeBot — Anthropic's training crawler. The old tokens
anthropic-aiandClaude-Webare deprecated — a file written in 2023 that only names those two leaves today's bots completely free. - Claude-SearchBot — Anthropic's search indexer for Claude's search features.
- Claude-User — Anthropic's user-triggered fetcher. Anthropic says it does follow robots.txt.
- Google-Extended — The opt-out token for Gemini and Vertex AI training. Google explicitly says blocking it has no effect on your Google Search inclusion or ranking.
- PerplexityBot — Perplexity's search index crawler. Blocking it removes you from Perplexity answers.
- Perplexity-User — User-triggered fetcher; Perplexity says robots.txt rules generally do not apply to it.
- Applebot-Extended — Opts you out of Apple Intelligence model training. Does not affect Siri, Spotlight, or Safari, which follow
Applebot. It does not crawl on its own. - CCBot — Common Crawl's crawler. Common Crawl publishes an open web corpus that many AI labs train on, so blocking CCBot cuts off dozens of downstream models at the source.
- Bytespider — ByteDance's training crawler. Multiple reports note it follows robots.txt inconsistently, so treat robots.txt as a statement of policy rather than a guarantee.
- Amazonbot — Amazon's crawler for Alexa and related services.
- Meta-ExternalAgent (case-insensitive:
meta-externalagent) — Meta's training crawler.
The pattern is consistent across every major lab: training bots and search bots are separate tokens. That separation is the entire reason templates must be deliberate.
Template 1: Block training, stay visible in AI search
This is the template most publishers, news sites, and original-content creators want. It opts out of model training while keeping every search-indexing bot allowed, so your pages can still be cited in ChatGPT, Claude, Perplexity, and Google AI Overviews.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
What this does and does not do:
- Stops OpenAI, Anthropic, Common Crawl, ByteDance, Apple, and Meta from using your content for model training.
- Leaves
OAI-SearchBot,Claude-SearchBot, andPerplexityBotalone, so your content can still appear — with attribution — in AI answers. - Leaves
GooglebotandBingbotalone, so classic SEO is untouched. BlockingGoogle-Extendeddoes not hurt Google rankings.
Template 2: Maximum privacy — block all AI crawlers
Some sites — intranets, membership content, personal blogs, portfolios — simply do not want AI systems reading anything. This template blocks every known AI token:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
Understand the trade-off before deploying this: blocking OAI-SearchBot, Claude-SearchBot, and PerplexityBot removes your site from AI search answers. For a marketing site that is self-sabotage; for a private site it is exactly the point.
Template 3: Selective blocking — keep search, block specific trainers
Maybe you are fine with OpenAI and Anthropic but not with ByteDance, or you want Common Crawl out because its corpus feeds models you cannot audit. The surgical template lets you pick bots individually. Example — block only Common Crawl and ByteDance:
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
Because of the most-specific-group rule, these two bots read only their own blocks. Every other crawler — including GPTBot and ClaudeBot — falls through to whatever rules you have for them (or for *) as if these lines did not exist. That is the key insight: you never need a giant file to make a small decision.
Template 4: Explicitly allow AI crawlers (the "come read me" file)
Most sites need no changes at all — a crawler treats an absent rule as allowed. But an explicit allow block is useful if you have a restrictive User-agent: * section elsewhere in your file and want AI bots exempted:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
Place these groups anywhere in the file. Because each bot reads only its own named group, position does not matter — contrary to the old folklore about rule ordering.
Common template mistakes that silently fail
Mistake 1: Blocking the training bot to hide from AI answers
This is the single most common configuration error on the web. A site blocks GPTBot and believes it has opted out of ChatGPT visibility — but ChatGPT search answers come from OAI-SearchBot, which is still happily crawling. Decide separately for each token or you will get the worst of both worlds: no training opt-out benefit you actually wanted, and surprise visibility outcomes.
Mistake 2: Using the deprecated Anthropic tokens
Files copied from 2023-era guides often contain only anthropic-ai and Claude-Web. Anthropic replaced those with ClaudeBot. Your old rules are decoration now.
Mistake 3: Matching the version number in the user agent
The full user-agent string looks like compatible; GPTBot/1.4; +https://openai.com/gptbot. Match on the product token (GPTBot), never the version. A rule keyed to GPTBot/1.4 stops matching the day OpenAI ships 1.5.
Mistake 4: Assuming Disallow equals a firewall
robots.txt is voluntary. Reputable AI crawlers honor it, but bad actors ignore it, and some trainers (Bytespider in particular) have spotty compliance records. robots.txt is a policy statement; if you need enforcement, add server-side user-agent checks that return 403, or a CDN rule.
Mistake 5: Forgetting the 24-hour lag
OpenAI and Perplexity both note it can take around 24 hours for their systems to pick up a robots.txt change. Test, then re-check tomorrow — do not assume instant effect.
Mistake 6: CDN firewalls overriding your file
Cloudflare's "block AI bots" toggle, and similar settings on other CDNs, can block search bots even when your robots.txt allows them. If an AI tool says it cannot read your site while your robots.txt looks fine, check your CDN dashboard next.
How to install and test your robots.txt
- Generate the rules. Use the free robots.txt generator to build a file with AI-bot presets — no signup, and it avoids the syntax errors that hand-typed files collect.
- Place the file at your domain root. It must be served at
https://yourdomain.com/robots.txt. A file at/static/robots.txtor a subfolder is never read by crawlers. - Serve it as plain text with HTTP 200. A 5xx server error on robots.txt makes cautious crawlers block everything, and a redirect to the homepage is treated as "no rules" by some bots and a mystery by others.
- Fetch it like a bot would:
curl -A "Mozilla/5.0; compatible; GPTBot/1.0; +https://openai.com/gptbot" https://yourdomain.com/robots.txt— confirm you get the file and the right group. - Validate the syntax with the robots.txt tester and your Google Search Console robots.txt report.
- Wait 24 hours, then re-check server logs for the crawler user agents to confirm the new rules took effect.
Pair it with a declaration file
robots.txt only says what bots may not do. If you want AI systems to understand what your site is — and you want citations, not just compliance — add an llms.txt file at your domain root. The combination covers both sides of AI discoverability: robots.txt controls access, llms.txt guides understanding. You can generate one in minutes with the free llms.txt generator.
FAQ
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is the training crawler. ChatGPT search answers use OAI-SearchBot. Blocking GPTBot alone keeps you citable in ChatGPT while opting out of model training.
Does blocking Google-Extended hurt my Google rankings?
No. Google states that the Google-Extended opt-out has no effect on inclusion or ranking in Google Search. It only controls Gemini and Vertex AI training use.
Will these templates stop AI companies from training on my site entirely?
They stop the well-behaved crawlers of the companies named. Training data also comes from Common Crawl dumps, data partners, and licensed datasets, so robots.txt is one layer of a larger strategy — not a complete shield.
How often should I review my AI-crawler rules?
New bots appear every few months. Check the official crawler documentation pages of the major labs once or twice a year — for example, OpenAI's bot documentation and the community-maintained AI crawler list — rather than trusting any single template forever, this one included.
Is Crawl-delay respected by AI bots?
Anthropic documents support for the non-standard Crawl-delay directive for ClaudeBot. Most other AI crawlers ignore it. If bot traffic volume is the problem, rate-limit at the server or CDN level instead.
Can I allow a bot on some pages but not others?
Yes. Within a bot's group you can mix rules, e.g. Disallow: /premium/ plus Allow: /blog/. The most specific path rule wins for that bot. Just keep each bot's rules inside its own named group.
