Blocking every AI bot with one line in robots.txt feels decisive, but it is usually the wrong move. The most common configuration error site owners make is blocking all of a company's crawlers at once — which quietly removes the site from that company's AI search answers while the owner believes they only opted out of training. The smarter approach is a selective allowlist: block the training bots that consume your content and give nothing back, while allowing the answer-engine bots that cite you and send traffic. This guide shows you exactly how to build one.
Why "Block All AI Bots" Backfires
AI crawlers are not one thing. They fall into two families with opposite consequences for your site:
- Training bots collect content to build and improve AI models. You get no traffic, no citation, no link — your work becomes model weights. Examples: GPTBot, CCBot, ClaudeBot, Google-Extended, Bytespider, Meta's meta-externalagent.
- Answer-engine bots index content to cite it in AI answers, with links back to your site. Blocking them makes you invisible in exactly the places your competitors are getting cited. Examples: OAI-SearchBot (ChatGPT search), PerplexityBot, Meta's meta-webindexer, Applebot.
The critical fact most guides bury: these are controlled independently. OpenAI's documentation is explicit — GPTBot handles training, OAI-SearchBot builds the index behind ChatGPT's search answers, and they are separate user agents with separate robots.txt tokens. Blocking GPTBot does not affect your ChatGPT search visibility at all. But blocking OAI-SearchBot means your pages stop appearing as sources in ChatGPT answers. One line too many in robots.txt, and you have traded an invisible training opt-out for a very visible disappearance.
The Allowlist Decision Framework
For every AI bot hitting your site, ask three questions:
1. Does it train on my content? If yes and you object to that, it goes on the block list. GPTBot, CCBot (whose dataset feeds countless downstream models), ClaudeBot, Google-Extended, and Bytespider are the usual candidates. Note that CCBot deserves special attention: blocking it cuts off an entire downstream supply chain of models trained on Common Crawl data, not just one company's.
2. Does it send me traffic or citations? If yes, think hard before blocking. Perplexity's answer engine cites sources with links back — many publishers keep PerplexityBot purely for the referral traffic. OAI-SearchBot powers cited answers in ChatGPT. Meta's meta-webindexer gets your pages cited in Meta AI answers. Applebot feeds Siri and Apple Intelligence features. These are distribution channels, not just costs.
3. Does it respect robots.txt? Reputable bots do; scrapers and aggressive harvesters don't. Your allowlist handles the polite ones. The rude ones need server-level blocking, which is a different fight — our guide to blocking bad bots covers that side.
The default posture that falls out of this framework: block the trainers, allow the answer engines. Deviate from it deliberately, not accidentally. If you haven't worked through the tradeoffs yet, our pros-and-cons guide to blocking AI crawlers walks through the full decision.
The Selective Template: Block Trainers, Keep Answer Engines
Here is a robots.txt template that implements the allowlist. The training bots get Disallow: / blocks by name; the answer-engine bots are simply left allowed (robots.txt allows everything not explicitly disallowed, so allowing is the default — you only need rules for what you block).
# AI crawler allowlist: block training bots, keep answer engines
# Training crawlers - blocked
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
# Answer-engine crawlers - deliberately allowed (no rules = allowed)
# OAI-SearchBot: powers cited answers in ChatGPT search
# PerplexityBot: crawls for Perplexity's cited answer engine
# ChatGPT-User: fetches pages users explicitly share (user-initiated)
# meta-webindexer: powers citations in Meta AI answers
# Applebot: feeds Siri and Apple IntelligenceA few notes on what is in there and why:
- Google-Extended is blocked, Googlebot is untouched. Google has said publicly that blocking the Google-Extended token does not affect Google Search rankings. Never confuse the two — blocking Googlebot itself would be catastrophic.
- ChatGPT-User is allowed. It only fetches pages a person explicitly shared with ChatGPT, and OpenAI notes robots.txt rules may not apply to it anyway since the fetches are user-initiated. Blocking it gains you almost nothing.
- Bytespider is blocked on bandwidth grounds alone for many sites — it is known for aggressive crawl rates — even by owners who are relaxed about training.
- Each bot gets its own group. Never put
Disallow: /underUser-agent: *— that blocks every crawler on the internet, including Google and Bing.
If writing these by hand feels fiddly, our free robots.txt generator builds per-bot rules with checkboxes, so you can't mistype a bot name — the single most common reason a block silently fails.
When to Go Stricter Than the Template
The template above is the sensible default. Go further — blocking the answer engines too — only for specific reasons:
- Your content has direct licensing value. If you sell the content AI answers would quote — premium research, licensed photography, a paid course library — citations don't compensate you. Block everything and pursue licensing instead.
- Bandwidth abuse is real. If your logs show a bot hammering your server regardless of robots.txt, politeness is off the table. Move to server-level blocking.
- You are in a legal dispute or negotiation. If you are negotiating a licensing deal with an AI company, your crawl policy is leverage. Get advice before changing it mid-negotiation.
When to Allow Everything
The opposite extreme also has its place. If your site is new, small, or fighting for discovery, AI citations are free distribution. An allow-everything posture maximizes the chance that AI answers mention you, link you, and send you the curious visitors who click through. You can always tighten the policy later once you have traffic worth protecting. Obscurity is a bigger threat than training for most new sites.
Verify It's Working
An allowlist you never check is a wish. Do this after every change:
- Test the syntax. Run the file through our free robots.txt tester — it tells you exactly which user agents are blocked and which are allowed. One typo in a bot name voids that block.
- Check your logs in 48 hours. Search server access logs for the bot names. Blocked training bots should taper off; allowed answer-engine bots should keep appearing. If a "blocked" bot is still hammering you, it is ignoring robots.txt — escalate to server-level blocking.
- Watch for impersonators. Anyone can put "GPTBot" in a user-agent string. Verify suspicious traffic against published IP ranges — OpenAI publishes theirs as machine-readable JSON files — or do a reverse-DNS check. Our guide to verifying AI bots walks through the exact process.
- Match on the product token, not the version. Bot user-agent strings include version numbers that change (GPTBot/1.4 today, something else tomorrow). Rules keyed to the token —
GPTBot, notGPTBot/1.4— keep working. - Review quarterly. New bots appear constantly — our complete AI crawler list is updated regularly. A policy from January may be missing half the crawlers hitting you by October.
A Worked Example: Three Sites, Three Allowlists
The framework is easier to apply with concrete cases. A food blog monetized by ads keeps every answer engine allowed — recipe questions are citation gold in AI answers — and blocks only the pure trainers (GPTBot, CCBot, ClaudeBot). A B2B SaaS docs site goes further: it allows everything, because every AI answer that cites its documentation is a qualified lead walking in the door. A stock-photo marketplace blocks the lot, trainers and answer engines alike, because its images are the product and citations don't pay the photographers. Same robots.txt syntax, three different business decisions. If you can't say which of the three you are in one sentence, that sentence is the strategy work to do before you touch the file.
FAQ
Will allowing OAI-SearchBot let OpenAI train on my content?
No. OAI-SearchBot and GPTBot are separate crawlers with separate jobs: OAI-SearchBot builds the index behind ChatGPT's search answers, while GPTBot collects training data. OpenAI documents them independently, and blocking one does not affect the other. Allowing search visibility does not grant training access.
Does blocking GPTBot hurt my ChatGPT citations?
No. Citations in ChatGPT's search answers come from OAI-SearchBot's index, not from GPTBot's training crawls. This is the single most misunderstood point in AI crawler control: training opt-out and search visibility are separate switches. Flip only the one you mean to flip.
What about ChatGPT-User — should I block it?
Usually not. It isn't a crawler in the ordinary sense; it fetches a specific page when a person asks ChatGPT or a custom GPT about it. OpenAI says it isn't used to determine search inclusion, and that robots.txt rules may not apply because the fetches are user-initiated. Blocking it prevents ChatGPT from opening pages your own readers share — mostly self-defeating.
Should I block CCBot if I already block GPTBot?
Yes, if your goal is keeping content out of training data. CCBot feeds the Common Crawl dataset, which is downloaded and used to train many third-party models beyond any single company's. Blocking GPTBot alone leaves that entire downstream supply chain open.
How often should I review my allowlist?
Quarterly at minimum. New AI crawlers launch constantly, bot purposes change — a training bot can gain a search feature, or vice versa — and your own strategy evolves as your traffic grows. Put a recurring calendar reminder next to your sitemap and analytics reviews.
Isn't it simpler to just block everything and be done?
Simpler, yes — and measurably worse for most sites. Blanket blocks forfeit AI citations, referral traffic from answer engines, and visibility in assistants your audience actually uses. The fifteen minutes a selective policy takes pays for itself the first time an AI answer cites you instead of a competitor. Reserve the total block for content with real licensing value.
Bottom line: Don't block every AI bot — block the trainers and allow the answer engines. GPTBot, CCBot, ClaudeBot, Google-Extended, Bytespider, and meta-externalagent consume your content for training; OAI-SearchBot, PerplexityBot, meta-webindexer, and Applebot can cite you and send traffic. They are independent switches in robots.txt, so use them that way: one group per bot, never a global disallow, verify in your logs after 48 hours, and review the list every quarter.
