Search your server logs for GPTBot and you will probably find it. Search for MistralAI-Index or Meta-WebIndexer and you might draw a blank — not because they don't exist, but because the AI crawler list grew faster than anyone can track. In 2026, AI companies document roughly 30 distinct crawler tokens, and most "blocking AI" confusion comes from treating them as one thing. They do three different jobs, obey different rules, and each gets its own switch in robots.txt.
The short answer: operators document roughly 30 AI crawler tokens in 2026. They do three different jobs — training models (like
GPTBot), powering AI search answers (likePerplexityBot), and fetching pages when a user asks (likeChatGPT-User). The big operators are OpenAI, Anthropic, Google, Perplexity, and Meta, and you control each token with a separate robots.txt rule. Blocking one never blocks the others.
Below: every token I could verify against official docs or independent traffic measurement as of October 2026, grouped by operator — exact token, what it does, one control note each. Tokens I couldn't verify are called out at the end, not silently listed.
OpenAI: four bots, four jobs
OpenAI runs separate systems with separate user agents and published IP ranges — different relationships with your content. Blocking one does nothing to the others.
GPTBot— the training crawler. It collects pages that may train OpenAI's models; it honors robots.txt and publishes its IP list at openai.com/gptbot.json. Control note:Disallow: /opts your content out of training only — ChatGPT search keeps working via OAI-SearchBot.OAI-SearchBot— the search crawler. It builds the index behind ChatGPT's search answers and citations; OpenAI says it is not used for training. Control note: allow it if you want ChatGPT citations; blocking it removes you from ChatGPT search answers.ChatGPT-User— the user-triggered fetcher. It visits a page when a specific person asks ChatGPT to read it, and OpenAI says robots.txt rules "may not apply" to these fetches. Control note: blocking it in robots.txt has limited effect — just make sure it gets a200with your content.OAI-AdsBot— the newest token. It validates landing pages submitted as ads on ChatGPT and only visits those pages; its data isn't used for training. Control note: leave it alone unless you advertise.
For the longer version of the training-vs-search split, our GPTBot vs OAI-SearchBot explainer covers the confusion that causes most accidental self-blocks.
Anthropic: the strictest of the big labs
Anthropic documents three tokens — and unlike every other major lab, says all three honor robots.txt, including the user-triggered fetcher. That makes Anthropic the most predictable to control.
ClaudeBot— training data collection. Honors robots.txt and supports the non-standardCrawl-delaydirective. Control note:Disallow: /for a clean training opt-out;Crawl-delayif load is the issue rather than principle.Claude-SearchBot— indexes pages for Claude's search features. Control note: allow it to stay eligible for Claude search citations.Claude-User— fetches pages when a Claude user asks about them. Control note: blocking it stops Claude retrieving your pages even when a user pastes your link. Allow it unless you want to be unreachable.
Legacy note: older lists include anthropic-ai and Claude-Web — Anthropic's 2024 strings, replaced by the 2025 triad. Harmless to keep as belt-and-suspenders, but they do nothing on their own.
Google: control tokens, not crawlers
Google works differently. It has no separate AI crawler — instead it offers robots.txt control tokens that act as permission switches over content Googlebot already crawls. The most misunderstood part of AI crawler control.
Google-Extended— opts your content out of Gemini model training and grounding (including Grounding with Google Search in Vertex AI). It has no separate HTTP user agent — you'll never see it in access logs. Control note:Disallow: /for a training opt-out. It doesn't affect Google Search inclusion or ranking, and doesn't keep you out of AI Overviews.GoogleOther— Google's crawler for internal research and one-off product crawls. Control note: allow it unless server load is a concern.Google-CloudVertexBot— crawls only when a site owner requests it, for Vertex AI use cases. Control note: only relevant if you use Vertex AI on your own site.Google-Agent— agents on Google infrastructure acting for users; user-triggered and generally ignores robots.txt. Control note: handle at the firewall level if you must; robots.txt won't reliably stop it.Google-GeminiNotebook— fetches pages users save as sources in Gemini's Notebook feature; it replacedGoogle-NotebookLMin August 2026. Control note: treat like any user fetcher — allow and serve properly.
Since AI Overviews are built from Google's regular indexed corpus, your lever is Googlebot presence, not any AI token — our Google-Extended guide covers the detail.
Perplexity: the search bot that explicitly isn't for training
PerplexityBot— crawls and indexes for Perplexity's search answers. Perplexity's docs state it is "not used to crawl content for AI foundation models." It honors robots.txt and publishes its IP ranges. Control note: allow it — Perplexity sends real referral traffic, and blocking it removes your citations without affecting training opt-out.Perplexity-User— visits a page when a user asks a question; "generally ignores robots.txt rules," per Perplexity's docs. Control note: you can't reliably block it; blocking PerplexityBot by mistake is the error to avoid.
Apple: one crawler, one permission switch
Applebot— Apple's real crawler, powering Siri, Spotlight, and Safari suggestions. Honors robots.txt; verify via reverse DNS (*.applebot.apple.com). Control note: allow it for Apple search visibility.Applebot-Extended— a control token, not a crawler: it doesn't fetch pages. It opts content out of training Apple's foundation models, and Apple says its rules aren't considered in search ranking. Control note:Disallow: /for training opt-out with zero search consequences.
Meta: the busiest AI crawler you're not watching
One naming detail: Meta's crawlers identify themselves in lowercase on the wire (meta-externalagent/1.1), though the docs print them capitalized. Robots.txt matching is case-insensitive, so either spelling works.
meta-externalagent— crawls for AI training and content indexing; honors robots.txt. Independent measurements rank it among the most active AI crawlers on small-business sites in 2026. Control note:Disallow: /to opt out of Meta AI training.Meta-WebIndexer— indexes pages for Meta AI search; honors robots.txt. Control note: allow it for visibility in Meta AI answers.meta-externalfetcher— fetches links for users and Meta's agentic AI features; user-initiated, documented as possibly bypassing robots.txt. Control note: allow and serve, like other user fetchers.
Amazon: three tokens, clearly separated
Amazonbot— powers Alexa and other Amazon services; data may also train Amazon's AI models. Honors robots.txt. Control note:Disallow: /if you don't want Amazon training on your content.Amzn-SearchBot— search indexing only, explicitly no generative-AI training. Control note: allow it to stay findable in Amazon's search surfaces.Amzn-User— user-initiated fetches (for example via Alexa); "may not follow all robots.txt directives." Control note: allowlist it like the other user fetchers.
Microsoft, Common Crawl, and ByteDance
bingbot— Microsoft's search crawler. Copilot's answers come from Bing's index, so this token matters for Microsoft's AI surfaces. Control note: allow it. Microsoft has no robots.txt training token yet — its opt-out is theNOARCHIVEmeta tag, which also keeps pages from being linked in Chat and Copilot.CCBot— the Common Crawl Foundation's crawler, building the free open web corpus many AI models train on. Honors robots.txt. Control note:Disallow: /keeps your content out of the dataset. Our CCBot opt-out guide walks through the trade-off.Bytespider— ByteDance's crawler. It shows up as the highest-volume AI bot on many sites, and its robots.txt compliance is disputed — some operators report it ignoring rules entirely. Control note: most owners block it, but enforce at the firewall or WAF level; robots.txt alone is unreliable.
Mistral, You.com, DuckDuckGo, and DeepSeek
MistralAI-User— user-initiated fetch for Mistral's Le Chat; honors robots.txt. Control note: allow and serve.MistralAI-Index— Mistral's search index crawler. Control note: allow for visibility in Mistral's search features.MistralAI-Training— Mistral's training crawler. Control note:Disallow: /for a training opt-out.YouBot— indexes for You.com's AI search; honors robots.txt. Control note: allow for You.com visibility.DuckAssistBot— fetches content for DuckDuckGo's AI answers; honors robots.txt, opt-outs taking effect within about 72 hours. Control note: allow it to stay eligible for Duck.ai answers.DeepSeekBot— a training crawler attributed to DeepSeek, widely seen in server logs. Honest caveat: no official DeepSeek documentation exists for this token, and its robots compliance is disputed. Control note:Disallow: /is harmless if it complies; enforce at the firewall if it matters.
Names you can ignore for now
You'll also see GrokBot (xAI), SoraBot, PetalBot (Huawei), Bravebot, and cohere-ai in some community lists. As of October 2026, I couldn't verify current first-party documentation for any of these — Cohere's own docs even say its crawlers aren't currently active. They stay out of the table on purpose. The discipline: don't write robots.txt rules for tokens you can't verify, and re-check vendor docs every few months, because this list changes.
Which bots actually show up the most
An independent measurement of small-business sites (September 6 to October 5, 2026) found the average site's AI traffic dominated by training crawlers in volume — meta-externalagent at 21.6% of AI requests, ClaudeBot at 18.5%, GPTBot at 12.2%, Amazonbot at 10.9% — while search and user-triggered bots reached more individual sites: OAI-SearchBot and ChatGPT-User each visited over 90% of measured sites. The takeaway: training bots cost the most bandwidth, but search bots decide whether you get cited.
How to control them: four patterns
Every robots.txt strategy combines four patterns:
- Block training, allow search. The most common setup: disallow
GPTBot,ClaudeBot,CCBot,Google-Extended,Applebot-Extended,meta-externalagent, andAmazonbot, while explicitly allowingOAI-SearchBot,Claude-SearchBot, andPerplexityBot. You stay citable; you stay out of training. - Allow everything. For maximum AI visibility: a permissive file or explicit allow rules per bot. Suits new sites chasing citations everywhere.
- Block everything. Disallow every token plus a catch-all
User-agent: *block. Know what you're buying — you lose all AI search citations, and the user-triggered fetchers (ChatGPT-User,Perplexity-User,meta-externalfetcher) will largely arrive anyway. Right for private or licensed-content sites; wrong for anyone wanting traffic. - Flip the permission switches.
Google-ExtendedandApplebot-Extendedaren't crawlers, so "blocking" them costs no visibility and buys a training opt-out. Unsure about everything else? Start here.
Hand-written rules are error-prone: one typo in a token name and the rule silently does nothing, and a stray catch-all wildcard can block the search bots you meant to allow. Build the file with the free robots.txt generator instead — it uses the verified tokens from this list and keeps training and search rules in separate groups. For the broader decision, see our pros-and-cons guide and the step-by-step robots.txt walkthrough.
Copy-paste robots.txt recipes
Three starting points. Pick one, adapt it, and test every token before publishing.
Recipe A — training out, search in (recommended for most sites):
# --- AI training crawlers: keep out --- User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / # --- AI search crawlers: welcome --- User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Applebot Allow: / User-agent: YouBot Allow: / User-agent: DuckAssistBot Allow: / # --- Everyone else, including user-triggered fetchers --- User-agent: * Allow: /
Recipe B — block nearly everything (private or licensed content):
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Disallow: /
Recipe C — maximum visibility (allow everything):
User-agent: * Allow: /
After publishing, paste your URL into the robots.txt tester and check each token — OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot, Google-Extended at minimum. Robots.txt matching follows the most specific group naming the bot, falling back to the wildcard group only when nothing names it — always confirm the fallback behaves as intended.
Don't trust the name tag: verifying a bot is real
A user-agent string is just a text label — anyone can send User-agent: GPTBot from a laptop. For enforcement that matters (rate limiting, firewall allows), check published IP lists: OpenAI publishes JSON lists for GPTBot, OAI-SearchBot, and ChatGPT-User; Perplexity publishes lists for both its bots. Anthropic publishes no IP ranges (it crawls from shared provider infrastructure), and Applebot verifies by reverse DNS. When a token has no verifiable identity and ignores your rules — Bytespider being the classic case — the honest answer is firewall-level enforcement, not a longer robots.txt.
FAQ
Will blocking GPTBot remove my site from ChatGPT's answers?
No. GPTBot only feeds model training. ChatGPT's search answers and citations come from OAI-SearchBot and ChatGPT-User — separate tokens with separate rules. Blocking GPTBot while allowing OAI-SearchBot is the standard "out of training, in of search" setup.
Does blocking Google-Extended keep me out of AI Overviews?
No — the most expensive misunderstanding in AI crawler control. Google-Extended is a training-and-grounding permission switch with no effect on Google Search inclusion or ranking. AI Overviews come from Google's regular indexed corpus, governed by Googlebot. To limit AI use of your snippets, the tools are nosnippet and max-snippet meta tags, not robots.txt.
Can I actually block ChatGPT-User or Perplexity-User with robots.txt?
Not reliably. Both vendors document that these user-triggered fetchers may ignore robots.txt — they act for a specific person asking about your page. The exception is Anthropic: Claude-User does honor robots.txt, so blocking it genuinely stops Claude retrieving your pages. For the others, allowlist them and serve a proper page.
Should I block Bytespider?
Most independent site owners do. It's consistently among the highest-volume AI crawlers, its robots.txt compliance is disputed, and it offers little in return — it feeds ByteDance's models, not a search engine that sends traffic. Pair the rule with firewall or WAF enforcement; robots.txt alone is unreliable.
How do I check which AI bots actually visit my site?
Search your last 30 days of access logs for the tokens in this guide — start with GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, meta-externalagent, and CCBot — and look at response codes, not just names. 200 means the crawl succeeded; 403/429 means your server or firewall is bouncing them. Then run your URL through the robots.txt tester to see which tokens your current rules actually allow.
