Everyone talks about blocking AI crawlers. Almost nobody shows real data about who actually does it. So we did something simple: we opened the robots.txt files of 40 major websites and wrote down exactly which AI bots each one blocks. The results surprised us — and they will probably surprise you too.
How we ran this survey
On 11 October 2026, we fetched the /robots.txt file of 40 well-known websites: global news outlets, tech platforms, e-commerce sites, reference sites, social networks, and Indian publishers. We used a standard desktop browser user-agent and a 10-second timeout per site.
For each site, we looked for rules naming 8 well-known AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, CCBot (Common Crawl), Bytespider (ByteDance/TikTok), Applebot-Extended, and Meta-ExternalAgent.
37 of the 40 sites returned a readable robots.txt. Three could not be surveyed: Flipkart returned an empty response, while NDTV and LinkedIn served bot-block pages instead of their robots.txt. We marked those as failed fetches rather than guessing — every number below comes from data we actually collected.
A quick note on what "blocked" means here: a Disallow rule in robots.txt is a request, not a lock on the door. Well-behaved crawlers obey it; badly-behaved ones ignore it. Our survey measures what sites ask for, which is still the clearest public signal of how the web feels about each AI bot.
The results: which AI bots get blocked most
Here is the headline table. "Effectively blocked" means the site either names the bot with a Disallow rule, or blocks all crawlers with a wildcard rule.
| AI crawler | Blocked by | Share |
|---|---|---|
| ClaudeBot (Anthropic) | 30 of 37 sites | 81.1% |
| Bytespider (ByteDance) | 30 of 37 sites | 81.1% |
| PerplexityBot | 27 of 37 sites | 73.0% |
| CCBot (Common Crawl) | 26 of 37 sites | 70.3% |
| Applebot-Extended | 25 of 37 sites | 67.6% |
| Meta-ExternalAgent | 25 of 37 sites | 67.6% |
| GPTBot (OpenAI) | 23 of 37 sites | 62.2% |
| Google-Extended | 21 of 37 sites | 56.8% |
The single biggest surprise: the most famous AI crawler is not the most blocked one. Let's go through what the data actually shows.
1. ClaudeBot and Bytespider top the block list
Anthropic's ClaudeBot and ByteDance's Bytespider are each blocked by 30 out of 37 sites — 81%. That is a striking number. Site owners appear to draw a line between crawlers that power a search-like product they can see (and potentially get traffic from) and crawlers that silently vacuum up content for model training.
Bytespider's position is easy to understand: it is widely reported as an aggressive crawler, and many webmasters have complained about its request volume. ClaudeBot's position is more interesting — Anthropic is seen as a responsible actor, yet sites block its training crawler more than any other named bot. The likely reason is simple: blocking ClaudeBot costs a site nothing in visible traffic, so there is no downside to saying no.
2. GPTBot is blocked far less than its fame suggests
Only 23 of 37 sites (62%) block GPTBot. Just 16 name and disallow it directly; 18 don't mention it at all. Given that GPTBot is the crawler most website owners have heard of, this is the survey's most counter-intuitive finding.
Why would sites block the lesser-known Bytespider but leave GPTBot alone? Two plausible reasons. First, many sites wrote their AI-crawler rules in a hurry during 2023 and copied block lists that emphasized aggressive crawlers. Second, some site owners may quietly hope that being crawled by OpenAI helps their content surface in ChatGPT answers — a visibility trade-off they are willing to make. If you want to block GPTBot yourself, our robots.txt generator includes it as a one-click option.
And then there is the outlier: The Verge is the only surveyed site that explicitly allows GPTBot — its robots.txt names the bot with an empty Disallow, which means "you may crawl everything." A tech publication betting on AI visibility rather than AI protection.
3. Google-Extended is the least blocked of all
Only 21 of 37 sites (57%) block Google-Extended, and 22 sites don't mention it at all. Google-Extended is the crawler Google uses for AI training (separate from regular Googlebot, which powers search). Publishers seem most comfortable with it — or they simply haven't gotten around to blocking it.
There may also be a fear factor at work: Google-Extended is documented by Google itself, and some webmasters worry — without evidence — that blocking anything with Google's name on it could affect their search rankings. (It doesn't. Google has stated that Google-Extended is independent of Googlebot.) If you want to opt out of Gemini training specifically, see our guide on blocking Google-Extended via robots.txt.
4. Eight sites use the nuclear option
Eight of the surveyed sites don't bother naming individual bots. They use User-agent: * with Disallow: / — a blanket block on every crawler, AI or otherwise: Reddit, Quora, Pinterest, Yelp, IMDb, Stack Overflow, Reuters, and the Wall Street Journal.
Note what this means: these sites block Googlebot too. They can afford to, because they don't depend on search traffic — they rely on logged-in users, apps, and APIs. For the rest of us, the nuclear option isn't practical, which is exactly why per-bot rules exist. Our robots.txt tester lets you check how your own rules treat each bot before you publish them.
5. News publishers are the strictest — with two notable gaps
The New York Times, the BBC, and Bloomberg each explicitly disallow all 8 AI bots. Forbes blocks 7 of 8. News organizations, whose entire business is original content, are the most aggressive defenders on the web. That matches their public posture: several major publishers are suing AI companies over training on their articles.
But two strict publishers have a genuine gap: The Guardian and the Washington Post each block 6 of 8 bots yet leave GPTBot and Google-Extended unmentioned. That means the two most important training crawlers — OpenAI's and Google's — are technically allowed on two of the world's biggest news sites, even as those sites block everyone else. Whether that's an oversight or a deliberate choice, it's the kind of hole worth checking for on your own site.
6. Wikipedia is fully open to AI crawlers
At the opposite extreme sits Wikipedia: it names zero of the 8 AI bots. No AI-specific restrictions at all — only standard crawl-delay and path rules that apply to everyone. That's consistent with Wikipedia's mission of freely reusable knowledge, and it partly explains why Wikipedia content is so heavily represented in AI training data and AI answers.
7. Indian publishers are comparatively relaxed
The Times of India names almost no AI bots in its robots.txt. The Indian Express leaves GPTBot, Google-Extended, and CCBot unmentioned. The Hindu leaves Google-Extended and Applebot-Extended open. Compared with their Western counterparts, Indian publishers appear to be earlier in the AI-crawler conversation — which also means it's a good time for Indian site owners to decide their policy deliberately rather than by default. Our pros-and-cons guide walks through the trade-offs.
8. GitHub treats AI bots like everyone else
GitHub takes a refreshingly boring approach: GPTBot, ClaudeBot, and PerplexityBot get the same partial restrictions on dynamic paths (like /*/issues and /*/pulse) as ordinary crawlers. No special AI panic, no blanket block — just the same technical hygiene applied evenly. For most sites, that's probably the sanest template.
What this means for your website
A few practical takeaways from the data:
Decide per bot, not per headline. The bots you have heard of (GPTBot) aren't necessarily the ones hitting your server hardest (Bytespider, PerplexityBot). Check your server logs before writing rules based on reputation.
Don't copy a block list blindly. The Guardian and Washington Post show that even sophisticated publishers leave gaps. Audit your own robots.txt against the full list of active AI crawlers — we maintain an updated one in our AI-crawler robots.txt template.
Remember the trade-off. Blocking a training crawler (ClaudeBot, Google-Extended, CCBot) protects your content but won't bring traffic. Blocking a search-facing crawler (like PerplexityBot) may cost you visibility in AI answers. Our guide on whether blocking GPTBot hurts SEO digs into this.
Test before you publish. A typo in robots.txt can block Googlebot by accident — the single most expensive mistake in technical SEO. Run every change through a robots.txt tester first.
Frequently asked questions
Which AI crawler is blocked by most websites?
According to our October 2026 survey of 40 major websites, ClaudeBot (Anthropic) and Bytespider (ByteDance) are tied as the most-blocked AI crawlers — 81% of surveyed sites block each of them.
Do most websites block GPTBot?
No — surprisingly, only 62% of the major sites we surveyed block GPTBot, and 18 of 37 don't mention it at all. The Verge even explicitly allows it. GPTBot is less blocked than ClaudeBot, Bytespider, PerplexityBot, and CCBot.
Can I just block all AI crawlers at once?
You can, with User-agent: * plus Disallow: / — but that also blocks Googlebot and kills your search traffic. Eight major sites (Reddit, Quora, Pinterest and others) do this because they don't depend on search. For everyone else, per-bot rules are the practical approach.
Does blocking an AI bot in robots.txt actually stop it?
Only if the bot respects robots.txt. Major crawlers from OpenAI, Anthropic, Google, and Perplexity document their compliance; aggressive or obscure crawlers may ignore it. A Disallow is a request, not enforcement — for sensitive content, combine it with server-level blocking.
How often do sites change their AI-crawler rules?
Frequently — this survey is a snapshot from 11 October 2026, and robots.txt files change all the time. Re-check the sites you care about periodically, and keep your own rules updated as new bots appear.
Where can I generate a robots.txt that blocks AI crawlers?
Our free robots.txt generator includes one-click toggles for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and more — no signup needed.
Methodology and limitations
Full transparency on how this data was produced: we fetched each site's /robots.txt once, on 11 October 2026, using curl with a desktop Chrome user-agent and a 10-second timeout. Bot matching was done on User-agent token substrings (case-insensitive). Near-identical bot names were treated as distinct — for example, Claude-Web was not counted as ClaudeBot, and OAI-SearchBot was not counted as GPTBot.
Three of forty sites could not be surveyed (empty responses or bot-block pages) and were excluded rather than estimated. The raw per-site, per-bot data is available as a CSV for anyone who wants to verify or extend the work. If you spot an error or a site whose rules have changed since the survey, let us know and we'll update the figures.
