News websites block AI crawlers more aggressively than any other industry. In a scan of 750 robots.txt files published on 20 October 2026, 41 of 69 news sites (about 59%) disallowed GPTBot at the root level. Finance came next-last: only 2 of 97 bank and insurer sites did the same. If you run a content site and wonder whether blocking AI bots is now "normal," the answer depends entirely on your industry — and the data shows the gap is enormous.
Below is what the numbers actually say, which bots each industry targets, and what to do with the findings on your own site.
The headline numbers
The figures come from Lantad, which parsed 750 robots.txt files across eight industry strata on 20 October 2026 and checked them against seven AI-crawler tokens. GPTBot — OpenAI's training crawler — was the test token reported per industry:
- News: 41 of 69 files disallow GPTBot (59%)
- Education: 10 of 108
- Healthcare: 9 of 96
- eCommerce: 7 of 87
- Government: 4 of 86
- SaaS: 3 of 120
- Travel: 2 files
- Finance: 2 of 97 (named: ally.com and bancolombia.com)
Two things stand out. First, news is not just ahead — it is in a different league, blocking at roughly six times the rate of education, the next-highest industry. Second, the industries that handle the most sensitive data (finance, healthcare, government) are among the least likely to block AI crawlers in robots.txt. That surprises people, but there is a logic to it, which we will get to.
Why news blocks hardest
News organisations have the strongest incentive to keep AI crawlers out: their product is text, text is exactly what AI models train on, and AI answers increasingly substitute for the visit that used to fund the newsroom. Several major publishers have sued AI companies over training on their content, so a robots.txt block is the cheap, public counterpart to the legal strategy.
The pattern is visible at the very top of the web. A separate analysis of the internet's top 2,000 domains, run on 2 August 2026 and published on Dev.to, found that of 1,530 domains with a reachable robots.txt file, 292 (19.1%) disallowed at least one of the four major AI crawlers — GPTBot, ClaudeBot, Google-Extended or PerplexityBot. Amazon blocks all five major AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and Bytespider), and Instagram, X, Pinterest and TikTok sit alongside it as platforms that shut the door almost entirely.
For a news publisher, the calculation is simple: an AI crawler takes the article and returns an answer, and the reader never visits. Blocking is brand protection and leverage at once — it is hard to negotiate licensing terms with a company whose bots you are still welcoming in for free.
Why finance, healthcare and government mostly don't
The Lantad data shows finance at 2 of 97 and government at 4 of 86. That looks like carelessness until you think about what those sites publish.
Banks, insurers, hospitals and government agencies do not sell articles. Their sites exist to serve customers and citizens: account logins, claim forms, service information, contact pages. Almost everything valuable sits behind authentication, which crawlers cannot reach anyway. The public pages are marketing and service content that the organisation actively wants found — including, increasingly, inside AI answers, where a customer asks "what is the overdraft fee at my bank" and gets the answer from the bank's own page rather than a hallucination.
Interestingly, Lantad found that 4 finance robots.txt files named GPTBot specifically in order to allow it — including commbank.com.au and geico.com. Some organisations have decided the trade runs the other way: being present in AI answers is distribution, and distribution is the point of the public site.
The honest read: blocking is a strategy for businesses whose content is the product. For everyone else, it is often an own goal.
The industry breakdown is the new part of the story; the baseline is older and more striking.
CipherCue pulled the robots.txt files of 110,744 websites and checked each against 19 AI crawlers — training bots, search-assistant bots and on-demand fetchers. Only 21.3% blocked at least one. Four in five sites have published no instruction at all.
That matters because silence defaults to permission. A crawler that finds no rule for its user agent follows the wildcard group, and most wildcard groups say "allow everything." So the effective state of the web is: a small, highly motivated minority blocks, and everyone else is open by default — whether they meant to be or not.
Training bots get blocked; answering bots get a pass
The most useful finding across these datasets is not which industry blocks, but what gets blocked. Publishers are not blocking "AI." They are blocking training and allowing answering — often on the same company.
The Q3 2026 robots.txt report from technologychecker.io, built on a Cloudflare Radar snapshot from 31 August 2026, puts ratios on it:
- GPTBot (OpenAI training) is disallowed 2.33 times for every time it is explicitly allowed.
- OAI-SearchBot (the crawler behind ChatGPT citations), from the same company, is allowed slightly more often than it is disallowed.
- ClaudeBot (Anthropic training) is disallowed more often than it is allowed.
- Applebot-Extended (Apple's training opt-out token) sits at 3.29:1 against, while plain Applebot is allowed more than it is blocked.
The pattern is consistent: site owners want their content cited in AI answers but not consumed for model training. OpenAI, Anthropic and Apple all split those two functions into separate crawler tokens precisely so robots.txt can express the distinction. The sites that write two stanzas instead of one get both outcomes; the sites that write "block everything AI" lose the citations too.
The wildcard trap hiding in these files
Lantad's FAQ section contains a detail that should make every site owner check their own file. Of the 55 files in its corpus that disallowed GPTBot while allowing OAI-SearchBot, the "allow" came from the wildcard group in 40 cases — because the site never named OAI-SearchBot at all.
That works today, under RFC 9309's rule that a crawler with no matching group follows the wildcard group. But it is fragile in two ways. First, it is invisible: anyone auditing the file sees a GPTBot block and assumes AI is handled, missing that the search bot's permission is accidental. Second, it breaks the moment you tighten the wildcard group later for an unrelated reason — one edit and your ChatGPT citations disappear with no warning.
The fix takes thirty seconds: name the search-assistant bots explicitly. If you want OpenAI's training crawler out but ChatGPT citations in, write both stanzas:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Do the same pairing for Anthropic (ClaudeBot vs Claude-SearchBot) and you have expressed the training-vs-answering split deliberately instead of by accident. You can generate the file with our free robots.txt generator and check it with the robots.txt tester before it goes live.
What this means for your site, by industry
If you publish content (news, blogs, magazines)
You are the 59% club's industry. The standard playbook: block the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider), keep the search-assistant crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed, and name each one explicitly. Read our pros-and-cons guide before you decide, because the citation trade-off is real — 44.5% of sites that block GPTBot accidentally block ChatGPT citations too.
If you sell products or services (ecommerce, SaaS, local business)
Your industry barely blocks, and for good reason: AI answers are becoming a discovery channel, and your public pages exist to be found. Blocking training crawlers is still defensible if you publish genuinely original content (buying guides, research), but think twice before touching the answering bots. Being the cited source when someone asks an AI "which CRM is best for a 10-person team" is worth more than the warm feeling of a blanket block.
If you are in finance, healthcare or government
Follow your industry's lead and focus on what actually matters: authentication for sensitive areas, accurate public information, and making sure AI answers quote your pages rather than inventing the numbers.
The trend is accelerating
Every dataset agrees on the direction. The Dev.to top-2,000 analysis called blocking's growth "a sharp jump from a year earlier, when blocking was still a fringe move rather than a default setting." CipherCue's one-in-five figure is a snapshot from 8 October 2026. And the infrastructure is moving underneath it: since 15 September 2026, Cloudflare blocks crawlers it classifies as "Training" or "Agent" by default for ad-carrying pages for new domains — sorting bots into Search, Training and Agent buckets and treating them differently at the edge.
AI crawlers now account for more than a fifth of web traffic while sending almost nothing back, and site owners have noticed. Expect the news industry's 59% to look normal rather than extreme within a year — and expect the training-vs-answering split, not the blanket block, to become the standard configuration.
FAQ
Which industry blocks AI crawlers the most?
News, by a wide margin. In Lantad's 20 October 2026 scan of 750 robots.txt files, 41 of 69 news sites disallowed GPTBot — about 59%, roughly six times the rate of the next-highest industry (education, at 10 of 108).
Do most websites block AI crawlers?
No. CipherCue's scan of 110,744 sites (8 October 2026) found only 21.3% block at least one of 19 AI crawlers. The other four in five have no AI-crawler rules at all, which means crawlers are allowed by default.
Should I block AI crawlers if my competitors do?
Not automatically. News sites block because their content is the product; finance and SaaS sites mostly don't because their public pages exist to be discovered. Match the strategy to your business model, and consider blocking training crawlers while allowing the answering bots that cite you.
What is the difference between blocking GPTBot and blocking OAI-SearchBot?
GPTBot collects content that may train OpenAI's models; OAI-SearchBot fetches pages to surface them in ChatGPT's search answers. Blocking GPTBot opts you out of training. Blocking OAI-SearchBot removes you from ChatGPT citations. They are separate tokens and can be controlled independently in robots.txt.
Is it safe to rely on the wildcard group to allow AI search bots?
It works, but it is fragile. Lantad found 40 of 55 files relied on the wildcard group for their OAI-SearchBot "allow" without naming the token. Name each bot explicitly so a future edit to the wildcard group cannot silently kill your AI citations.
Where does this data come from?
The industry breakdown is from Lantad's robots.txt parse of 750 sites across eight industries (20 October 2026). The web-wide baseline is CipherCue's 110,744-site scan (8 October 2026). The training-vs-answering ratios are from technologychecker's Q3 2026 report on Cloudflare Radar's 31 August snapshot. All three are linked above.
