Blocking GPTBot May Be Blocking Your ChatGPT Citations: What the 2026 Robots.txt Census Found

Here is the short version: blocking GPTBot in your robots.txt may be blocking your ChatGPT citations too — if the same file also blocks OAI-SearchBot. A new GeoParrot census of the top 5,000 sites found that 238 out of 535 GPTBot-blocking sites (44.5%) also block OAI-SearchBot, the crawler that powers ChatGPT search citations. Nearly half the sites that meant to opt out of AI training probably accidentally opted out of being cited by ChatGPT as well.

That is a painful own goal. Training opt-outs and citation eligibility are controlled by two different OpenAI crawlers, and OpenAI documents them separately. Block the wrong one — or block both with a sloppy catch-all — and you lose the one thing most small sites actually want: free discovery traffic from AI answers.

This article walks through the census numbers, explains the three OpenAI crawlers and what blocking each one costs you, shows exactly how the accident happens in a typical robots.txt, and gives you a 10-minute audit so you can check whether your own site is in that 44.5%.

The 44.5% mistake

The data comes from GeoParrot, which fetched the robots.txt files of the top 5,000 sites on the Tranco list on 7 September 2026. Of those 5,000 sites, 2,771 actually served a robots.txt file — a response rate of about 55%, which is already a useful reminder that a lot of sites never bother with one at all.

Here is what the census found, as published in GeoParrot's write-up:

• 535 sites block GPTBot. That is the training-data crawler. So far, so deliberate.
• 238 of those 535 also block OAI-SearchBot. That is 44.5% — nearly half. OAI-SearchBot is the crawler that feeds ChatGPT's search features. Blocking it removes your eligibility to be cited in ChatGPT answers.
• The other 297 block GPTBot alone. They are out of the training data but still citable. That is the split that works, and it is exactly what the other half of sites got right.

The headline, in plain language: nearly half the sites that said "don't train on my content" also said — probably without meaning to — "don't quote me either."

Two more numbers from the census are worth sitting with. 25.2% of sites block at least one AI training crawler, which makes sense in an era of licensing disputes and opt-out tutorials. But 13.3% block at least one AI search crawler — the kind that decides whether your pages can appear in an AI-generated answer at all. And 10.7% block an AI search crawler while still allowing Googlebot: sites that show up fine in Google while being invisible to AI search. That split is the oddest finding of all, because Googlebot and AI search bots serve overlapping discovery jobs.

GPTBot vs OAI-SearchBot vs ChatGPT-User: the three OpenAI crawlers

OpenAI documents its crawlers separately, and the differences matter enormously. Here is the breakdown in table form:

The key line from OpenAI's docs, paraphrased: GPTBot and OAI-SearchBot have distinct jobs, distinct user-agent strings, and distinct robots.txt stanzas. You can block one and allow the other. Most site owners who end up in the 44.5% simply never realized they had to make the choice at all.

If you want the deeper crawler-by-crawler breakdown, we already published a full comparison of GPTBot vs OAI-SearchBot — this article builds on it with the new census data rather than repeating it.

How the accident happens: the catch-all stanza

How do 238 big, presumably well-managed sites end up blocking a crawler they probably never heard of? Almost always the same way: a lazy catch-all.

Here is what the bad version typically looks like. At some point in 2023 or 2024, a developer or an SEO consultant added a training opt-out — often copied from a blog post or a generator — and wrote something like this:

User-agent: *
Disallow: /

Or the slightly more targeted version that still sweeps everything up:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /private/
Disallow: /api/

The first version is the nuclear option: User-agent: * with Disallow: / blocks every bot on the web, including OAI-SearchBot, Googlebot, Bingbot, and your own uptime monitor. The second version looks more careful, but any User-agent: * group that contains a blanket Disallow: / will still catch OAI-SearchBot, because a bot that matches no specific stanza falls back to the wildcard group.

Here is the fixed version. Each bot gets its own stanza, and OAI-SearchBot is explicitly allowed:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

That is the whole fix. Three stanzas. GPTBot is shut out of training-data collection; OAI-SearchBot can crawl and index you for ChatGPT search citations; everything else follows your normal rules. If your site builder or AI-opt-out plugin generated a blanket block, this is the edit you need to make.

One detail worth noting: in robots.txt, an explicit Allow: / under a named user-agent overrides a broader Disallow in the wildcard group, because the most specific matching group wins. So even if you want to keep a restrictive wildcard group for other reasons, adding a named stanza for OAI-SearchBot rescues it. You can generate a clean file with distinct bot stanzas using CrawlGuards' free robots.txt generator — it treats AI training and AI search crawlers as separate choices, which is exactly the habit the census shows most sites are missing.

The 296 sites visible in Google but invisible in AI answers

The strangest census finding is the one that is easiest to overlook. 296 sites block at least one AI search crawler while allowing Googlebot. They rank in Google. They can be found through a search box. But they have removed themselves from the pool of pages that AI search can cite.

The names on GeoParrot's list are striking: facebook.com, instagram.com, twitter.com, amazon.com, x.com, tiktok.com, pinterest.com, yahoo.com, ebay.com, imdb.com, unsplash.com — plus major publishers including nytimes.com, cnn.com, theguardian.com, forbes.com, bbc.com, and reuters.com.

For the big publishers, this is often deliberate. Outlets like the New York Times have licensing deals and active legal strategies around AI training and content use; blocking AI crawlers is a negotiating position, not an oversight. But for the thousands of smaller sites that copy-pasted a big publisher's robots.txt snippet, or clicked a one-tap "block all AI bots" option in a plugin, it is usually an accident nobody has checked since.

And here is the asymmetry that makes the accident costly: as AI search grows as a discovery channel, "visible in Google, invisible in ChatGPT" is a shrinking comfort. You keep the legacy channel and forfeit the new one. For a publisher with lawyers and licensing revenue, that is a calculated trade. For a small blog, a local business site, or an affiliate site that depends on discovery traffic, it is a trade they never agreed to.

Check your own site in 10 minutes

You do not need a tool subscription or a consultant to find out whether you are in the 44.5%. Here is the audit:

Step 1: Fetch your robots.txt

Open https://yoursite.com/robots.txt in a browser. If it returns a 404 or an empty file, you have no robots.txt — meaning nothing is blocked, and you can stop here (your issue is the opposite one: you might want to add a GPTBot block, which we cover in our guide to whether blocking GPTBot hurts SEO).

Step 2: Read the stanzas, looking for these three strings

Search the file for GPTBot, OAI-SearchBot, and User-agent: *. There are four possible outcomes:

• GPTBot blocked, OAI-SearchBot explicitly allowed: you are in the good 55.5%. No action needed.
• GPTBot blocked, no OAI-SearchBot stanza, but a restrictive User-agent: * group: you are almost certainly in the accidental 44.5%. OAI-SearchBot falls back to the wildcard group and inherits the block. Fix it with the three-stanza pattern above.
• Both explicitly blocked: deliberate or not, check whether you meant it. If you copied the file from a tutorial, you probably did not.
• Neither mentioned, permissive wildcard: everything is allowed, including training crawlers. Decide whether that is what you want (see the decision framework below).

Step 3: Test the file properly

Reading the file is not enough — you need to test how bots interpret it. Google Search Console has a robots.txt tester that shows which rules apply to a given URL and user-agent. Paste a few of your key URLs, and check the effective rules for both GPTBot and OAI-SearchBot. A dedicated robots.txt tester does the same thing without a Search Console property, which is handy if you manage client sites.

Step 4: Check your server logs for who actually visits

robots.txt tells bots what to do; logs tell you what they actually did. Look for the user-agent strings GPTBot and OAI-SearchBot in your last few weeks of access logs. A simple scan works:

grep -E "GPTBot|OAI-SearchBot" /var/log/nginx/access.log | wc -l

If OAI-SearchBot shows up and fetches pages successfully, your file is already fine — the bot is visiting. If you see it hitting your robots.txt and then never coming back, your file is probably blocking it. If you see neither, you are either too small to be on its crawl schedule yet, or it was blocked long ago and gave up.

One caution on logs: the absence of GPTBot in your logs does not prove your block is working. Absence of OAI-SearchBot in your logs, combined with a restrictive wildcard stanza, is strong evidence your block is accidentally working against you.

The honest caveats

Before you rush to allow OAI-SearchBot everywhere, three things this census does not claim — and that honest SEO advice has to include.

Caveat 1: eligibility is not citation. Allowing OAI-SearchBot makes your pages eligible to be cited in ChatGPT answers. It does not guarantee they will be. ChatGPT cites pages it considers authoritative, current, and directly relevant to the question. Content quality decides citation; robots.txt only decides whether you are allowed in the room. If your content is thin, outdated, or indistinguishable from ten competitors, fixing robots.txt will not summon citations out of nowhere. The fundamentals — expertise, originality, freshness — still do the heavy lifting, which is why we published a full guide to getting cited by ChatGPT that starts from content, not configuration.

Caveat 2: blocking is asymmetric across AI platforms. GeoParrot notes an instructive example: one outlet whose blocking killed its ChatGPT and Gemini citations was still cited by Grok 37,642 times. Different AI systems use different crawlers, honor robots.txt differently, and draw on different indexes. Blocking OAI-SearchBot does not block Perplexity's crawlers, Anthropic's, or xAI's. If your strategy is a principled "no AI use of my content at all," a single robots.txt stanza against GPTBot is nowhere near enough — you need a platform-by-platform approach. Conversely, if you only fix the OAI-SearchBot line, do not assume the rest of the AI ecosystem is handled.

Caveat 3: robots.txt is a request, not a lock. It is a voluntary protocol. Respectable crawlers — Googlebot, Bingbot, OAI-SearchBot, and yes, GPTBot — honor it. Scrapers, rogue AI data pipelines, and anyone fetching your pages through a headless browser do not. Blocking GPTBot keeps your content out of OpenAI's documented training pipeline; it does not make your content un-scrapable. Treat robots.txt as one layer (the polite layer), and use access controls, licensing, and legal terms for the rest.

The decision framework: who should block what

Not everyone should copy the same file. Here is how to think about it:

Block training crawlers (GPTBot) if:

• You are a publisher with, or pursuing, a licensing deal for AI training data. Your content is the product; giving it away as training data undercuts the deal.
• You run a paywalled or subscription product and your competitive edge is proprietary content.
• You have a principled objection to AI training on your work and understand the trade-offs.

Do not block AI search crawlers (OAI-SearchBot) if:

• You depend on discovery traffic — blogs, niche sites, local businesses, affiliate sites, documentation, SaaS marketing sites.
• You have no licensing leverage and no legal department. For small sites, a ChatGPT citation is pure upside: a new traffic channel you do not have to pay for.
• You are not sure why the block is there. An inherited rule from a plugin or a copied tutorial is not a strategy.

For most independent site owners, the correct configuration is the one the 297 got right: block GPTBot, allow OAI-SearchBot. Opt out of training, stay citable. That is the split that works.

What to do this week

The census is a mirror, not a prophecy. Its value is that it turns a vague anxiety — "did I break something with that AI block I added two years ago?" — into a concrete check you can run in ten minutes.

So: fetch your robots.txt today. Search for the three strings. If a catch-all is quietly blocking OAI-SearchBot, split the stanzas. Then test the file, glance at your logs, and move on — the configuration part of AI search visibility is a ten-minute job, and the content part is the actual career.

The 238 sites in the census did not make a decision; they inherited one. You do not have to.

FAQ

Will blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI's training-data crawler and has nothing to do with Google's index. Blocking it does not affect how Googlebot crawls or ranks your pages. We covered this in detail in our article on whether blocking GPTBot hurts SEO.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects training data for OpenAI's models; OAI-SearchBot crawls and indexes pages so ChatGPT can cite them in search answers. OpenAI documents them as separate crawlers with separate robots.txt user-agent strings. Blocking GPTBot keeps you out of training data with no effect on citations; blocking OAI-SearchBot removes your citation eligibility. See our full GPTBot vs OAI-SearchBot comparison for the complete breakdown.

How do I allow OAI-SearchBot while blocking GPTBot?

Give each bot its own robots.txt stanza: Disallow: / under User-agent: GPTBot, and Allow: / under User-agent: OAI-SearchBot. Named stanzas override the wildcard group, so this works even if you keep a restrictive User-agent: * section. You can build the file with CrawlGuards' robots.txt generator.

Does allowing OAI-SearchBot guarantee ChatGPT will cite my site?

No. It only makes your pages eligible for citation. ChatGPT cites pages it judges authoritative, current, and directly relevant to the question. Content quality — expertise, originality, freshness — is what earns citations; robots.txt just keeps you in the candidate pool.

I blocked GPTBot two years ago. How do I know if I also blocked OAI-SearchBot by accident?

Open your robots.txt and look for a User-agent: * group containing Disallow: / or broad disallows, with no separate OAI-SearchBot stanza. If that is what you have, OAI-SearchBot falls back to the wildcard group and inherits the block — you are in the 44.5%. Add an explicit User-agent: OAI-SearchBot stanza with Allow: /.

Do other AI search engines respect the same blocks?

Not necessarily. Perplexity, Anthropic, Google, and xAI each run their own crawlers with their own user-agent strings, and they honor robots.txt independently. Blocking OAI-SearchBot does not block them, and allowing it does not guarantee they crawl you. Audit each platform's crawlers separately if you want a complete picture.