You searched ChatGPT or Google's AI Overviews for a topic your site covers well — and your site isn't there. Before you assume your content isn't good enough, know this: in most cases we investigate, the site is blocking the AI search crawlers itself. A robots.txt file or a firewall setting you added months ago is quietly telling those bots to stay out, and they listen. The good news: you can run a full diagnosis in about 10 minutes and fix most causes today.
Here's the honest version: there is no single "AI index" to submit to. ChatGPT, Claude, Perplexity, and Google's AI Overviews each pull from their own crawled content. If a crawler never fetched your page — or fetched it and found nothing usable — you simply don't exist in its answers. The nine reasons below cover almost every case, in the order you should check them.
Run the 10-minute diagnosis first
Before touching anything, run these four checks. They tell you exactly which of the nine reasons is biting you, so you fix the right thing instead of guessing.
1. Read your robots.txt the way a bot does
Open yoursite.com/robots.txt and read it slowly. Look for any group that names AI bots — GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Google-Extended — and any User-agent: * group with a Disallow: /. If both exist, you may have reason #1. Paste your URL into the robots.txt tester to see exactly which bot is allowed and which is blocked — faster than reasoning through group matching by hand.
2. Check your server logs for the bot tokens
Search your last 30 days of access logs for these exact strings: OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User. Zero hits at all means the bots never arrive — blocked at the DNS, firewall, or robots level. Hits with 403 or 429 status codes mean they arrive and get rejected — that's a server-level block (reason #3). Hits with 200 but still no citations means the crawl happens but something downstream is wrong (reasons #4–#7).
3. Ask the engines directly
Open ChatGPT, Claude, and Perplexity and ask: "What does [yoursite.com] say about [your topic]?" Try a fresh, unlogged-in session too, since chat history can shape answers. If Perplexity lists your domain in its sources but ChatGPT doesn't, your problem is OpenAI-bot-specific. If none of them know you, it's a crawling or indexing problem, not a ranking problem.
4. Check Google Search Console for index coverage
For AI Overviews specifically, open GSC's Pages report and confirm your target page is actually indexed. If it's excluded ("Crawled - currently not indexed," "Discovered - currently not indexed," or blocked by robots), AI Overviews can never use it, because AI Overviews are built from Google's regular indexed corpus. Fix the Google problem first and the AI Overview problem often follows.
Reason 1: A catch-all wildcard in robots.txt blocks the search crawlers
This is the #1 cause, and it's almost always an accident. Sometime in the last two years, you (or a plugin, or a tutorial) added AI training bots to your robots.txt. The typical file looks like this:
User-agent: GPTBot Disallow: / User-agent: * Disallow: /
Here's what most people miss about how robots.txt matching works: a crawler follows the most specific group that names it, and falls back to the User-agent: * group only if nothing names it. In this example, GPTBot is blocked by its own group. But OAI-SearchBot (ChatGPT's search crawler), Claude-SearchBot, and PerplexityBot match nothing — so they fall back to User-agent: *, which blocks everything. You meant to stop training crawls. You also stopped every AI search crawler, which is what kills ChatGPT, Claude, and Perplexity citations.
How to check: Open your robots.txt. If you see a User-agent: * group with Disallow: / (or any disallow covering your content pages) and separate groups for AI bots, every AI search bot not explicitly named is blocked by the fallback. Test it with the robots.txt tester — enter your URL and check OAI-SearchBot, Claude-SearchBot, and PerplexityBot specifically.
The fix: Rewrite the file with explicit Allow or narrow Disallow rules for the search bots you want. To keep training bots out but let search bots in:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: /
Then re-test every bot token in the tester before publishing. If writing this by hand feels risky, use the robots.txt generator to build a clean file with separate, explicit rules for training and search bots.
Reason 2: You blocked the training bot thinking it was the search bot
Training and search are separate switches, and the bot names reflect that. OpenAI documents this plainly: GPTBot is the training crawler; OAI-SearchBot feeds ChatGPT search. Anthropic has three: ClaudeBot (training), Claude-SearchBot (search citations), and Claude-User (user-initiated retrieval). Blocking the training bot does not block search visibility — and many site owners who wanted "no AI" blocked the wrong one, or blocked both when they only meant to block one.
The most common mix-up: someone blocks GPTBot to keep content out of training data (a reasonable choice), then assumes their pages can still appear in ChatGPT's search results. That part is actually fine — ChatGPT search uses OAI-SearchBot, which is unaffected. The problem runs the other way: owners who copy-pasted a "block all AI" snippet blocked the search bots too, or never realized Claude-SearchBot existed as a separate token from ClaudeBot.
How to check: List every AI user-agent in your robots.txt and map each one to training or search. Training: GPTBot, ClaudeBot, CCBot, Google-Extended (training/grounding only). Search/indexing: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot.
The fix: Decide per purpose, not per provider. A common sensible setup: block the training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) and allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). That keeps your content out of model training while staying eligible for citations and traffic from AI search. For a full token-by-token table of training vs. search crawlers, see this breakdown of AI bot tokens before you edit the file.
Reason 3: Your firewall or bot protection blocks them at the server level
Robots.txt is only consulted if the bot can reach your server. A WAF or bot-protection layer — Cloudflare Bot Fight Mode is the usual suspect — can return 403 or a JavaScript challenge to AI crawlers before robots.txt ever comes into play. Your robots.txt looks perfectly permissive while the bots are being bounced at the door. This one is invisible in every robots.txt test, which is why it's so often missed.
Why does it happen? Aggressive bot-fighting rules treat unfamiliar user agents as threats. AI search crawlers are newer tokens, and some WAF rulesets score them like scrapers. The giveaway in your logs: the bot's user agent appears, but every response is a 403, 503, or a challenge page instead of your content.
How to check: Search server logs for the AI bot tokens and look at the HTTP status codes, not just the user agents. If you see 403 or 429 responses to OAI-SearchBot, Claude-SearchBot, or PerplexityBot, your server layer is the blocker. Also check your Cloudflare (or other WAF) firewall event log filtered by those user agents — blocked requests show up there with the rule that fired.
The fix: Whitelist the AI search crawlers in your WAF. Perplexity recommends verifying both the user agent string and the published IP ranges when writing allow rules, because user agents alone can be spoofed. Turn off blanket "fight mode" behavior for these tokens and give them a dedicated allow rule above your aggressive bot rules. Then confirm in the logs that the next crawl gets a 200.
Reason 4: You're not indexed by Google — and AI Overviews inherit that
Google's AI Overviews are generated from the standard Google-indexed corpus. That means every Google indexing problem is automatically an AI Overview problem: if your page isn't indexed, is indexed but ranks poorly, or is excluded by a crawl issue, it can't be cited. Blocking Google-Extended does not remove you from AI Overviews — that token only controls training/grounding use, not the search index. So if you blocked Google-Extended hoping to stay out of AI Overviews, it didn't work, and if you're trying to get in, Google-Extended is irrelevant either way.
This reason matters mostly for AI Overviews, not for ChatGPT or Perplexity, which crawl independently of Google. If you're missing from ChatGPT but present in AI Overviews, skip this one — your problem is elsewhere.
How to check: In Google Search Console, inspect the exact URL. Look for "URL is on Google" — if it isn't, read the coverage reason: robots block, noindex, "crawled - currently not indexed," "discovered - currently not indexed," or server errors. Also check the page's actual ranking for its target query; AI Overviews heavily favor pages already ranking well in classic results.
The fix: Resolve the specific coverage issue (request indexing after fixing robots/noindex problems, improve internal linking to thin pages, speed up slow pages), then wait for Google to recrawl. AI Overviews eligibility follows the index — there's no separate AI Overview submission.
Reason 5: Your content only renders with JavaScript
Some AI crawlers do not execute JavaScript. If your main content is rendered client-side — a React/Next.js app serving an empty shell, text injected by JS after load, tabs and accordions that only populate on click — a bot that fetches the raw HTML sees an empty page. Nothing to quote, nothing to cite. Your site looks rich in a browser and blank to the crawler.
This one is getting more common as sites move to JS-heavy stacks, and it's brutal because every browser-based check you do shows the content fine. You have to look at what the bot sees.
How to check: Disable JavaScript in your browser (or use curl to fetch the page) and look at the raw HTML. Is your article text present in the response, or only after scripts run? If the headline and body copy appear only with JS enabled, AI crawlers that skip JS see nothing. Google's "view rendered HTML" in GSC helps too, but curl is the honest test for non-Google bots.
The fix: Serve the main content as static HTML. Server-side rendering (SSR) or static generation for article pages puts the text in the initial HTML response, which every crawler can read. Keep JS for interactivity, but never make the article body depend on it. After fixing, re-fetch with curl and confirm the text is in the source.
Reason 6: A stray noindex, canonical, or meta tag is disqualifying the page
Sometimes the bots crawl fine and find your content — but a page-level signal tells every search system to ignore it. A noindex meta tag or HTTP header, a canonical tag pointing at a different URL, or an x-robots-tag from a plugin can remove a page from eligibility silently. Staging settings pushed to production are the classic source: a developer's noindex from the test environment survives the launch, and months later the site owner wonders why AI engines never mention them.
How to check: View the page source and search for noindex and canonical. The canonical must point to the page's own URL (or a deliberate preferred variant). Check response headers too: curl -I yoursite.com/page and look for x-robots-tag: noindex. In GSC, the URL inspection tool reports the "user-declared canonical" vs. "Google-selected canonical" — a mismatch is a red flag.
The fix: Remove the stray noindex, fix the canonical to self-reference, and strip rogue x-robots headers. Then request reindexing in GSC and re-test the AI engines after recrawl.
Reason 7: The site is too new or too thin to quote
Crawling is only half the battle; selection is the other half. Answer engines favor pages with direct answers, quotable facts, and clear structure. If your site is new, has thin pages, or buries the answer under long intros, the bots may crawl you regularly and still never cite you. This isn't a block — it's a quality bar.
What "quotable" actually means: a page that answers a specific question in the first hundred words, uses descriptive headings, states facts plainly (with numbers, names, dates where relevant), and organizes steps in lists. Compare your page honestly against whatever currently gets cited for your query. If the cited page gives the answer in paragraph one and yours reaches it in paragraph six, that's your gap.
How to check: Ask the AI engines the exact question your page targets and read the cited sources. Note what those pages do that yours doesn't: direct lead answer, clear headings, concrete specifics. Also check whether the bots even visit you — a crawl in the logs with no citations points at content quality, not access.
The fix: Restructure around the question. Put the direct answer first, add specific facts a model can quote, use real headings instead of clever ones, and cut filler. New sites should also build topical depth — a cluster of related articles signals expertise better than one lonely page. Expect this one to take weeks of consistent publishing, not an afternoon.
Reason 8: You blocked the user-initiated fetchers
There's a subtle third category beyond training and search crawlers: user-initiated fetchers like ChatGPT-User and Perplexity-User. These act when a user asks the AI to look at a specific page — "summarize this article" with your URL pasted in. The nuance matters because they behave differently from the crawlers.
OpenAI says robots.txt rules may not apply to ChatGPT-User, since those fetches are triggered by an individual user asking about your specific page — the user agent just identifies itself. Perplexity-User generally doesn't respect robots.txt either, for the same reason. But Anthropic takes the stricter line: it says all three of its bots — ClaudeBot, Claude-SearchBot, and Claude-User — honor robots.txt, and blocking Claude-User stops Claude from retrieving your pages for users who link them. That last detail surprises people: a blanket Claude block can remove you from conversations where a user explicitly pastes your link.
How to check: Look for these tokens in your robots.txt and WAF rules. A broad "block everything Anthropic" rule that names Claude-User is the usual culprit.
The fix: Decide whether you want to be retrievable when users ask about your pages. If yes, allow Claude-User explicitly (Anthropic documents its three-bot behavior in this breakdown). For OpenAI and Perplexity, user-initiated fetches will generally happen regardless of your robots.txt — that's their documented behavior, not a bug you need to fix.
Reason 9: You expected blocking Google-Extended to affect AI Overviews — it doesn't
This one is a misunderstanding rather than a block, but it wastes a lot of people's time. Google-Extended is the token that controls whether your content is used for Gemini training and grounding. Blocking it does not remove you from AI Overviews, because AI Overviews draw from the regular Google search index, which is governed by Googlebot — a completely separate token. Conversely, if you're trying to appear in AI Overviews, unblocking Google-Extended changes nothing; your lever is ranking in classic Google results.
People land here from both directions: owners who blocked Google-Extended and are surprised to still see their content in AI answers, and owners who unblocked it expecting citations to appear. Neither expectation matches how the systems work.
How to check: Search your robots.txt for Google-Extended. Note which problem you actually have: unwanted AI Overview appearances (a Googlebot/index problem, not a Google-Extended problem) or missing citations (also a Google index/ranking problem).
The fix: If you want out of AI Overviews, the path runs through your Google index presence, not Google-Extended. If you want in, improve your classic Google rankings for the query — that's the corpus AI Overviews cite from.
What to fix first, second, third
Work in this order. Each step is cheap, and most sites find their answer in the first two.
- First: fix robots.txt. Run the robots.txt tester against
OAI-SearchBot,Claude-SearchBot, andPerplexityBot. If any are blocked, rewrite the file with explicit allow rules for search bots (reasons #1, #2). Rebuild cleanly with the robots.txt generator if the file has grown messy. This single step resolves the majority of cases. - Second: check the server layer. Search logs for 403/429 responses to AI bot tokens and check your WAF's blocked-request log. Whitelist the search crawlers with user-agent plus IP-range rules (reason #3). A perfect robots.txt means nothing if the firewall bounces the bots.
- Third: verify Google indexing. For AI Overviews, confirm the page is indexed and ranking in GSC; fix coverage issues and request reindexing (reason #4). This step is irrelevant for ChatGPT/Perplexity-only problems.
- Fourth: audit the page itself. Curl the page with JS disabled to confirm the content is in the raw HTML (reason #5), then check source and headers for noindex and canonical problems (reason #6).
- Fifth: improve quotability. If bots crawl you with 200s and still never cite you, restructure the content — direct answer first, concrete facts, clear headings — and build topical depth over the following weeks (reason #7). Review the remaining token nuances (#8, #9) only if they match your specific setup.
Then give it time. After you unblock a bot, it has to recrawl your site, and answer engines update their cited sources on their own schedules. Check back with direct questions to each engine after a week or two rather than assuming the fix failed on day one. For a broader look at controlling how AI bots treat your site, browse the CrawlGuards blog — the robots.txt and AI-search guides there go deeper on each of these controls.
FAQ
How long after fixing robots.txt will AI engines start citing my site?
There's no fixed timeline. The crawler has to revisit your robots.txt, recrawl your pages, and the answer engine has to refresh its sources — each step runs on the provider's own schedule. In practice, site owners typically see changes within one to three weeks of the fix. Ask each engine directly about your site weekly rather than daily; the refresh is not instant.
Does blocking GPTBot hurt my visibility in ChatGPT search results?
No. GPTBot is OpenAI's training crawler, and ChatGPT's search answers are fed by OAI-SearchBot, a separate crawler with its own robots.txt rules. Blocking GPTBot keeps your content out of training data without affecting search citations. Just make sure you haven't also blocked OAI-SearchBot — that's the token that actually matters for visibility.
Can I block AI training but still appear in AI search answers?
Yes — that's exactly what the separate tokens are for. Block the training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) and explicitly allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). You keep training out while staying eligible for citations and the traffic that comes with them.
Will AI engines cite me if I'm not indexed by Google?
It depends on the engine. ChatGPT, Claude, and Perplexity crawl independently of Google, so a Google indexing problem doesn't block them. Google's AI Overviews are different: they're generated from Google's indexed corpus, so a page Google hasn't indexed can't appear there. Diagnose per engine instead of assuming one fix covers all of them.
Why does my competitor get cited but I don't?
Usually one of three things: their robots.txt and firewall let the search bots in while yours doesn't, their pages are indexed and ranking in Google (which feeds AI Overviews), or their content is more quotable — direct answers up top, concrete facts, clear headings. Run the 10-minute diagnosis on your own site first; the answer is almost always in reasons #1 through #4.
Is there a quick way to test which AI bots my robots.txt blocks?
Yes. Paste your domain into the free robots.txt tester and check each AI bot token individually — OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, and Google-Extended. It shows exactly which group matches each bot, which is much faster than tracing the fallback logic by hand.
