Should You Block AI Crawlers? The Honest Pros and Cons for Site Owners

The short answer

Should you block AI crawlers? For most site owners, the answer is: not all of them. Block the training crawlers — the bots that collect your content to teach AI models — but leave the door open for the search and answer crawlers. That way you keep your content out of training datasets while still showing up in ChatGPT search, Claude answers, and Perplexity results.

If you only remember one thing from this guide, make it this: blocking training does not have to mean blocking citations. They are controlled by different bots, and you can allow one while blocking the other. Most of the loud advice online misses this distinction, and it leads people to either block everything (and quietly disappear from AI search) or allow everything (and give their content away to training pipelines).

This guide walks through both sides honestly, then gives you a middle path that works for most sites: block the training bots, allow the answer bots. There is a copy-paste robots.txt template below, plus a simple decision framework by site type.

The one thing most people get wrong about AI crawlers

The phrase "AI crawler" lumps together bots that do completely different jobs. Once you separate them, the decision gets much easier. There are really two buckets.

Bucket 1: Training crawlers

These bots collect web content that may be used to train or improve AI models. The well-known ones:

• GPTBot — OpenAI's crawler for content that may train its foundation models.
• ClaudeBot — Anthropic's crawler for content that could contribute to training Claude models.
• Google-Extended — a product token that controls whether Googlebot-crawled content can be used for Google's AI products like Gemini. Note that this is not the same as Googlebot: blocking Google-Extended does not affect your Google Search indexing or rankings.
• CCBot — Common Crawl's bot. Common Crawl publishes an open web archive that many AI companies and researchers use, so it is effectively a training-data feeder.

Blocking these bots is about training opt-out. OpenAI's own documentation confirms that disallowing GPTBot means your content should not be used in training its generative AI models, and Anthropic's documentation confirms that restricting ClaudeBot signals your future materials should be excluded from its training datasets. (Links to both official docs below.)

Bucket 2: Search and answer crawlers

These bots index your content so AI products can find it, cite it, and show it to users asking questions:

• OAI-SearchBot — OpenAI's search crawler. It surfaces websites in ChatGPT's search features.
• Claude-SearchBot — Anthropic's crawler for improving search result quality in Claude's answers.
• PerplexityBot — Perplexity's crawler, which indexes content for its answer engine.

Crucially, the companies behind these bots treat the two buckets as independent settings. OpenAI states plainly that each setting is independent — you can allow OAI-SearchBot so you appear in search results while disallowing GPTBot so your content is not used for training. Anthropic likewise documents each bot separately with its own robots.txt rules.

If you want a deeper breakdown of how the two OpenAI bots differ, we have a dedicated guide: GPTBot vs OAI-SearchBot: what each one does and why the difference matters.

The honest case for blocking AI crawlers

There are real reasons site owners block AI crawlers. Here they are, stated plainly.

1. Your content does not get used to train models

This is the big one. If you have spent years building tutorials, analysis, creative writing, or proprietary documentation, you may not want that work feeding models that could later reproduce or compete with it. Blocking training crawlers is the officially supported way to opt out.

2. Less server load and bandwidth

Some AI crawlers are aggressive. On smaller servers, heavy bot traffic can slow the site down for real visitors and, on metered hosting, add cost. Blocking the crawlers you do not want is a straightforward way to cut that load.

3. Less repackaging of your work without credit

AI answers do not always cite sources well. If your business depends on people visiting your pages (ad revenue, affiliate links, lead generation), content that gets absorbed and paraphrased without a link back is a real concern.

4. Control over copyright and licensing questions

The legal questions around AI training are still being argued in courtrooms. Blocking training crawlers does not resolve the law, but it is a clear, documented signal of your preference — and for sites with licensed or client-owned content, it is the responsible default.

The honest case against blocking AI crawlers

Now the other side. These costs are real too, and the loudest "block everything" advice rarely mentions them.

1. You can disappear from AI answers

If you block the search and answer crawlers — not just the training bots — your content may stop appearing in AI-generated answers. OpenAI is explicit: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce your site's visibility and accuracy in user search results. For any site that lives on discovery, that is a heavy price.

2. AI search can send real traffic

Perplexity and the search features inside ChatGPT and Claude cite sources with links, and people click them. Many publishers are already seeing meaningful referral sessions from AI answer engines. Blocking answer crawlers throws that traffic away.

3. If your competitors stay crawlable, the AI learns them instead of you

AI answers have to cite somebody. If you block answer crawlers and your competitors do not, the answers in your niche will quote your competitors, link to your competitors, and build their authority. You have not protected your niche — you have handed it over.

4. Blocking training bots does not undo what is already trained

Robots.txt blocks are forward-looking. Anthropic's own documentation frames the opt-out as applying to your site's "future materials." If your content was already crawled and used to train existing models, a new robots.txt rule does not remove it. That does not make blocking pointless — it still affects future models — but it means blocking is not the time machine some people expect.

5. robots.txt is a request, not a lock

The robots.txt standard (published as RFC 9309) is a politeness protocol. Reputable AI companies honor it — OpenAI, Anthropic, Google, and Perplexity all document their robots.txt support — but a scraper that does not care about reputation will ignore it. If your content must stay private, put it behind a login, not behind a robots.txt rule.

The middle path: three buckets, pick one

Instead of an all-or-nothing choice, think in three buckets:

1. Block all AI crawlers. Maximum content protection, minimum AI visibility. Makes sense for a small number of sites — see the framework below.
2. Middle path: block training bots, allow answer bots. Your content stays out of training datasets, but your site can still be cited and linked in AI answers. This is the right default for most sites.
3. Allow everything. You are happy for AI models to train on your content and you want maximum visibility everywhere. Some open-knowledge and promotional sites genuinely prefer this.

For the middle path, here is a ready-to-use robots.txt template. It blocks the four major training tokens and explicitly allows the three major answer/search crawlers:

# AI crawler policy: block training bots, allow search/answer bots
# Training crawlers -- opted out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Search/answer crawlers -- welcome (citations and AI search visibility)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Everything else -- your normal rules go here
User-agent: *
Allow: /

A few notes on using it:

• Do not block Googlebot. Googlebot is what gets you into Google Search. The training opt-out for Google is the Google-Extended token, and blocking it has no effect on your Search indexing or rankings.
• Repeat per subdomain. A robots.txt file on www.example.com does not cover blog.example.com. Anthropic's documentation explicitly says to add the rules for every subdomain you want to opt out.
• Allow up to a day for changes to take effect. OpenAI notes it can take roughly 24 hours from a robots.txt update for its systems to adjust.
• If server load is the problem, not training, consider Crawl-delay first. It is a non-standard directive, but Anthropic supports it, and it reduces load without cutting off visibility.
• Generate it safely. You can build and check your file with our free robots.txt generator, then test how it applies to specific bots before going live.

Which bucket fits your site?

Here is a quick way to place yourself. Find your site type and see where it lands.

News sites and publishers: middle path, carefully

Publishers are pulled both ways. Training opt-out protects original reporting, but news brands live on discovery — and AI answers are becoming a discovery channel. The middle path (block training, allow answers) fits most publishers. The exception: if your entire business is selling content licenses to AI companies, keep everything restricted and negotiate directly.

SaaS, blogs, and businesses that want leads: middle path

If you want customers, clients, or readers, visibility wins. Block the training bots if you are uncomfortable with model training, but let the answer crawlers in so your guides and product pages can be cited. A competitor that stays citable will take the mentions you declined.

Ecommerce stores: allow answers, block training selectively

Product pages, buying guides, and reviews are exactly the content AI shoppers' assistants cite. Blocking answer crawlers here costs you citations with zero payoff. Blocking training bots is harmless for most stores, since product descriptions were never their crown jewels — but check whether your supplier or brand contracts restrict reuse first.

Personal portfolios and hobby sites: your call

If the site is for fun or for showing work, there is no commercial downside either way. Many portfolio owners pick the middle path on principle: they are happy to be cited but do not want their creative work in training sets.

Sites with licensed, client, or paid content: block training bots

If you host content you do not fully own — client work, licensed images and text, paid courses, members-only material — opting out of training crawlers is the responsible default. The answer crawlers can stay allowed for your public marketing pages, but keep training bots away from anything with rights attached. And remember: anything truly private belongs behind authentication, not behind robots.txt.

Putting it in place without breaking things

A bad robots.txt edit can accidentally block all of Google, so do this carefully:

1. Back up your current robots.txt before touching it. Copy the existing file into a text note.
2. Make one change at a time. Add the AI crawler block first; do not rewrite your whole file in the same edit.
3. Never touch your Googlebot rules unless you mean it. Double-check that no new Disallow accidentally catches Googlebot.
4. Apply to every subdomain you care about — www, blog, shop, docs, and any others.
5. Test the result. Fetch your updated robots.txt and check how it reads for each bot token. Our walkthrough covers the process: How to block AI crawlers with robots.txt in 2026.
6. Watch your server logs for a week. Confirm the training crawlers stop and the answer crawlers still visit. A sudden drop in all bot traffic means you over-blocked.

Revisit this decision in six months

The AI crawling landscape moves fast. New bots launch, old ones get renamed, and policies shift — OpenAI has already expanded its crawler list several times, and Anthropic clarified its three-bot setup in updated documentation this year. A robots.txt policy you set today can quietly go stale.

Put a calendar reminder for six months from now. Check each company's official crawler docs for new bots, re-read the rules, and ask the same question again: block all, middle path, or allow all? The answer that is right today may not be right then. Staying current matters just as much as the choice itself — and if you are building an AI-search strategy around this, our guide on GEO vs SEO vs AEO explains how generative-engine visibility fits with traditional search.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot only collects content that may be used for training. ChatGPT search visibility is controlled by a separate crawler, OAI-SearchBot. OpenAI documents the two as independent settings: you can disallow GPTBot and still allow OAI-SearchBot, and your pages can appear in ChatGPT search answers.

Does blocking training crawlers delete my content from existing AI models?

No. Robots.txt is forward-looking. If your content was already crawled and used to train a released model, a new block does not remove it from that model. It applies to future crawling and future training datasets, which still makes it worth doing if training opt-out matters to you.

Do I really need a separate robots.txt for each subdomain?

Yes. A robots.txt file only applies to the host it lives on. If your blog, shop, or docs sit on subdomains, each one needs its own copy of the rules. This is explicitly stated in Anthropic's crawler documentation.

What if a bot ignores my robots.txt?

Reputable AI companies — OpenAI, Anthropic, Google, Perplexity — document their robots.txt support and honor it. But robots.txt is advisory, not access control. Bots that do not care about reputation may ignore it. For those, you need server-level blocking (firewalls, CDN bot management), not another robots.txt line.

Should I block PerplexityBot?

Only if you want out of Perplexity's answers. PerplexityBot indexes content for the answer engine, and Perplexity's answers link to their sources — many publishers treat it as a traffic source. Blocking it removes you from Perplexity citations almost entirely.

Can I slow crawlers down instead of blocking them?

Yes, and that is often the smarter fix when the real complaint is server load. The non-standard Crawl-delay directive asks polite bots to wait between requests. Anthropic officially supports it for its crawlers. If your site stays fast with the delay in place, you get the traffic and citations without the hosting bill.

Sources

Bot behavior and robots.txt rules referenced from official documentation:

• OpenAI — Overview of OpenAI crawlers (covers GPTBot, OAI-SearchBot, and ChatGPT-User; confirms settings are independent and updates take ~24 hours)
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? (covers ClaudeBot, Claude-SearchBot, and Claude-User; confirms Crawl-delay support and per-subdomain robots.txt)