Yes, you can reduce how much of your content AI companies scrape — but there is no single switch that stops it completely. The honest approach is a layered one: block the training crawlers with robots.txt, use one-click AI bot blocking in Cloudflare, fix your copyright and terms pages, and understand which bots you should never block. This guide walks through each layer from a blogger's and creator's point of view, including the mistakes that cost people their search traffic.
The uncomfortable truth about "protection"
Before touching any settings, you need to understand what technical protection can and cannot do. This saves you from the two most common mistakes: believing you are fully protected when you are not, and blocking the wrong bots and killing your own traffic.
First, robots.txt is a request, not a lock. Reputable AI companies — OpenAI, Anthropic, Google — respect it. Aggressive scrapers, data brokers, and bad actors ignore it completely. A robots.txt block stops polite crawlers from fetching your pages going forward. It does not remove content from models that already trained on it, and it does not stop datasets that already copied your site.
Second, blocking is forward-looking only. If your ten years of blog posts are already in a training dataset, no robots.txt rule un-trains a model. What you are doing is drawing a line for the future: from today on, compliant crawlers stay out of your training data.
Third, some "AI bots" are actually your traffic. Blocking the wrong user-agent can remove you from ChatGPT search answers, Perplexity citations, and Google's AI Overviews. For most creators, citations in AI answers are free exposure. The goal is to block training while staying visible in search and AI answers. That distinction is the whole game.
Know the bots visiting your site
AI companies run different crawlers for different jobs, and the robots.txt tokens are different for each. Mixing them up is how people accidentally disappear from AI search. Here are the ones that matter for creators:
- GPTBot (OpenAI) — crawls the web to train OpenAI's models. Blocking this token opts you out of future OpenAI training data. It does not affect whether ChatGPT's search feature can find and cite you.
- OAI-SearchBot (OpenAI) — builds the index behind ChatGPT search. If you block this one, ChatGPT cannot cite your pages. Leave it alone unless you want out of ChatGPT answers entirely.
- ChatGPT-User (OpenAI) — fetches a page when a ChatGPT user explicitly pastes your link and asks about it. Blocking it only stops that on-demand fetch.
- ClaudeBot (Anthropic) — crawls for training Anthropic's models. The training opt-out token.
- Claude-SearchBot (Anthropic) — powers citations in Claude's search features. Keep it allowed if you want Claude to cite you.
- Google-Extended (Google) — a control token, not a crawler. It tells Google whether your content may be used to train Gemini and Vertex AI models. Blocking it costs you nothing in search.
- CCBot (Common Crawl) — builds a public web corpus that feeds many third-party training datasets. Blocking it cuts off a major indirect route into training data.
- Bytespider (ByteDance) — aggressive training crawler. Worth blocking on bandwidth grounds alone.
- meta-externalagent (Meta) — crawls for Meta's AI products and training.
- Applebot-Extended (Apple) — opts you out of Apple Intelligence model training. Note this is different from plain Applebot, which powers Siri and Spotlight — do not block that one.
- PerplexityBot (Perplexity) — builds Perplexity's search index. Blocking it removes you from Perplexity answers.
The pattern: training tokens get blocked, search tokens stay open. Get this backwards and you trade away visibility for no benefit.
Step 1: Write a robots.txt that blocks training but keeps your search traffic
This is the foundation layer. A robots.txt aimed at AI training looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /Notice what is not in the block list: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. Your regular search indexing and your citability in AI answers stay intact.
Two critical warnings. First, never put Disallow: / under User-agent: * as a shortcut — that tells every polite crawler on the internet, including Googlebot and every AI search bot, to skip your entire site. Second, a named user-agent group replaces the * group for that crawler; it does not add to it. So if your * group disallows /wp-admin/ but your GPTBot group only says Disallow: /, that is fine — but if you allow a bot in its own group, remember it will ignore the * rules entirely.
Generating this by hand is error-prone. Use the robots.txt generator to build the file with the AI crawler rules included, then verify it behaves the way you expect with the robots.txt tester before you upload it.
The Google-Extended myth, corrected
You will see blog posts claiming that blocking Google-Extended removes you from Google's AI Overviews. That is false. Google's AI Overviews and AI Mode are served from the regular Search index built by Googlebot. Google-Extended only controls whether your content is used to train Gemini models and ground Vertex AI. Blocking Google-Extended is the one training opt-out with zero visibility downside — but it does nothing about AI Overviews. If you want out of AI Overviews, you would have to block Googlebot itself, which also removes you from Google Search entirely. Almost no creator should do that.
Step 2: Turn on one-click AI bot blocking in Cloudflare
Robots.txt only works on crawlers that read it. Cloudflare gives you enforcement at the network edge, and the basic version is free.
In your Cloudflare dashboard, under Security → Bots, there is a one-click "Block AI bots" toggle that Cloudflare rolled out to all customers. It blocks known AI training crawlers and scrapers before they reach your server, based on fingerprints Cloudflare maintains — so it catches bots that would have ignored your robots.txt anyway. Pair it with Bot Fight Mode (free plan) for broader automated-bot mitigation.
This is genuinely the highest-effort-to-impact step in this guide. It takes about thirty seconds and covers the non-compliant crawlers that robots.txt cannot touch. Check your Cloudflare analytics afterward under Security → Events to see what was actually being blocked — most bloggers are surprised by the volume.
Step 3: Fix your copyright notice and terms page
Let us be honest: a copyright notice does not technically stop a bot. Bots do not read footers. What it does is establish clear ownership and a stated policy, which is the foundation of every legal remedy — DMCA takedowns, licensing claims, and platform complaints all start from "I am the rights holder and I did not grant permission."
Practical steps for creators:
- Put a real copyright line in your footer with the current year and your name or business name: © 2026 [Your Name]. All rights reserved.
- Add a Terms of Use page (not just a privacy policy) with an explicit clause: automated scraping, crawling for AI training, and republication of your content without written permission are prohibited. Keep the language plain.
- Register your most valuable works with the US Copyright Office if you are US-based. Registration is what unlocks statutory damages in a lawsuit — without it, enforcement against serious infringers is much weaker. It costs a small fee per batch of works.
- If you license content (photos, guest posts), keep written records of what you own versus what you merely have permission to use. You can only enforce rights you actually hold.
This layer will not stop a single scraper today. It is what makes you dangerous to scrapers tomorrow.
Step 4: Set up llms.txt correctly — it is not a permissions file
There is a lot of confusion about llms.txt, so here is the correction: llms.txt has no Permissions section and is not a legal opt-out mechanism. It is a proposed standard by Jeremy Howard (v2 of the proposal landed in August 2026) that gives AI assistants a clean markdown summary of your site — an H1 title, a blockquote describing the site, and H2 sections of link lists pointing to your most important pages.
Think of llms.txt as a sitemap for AI assistants, not a fence. Its job is to help AI tools find and accurately summarize your best content. If your goal is to block AI use, llms.txt is the wrong tool — that is what robots.txt and Cloudflare are for. If your goal is to be cited accurately by AI assistants, a good llms.txt helps.
Many creators want both: block training crawlers, but make sure AI answers cite their site correctly when they do appear. That combination — robots.txt blocking training tokens plus an llms.txt pointing at your cornerstone content — is the sensible setup. You can generate a spec-shaped llms.txt with the llms.txt generator.
Step 5: Protect images and premium content specifically
Text gets the most attention, but for many creators the images are the asset — photographers, recipe bloggers, designers, and tutorial sites live and die by their visuals.
- Watermark visibly on your most-stolen images. Invisible metadata gets stripped by every social platform and most scrapers. A small, tasteful watermark in a corner will not stop theft, but it makes stolen copies advertise you.
- Serve appropriately sized images. There is no reason to serve a 4000-pixel original when the display size is 800 pixels. Smaller files are less useful to thieves and faster for your readers.
- Do not rely on "disable right-click" plugins. They annoy legitimate visitors and stop nobody — any image rendered in a browser can be saved from dev tools or a screenshot.
- Consider hotlink protection so other sites cannot embed your images on their pages at your bandwidth expense. Most hosts and Cloudflare support this.
- For premium content, use real access control. If an article, course, or dataset is genuinely valuable, robots.txt will not protect it — put it behind membership, login, or metered access. Authentication is the only technical measure scrapers cannot politely bypass.
Step 6: Monitor who is actually scraping you
You cannot manage what you do not measure. Two practical ways to see your AI crawler traffic:
- Server access logs: search for the user-agent tokens listed above (GPTBot, ClaudeBot, CCBot, Bytespider). Look at frequency and volume — a training crawler hitting you thousands of times a day is a bandwidth problem as well as a data problem.
- Cloudflare analytics: the Security → Bots section shows automated traffic categorized by type, including AI crawlers specifically. This is the easiest view for non-technical bloggers.
Check quarterly. Crawler behavior changes — new bots appear (DeepSeekBot and Amazonbot are recent additions to the landscape), and a bot you blocked last year may have rebranded its user agent.
When blocking is the wrong move
Not every creator should block everything. Be deliberate about the trade-off:
- If your business depends on being discovered — affiliate blogs, new publications, personal brands — AI answer citations are free, high-intent exposure. Block training crawlers, but keep every search and retrieval bot open.
- If you sell the content itself — courses, stock photos, premium newsletters, proprietary data — lean harder into access control and enforcement, because your content is the product.
- Never block Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, or PerplexityBot "just to be safe." That is not safety; that is invisibility.
If your content was already copied
For content already scraped or republished without permission, the remedy is legal, not technical:
- DMCA takedown: if your work appears on a website, search engine, or platform without permission, file a DMCA takedown notice with the site's host and with Google Search. Most large platforms process these in days.
- Document everything: screenshots with dates, URLs, and your original publication timestamps. Your CMS publish dates and the Wayback Machine are your evidence.
- For AI model outputs: if an AI product reproduces your work verbatim at scale, document the outputs and talk to an attorney who handles copyright — this area of law is actively being litigated, and the rules are still being written.
FAQ
Can I remove my content from an AI model that already trained on it?
Realistically, no. Blocking crawlers is forward-looking: it stops compliant bots from fetching your pages from now on. It does not un-train models or delete copies already in datasets. Your remedies for existing copies are DMCA takedowns against republication and, for large-scale cases, legal action.
Does blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot is the training crawler; ChatGPT's search answers come from a separate index built by OAI-SearchBot, plus on-demand fetches by ChatGPT-User. Blocking GPTBot while allowing OAI-SearchBot is exactly how you opt out of training while staying citable in ChatGPT.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews are generated from Google's regular Search index, which is built by Googlebot. Google-Extended only controls use of your content for training Gemini models. Blocking it has no effect on AI Overviews, Search rankings, or indexing.
Will a copyright notice or llms.txt file stop scrapers?
No. A copyright notice establishes ownership for legal enforcement but no bot obeys it. And llms.txt is not a permissions file at all — it is a proposed standard (H1, blockquote summary, H2 link lists) for helping AI assistants navigate your site. Blocking requires robots.txt rules plus edge enforcement like Cloudflare's AI bot blocking.
Should a small blogger bother with all of this?
The two highest-value steps take under an hour total: a proper robots.txt and Cloudflare's one-click AI bot blocking. Do those regardless of your size. The legal layers — terms pages, copyright registration — matter more as your content library and revenue grow.
What is the single biggest mistake creators make here?
Blocking too broadly. A Disallow: / under User-agent: *, or blocking Googlebot "to stop AI," removes you from search engines and AI answers at the same time. Always block specific training tokens and keep search and retrieval crawlers allowed.
Your action checklist
- Audit which AI bots hit your site (server logs or Cloudflare analytics)
- Deploy a robots.txt that blocks training tokens and keeps search tokens open — generate it with the robots.txt generator and check it with the robots.txt tester
- Enable Cloudflare's one-click "Block AI bots" plus Bot Fight Mode
- Add an explicit anti-scraping clause to your Terms of Use and a proper copyright footer
- Publish a spec-shaped llms.txt pointing AI assistants at your best pages — use the llms.txt generator
- Watermark key images and put premium content behind real access control
For the authoritative details on how the major crawlers identify themselves and what each token controls, read OpenAI's crawler documentation and Anthropic's crawler documentation. Crawler behavior changes over time, so re-check those pages — not random blog posts — when you update your setup.
