Want your site out of Common Crawl? Here is the short answer. To stop future crawls, add User-agent: CCBot and Disallow: / to your robots.txt — it takes about two minutes and works reliably. But if your content is already sitting in Common Crawl's archive, the open web dataset that has been widely used as AI training data, robots.txt will not erase those old copies. Removing archived content requires a direct opt-out and removal request to Common Crawl itself, and even then it is not instant or guaranteed. This guide walks you through both: blocking CCBot going forward, and how to get already-archived content removed.
What Is Common Crawl, and Why AI Builders Love Its Archive
Common Crawl is a non-profit foundation that maintains an open repository of web crawl data. It sends out a crawler on a regular schedule, copies the pages it finds, and stores the results as massive public archives that anyone can download and use. For years it has been one of the largest freely available sources of web text on the internet.
The reason you are reading about it now is artificial intelligence. Large language models need enormous amounts of text to train on, and building a crawl of the whole web from scratch is expensive and slow. Common Crawl's archive solved that problem for many AI teams at once: instead of crawling the web themselves, researchers and companies could start from a shared, ready-made snapshot. The archive has been widely used as AI training data — so widely that in The New York Times' copyright suit against OpenAI, Common Crawl was cited as making up a large share of GPT-3's training mix.
That is the tension behind every opt-out request. For researchers and open-source AI projects, the archive is a public good: free, enormous, and not locked behind any company's doors. For site owners and publishers, the same archive can feel like their work being harvested without consent. Common Crawl itself is a non-profit research organization, not an AI company, which makes the situation more complicated than a simple fight between publishers and tech giants. Its mission is to keep an open copy of the web for research — but that mission directly collides with publishers who want control over where their content ends up.
Whatever side you land on, the mechanics of opting out are the same. You cannot control who downloaded the archive in the past, but you can control whether Common Crawl keeps crawling you, and you can ask for your content to be taken out of the parts of the archive it directly controls.
What Is CCBot?
CCBot is the web crawler Common Crawl uses to collect the archive. When it visits your site, it identifies itself honestly: the user-agent string contains "CCBot", which is exactly what you match on when you want to block it. The crawler runs on dedicated IP ranges with reverse DNS hostnames under the crawl.commoncrawl.org domain — their documentation shows an example like 18-97-14-84.crawl.commoncrawl.org — so a real CCBot visit is verifiable in your server logs (more on that below).
One thing worth knowing: CCBot is polite by crawler standards. It respects robots.txt, and Common Crawl's official documentation even lists support for the Crawl-delay directive, so you can slow it down rather than banning it outright. That means the simplest blocking method is also the one Common Crawl itself recommends: tell CCBot to stay away in robots.txt, and it will.
How to Block CCBot With robots.txt
This is the fastest, most reliable step. Common Crawl documents these exact rules in its official FAQ, and CCBot honors them.
Option 1: Block CCBot Completely
If you want no Common Crawl crawling at all, add this block to your robots.txt:
User-agent: CCBot
Disallow: /That's it. Disallow: / tells CCBot it may not fetch anything on your site. Keep the rule specific to CCBot — do not put it under User-agent: *, or you will accidentally block every crawler that respects robots.txt, including search engines. If your robots.txt already has rules for other bots, you can add the CCBot block anywhere in the file; a crawler only reads the rules under its own user-agent (plus the wildcard block, if any).
Option 2: Slow It Down With Crawl-Delay
Maybe you don't mind being in the archive, but CCBot is hitting your server too often and you want it to back off. Common Crawl's FAQ documents Crawl-delay support — for example:
User-agent: CCBot
Crawl-delay: 2This asks CCBot to wait at least 2 seconds between requests. It is a compromise: your content stays available in the archive, but the crawl is gentler on your server. Note that Crawl-delay is not an official standard that every bot supports, but Common Crawl explicitly lists it, so it works for this specific crawler.
Option 3: Block Only Part of Your Site
You can also be selective. For example, to keep CCBot out of a members' area or a directory with sensitive content while leaving the rest of the site open:
User-agent: CCBot
Disallow: /members/
Disallow: /private/Just remember: this only prevents future crawls of those paths. Pages CCBot already fetched before you added the rule are already in the archive, and robots.txt does nothing about those. For old copies, you need the removal process described below.
Test Your Rule Before You Trust It
One typo in robots.txt can accidentally block Googlebot or leave the door wide open. After editing the file, run it through a robots.txt tester to confirm that CCBot is disallowed and everything else is untouched. The fastest way to do this without mistakes is to generate the rule and verify it:
- Generate the CCBot block rule with the free robots.txt generator — pick the bot from the list and it writes the syntax for you.
- Verify it with the robots.txt tester — enter your URL and confirm CCBot gets a "disallowed" while Googlebot stays allowed.
Also double-check that the file is served at https://yourdomain.com/robots.txt (not behind a login or redirect chain) — if crawlers can't fetch it, they can't read your rules.
Join the Common Crawl Opt-Out Registry
Beyond robots.txt, Common Crawl publishes a public opt-out registry — a list of domains whose owners have asked not to be crawled. Being on this registry is a stronger, more explicit signal than robots.txt alone, and it puts your site on the same list as some of the biggest names in publishing: the BBC, The Guardian, The New York Times, The Washington Post, News Corp, The Atlantic, and others are on it.
To join, you submit an opt-out request to Common Crawl through its official opt-out process. The details of how to submit are on Common Crawl's own site — the canonical starting points are its official CCBot page and the official FAQ on blocking CCBot, which lay out both the robots.txt method and the formal opt-out route. Make the request from a domain-associated email address and be specific about the exact domains and subdomains you want excluded.
Pressure on this process has been growing. In 2026, US publisher groups including the News/Media Alliance (NMA) and Digital Content Next (DCN) sent formal letters to Common Crawl demanding removal of archived content and a clearer, more usable opt-out process. That pressure is exactly why the formal request route matters: publishers are treating it as the official channel, and having your request on record there is the strongest paper trail you can get.
How to Request Removal of Already-Crawled Content
This is the part most guides gloss over, so let's be precise about what happens and what doesn't.
When you ask Common Crawl to remove your content, their stated policy is to remove or filter the affected URLs from subsequent crawls and to make those URLs inaccessible through their public tools and index — in other words, they pull your pages out of the searchable index and the download tools going forward. What they do not promise is surgical deletion of every old copy. Their archive file formats are designed to be immutable — once a monthly crawl snapshot is written and distributed, going back and editing petabytes of historical files is technically complex. Publishers have disputed whether removals were complete, and Common Crawl itself acknowledges the difficulty with old archive copies.
Here is the honest practical takeaway:
- robots.txt stops future crawls. This works and is fully in your control. It is also the only part that is instant.
- A direct opt-out request to Common Crawl is required for old archive copies. Use the formal opt-out channel described above, and state clearly that you want removal of already-crawled content, not just exclusion from future crawls.
- Expect it to take time, and don't expect a guarantee. Removal is processed by a small non-profit team, not automated in real time, and the immutable nature of old archives means some historical copies may persist in copies that were already downloaded by third parties. Anyone who already downloaded the archive has their own copy, and Common Crawl cannot recall those.
One more reality check: opting out of Common Crawl does not remove your content from AI models that were already trained on it. Training datasets are snapshots; a model trained in 2024 on a 2023 Common Crawl snapshot does not "forget" your pages because you opted out in 2026. The opt-out protects you against future models trained on future snapshots.
Is That CCBot Hit in Your Logs Real? Verifying Against Impostors
Common Crawl warns that impostor crawlers use the CCBot name. Anyone can set their user-agent string to anything, so a log line saying "CCBot" is a claim, not proof. Before you build firewall rules, allowlists, or analytics around it, verify the visitor. Here is the standard two-step check:
- Reverse DNS lookup on the visitor's IP. Take the IP from your server log and run a reverse lookup:
dig -x 18.97.14.84 +short(Linux) ornslookup 18.97.14.84(Windows). A genuine CCBot IP should resolve to a hostname ending in.crawl.commoncrawl.org, like the example 18-97-14-84.crawl.commoncrawl.org in Common Crawl's own docs. - Forward-lookup confirmation. Take that hostname and resolve it back to an IP:
dig 18-97-14-84.crawl.commoncrawl.org +short. If the result matches the original visitor IP, you have a genuine CCBot. If the reverse lookup returns no hostname, a generic hosting hostname, or a different domain entirely, it's an impostor — treat it like any other unidentified crawler.
This matters for firewall rules. If you ever whitelist or block by IP for CCBot, only use ranges you've verified this way. Hard-coding IP ranges from memory is risky because ranges can change; the DNS verification method stays correct as long as Common Crawl keeps its naming convention. For most site owners, though, the robots.txt block is enough — verified CCBot respects it, and impostors aren't going to respect anything you write in a file.
Will Blocking CCBot Hurt Your SEO?
No. This is one of the most common worries, and the answer is simple: CCBot has nothing to do with search indexing.
Google uses Googlebot, Bing uses Bingbot, and each crawler only reads the robots.txt rules written under its own user-agent name (or the wildcard User-agent: * block). A User-agent: CCBot + Disallow: / block is invisible to search crawlers. Your rankings, indexing, and search traffic are completely unaffected — as long as you don't accidentally put the disallow under the wildcard.
The only real trade-off is discoverability outside of search. Some research tools, datasets, and analytics products are built on Common Crawl data, so blocking CCBot means your site won't appear in those. For most commercial websites, that is no loss at all. For sites that want maximum open-data visibility — academic projects, open-source documentation — the Crawl-delay compromise from the section above is the better choice.
Cloudflare's own data has shown CCBot among the bots most commonly given full Disallow rules on top domains, so if you block it, you are in very large company — and none of those sites blocked it for SEO reasons, because there is no SEO downside.
The Publisher Dispute: Why Big News Sites Want Out
The opt-out trend isn't just individual site owners. Major publishers have been exiting Common Crawl at scale — the opt-out registry now reads like a who's who of the news industry, from the BBC and The Guardian to The New York Times, The Washington Post, News Corp, and The Atlantic.
The flashpoint came in 2026, when US publisher groups including the News/Media Alliance and Digital Content Next sent formal letters to Common Crawl demanding removal of archived content and a clearer opt-out process. The underlying argument is about consent and value: publishers say their reporting is being used to train commercial AI products without permission or payment, and they want a reliable way to say no — and to have "no" apply to the archive, not just future crawls.
Common Crawl's position, for its part, is that it is a non-profit research archive, not an AI vendor, and that open web data has independent research value. That disagreement is unresolved, and it is bigger than any one robots.txt file. But whatever the outcome of that fight, the individual tools stay the same: robots.txt and the opt-out registry are what site owners can act on today.
Common Crawl Opt-Out FAQ
Does adding CCBot to robots.txt delete my content from Common Crawl's existing archive?
No. robots.txt only controls future crawling — Common Crawl's crawler will stop fetching your pages from that point on. Content that was already crawled stays in the existing archive snapshots. To address old copies, you need to submit a direct opt-out and removal request to Common Crawl, as described above.
How long does removal from the archive take?
There is no published SLA. Common Crawl is a small non-profit team, and removal requests are handled manually rather than through an automated self-serve dashboard. Expect weeks rather than hours, and follow up if you don't hear back. The removal applies to Common Crawl's own tools and index; copies of the archive already downloaded by third parties are out of their control.
Does CCBot respect Crawl-delay?
Yes. Common Crawl's official FAQ documents Crawl-delay support — for example, Crawl-delay: 2 limits CCBot to one request every 2 seconds. Keep in mind this is a per-crawler convention rather than a universal standard, so it works for CCBot specifically because Common Crawl says it does.
Should I block all Common Crawl crawlers, or just CCBot?
CCBot is the user-agent Common Crawl documents for its web crawl, so User-agent: CCBot is the correct target. Keep your rules targeted at the documented name rather than guessing at other agent strings — invented user-agent names in your robots.txt do nothing and just clutter the file.
Can I allow CCBot on only part of my site?
Yes. Use path-specific Disallow rules, like Disallow: /members/, instead of Disallow: /. Everything not disallowed remains crawlable. This is a good option if you want open pages in research datasets but private or paywalled sections excluded.
If I opt out now, will my content be removed from AI models already trained on it?
No. Opting out affects future crawls and (after a removal request) Common Crawl's own archive tools — it cannot reach into models that were already trained on older snapshots. The value of opting out is forward-looking: keeping your future content out of future training datasets.
If you want out of Common Crawl, don't wait for the perfect moment. Add the robots.txt rule today — it takes two minutes, it works, and it stops the bleeding. Then submit the formal opt-out request for your archived content, and verify real CCBot visits in your logs so you know the block is holding. For more guides on controlling which bots can touch your site, see the CrawlGuards blog.
