Cloudflare lets you block AI bots with a single toggle: go to Security → Bots in your Cloudflare dashboard and switch on "AI Scrapers and Crawlers" — it's available on every plan, including free. Robots.txt is a polite request that only honest bots honor. Cloudflare sits in front of your site and can actually enforce your policy: blocking known AI bots by fingerprint (not just user-agent), challenging suspicious traffic, and rate-limiting aggressive scrapers. Here's how to set it up properly, layer by layer.
Why Cloudflare instead of just robots.txt
Robots.txt works on the honor system. Compliant bots like GPTBot and ClaudeBot read it and obey — but scrapers that spoof user-agents, ignore the file entirely, or crawl from rotating IPs sail right past it. Cloudflare sees every request before it reaches your server, so it can identify bots by behavior and fingerprint, not just by what they claim to be in their user-agent string.
The scale of the problem is worth understanding. Cloudflare's Radar team has published crawl-to-referral data showing how lopsided the AI crawling economy is: traditional search crawlers return a meaningful share of traffic relative to what they take, while some AI training bots crawl thousands of times for every single visitor they refer. When bots consume your bandwidth and content without sending readers back, enforcement stops being optional.
Layer 1: The one-click AI bot blocker (free)
In 2024 Cloudflare added a dedicated toggle for exactly this problem, and their announcement made clear it's available to all customers including the free tier:
- Log in at dash.cloudflare.com and select your site.
- Go to Security → Bots.
- Find "AI Scrapers and Crawlers" and toggle it on.
That's it. Cloudflare maintains the list of known AI bots — training crawlers from the major AI companies and others — and updates it as new bots emerge. It uses fingerprinting and behavioral analysis, so it catches bots even when they lie about their user-agent. Legitimate search engines like Googlebot and Bingbot are not affected; Cloudflare distinguishes them from AI scrapers.
Important: your DNS records must be proxied through Cloudflare (the orange cloud icon) for any of this to work. If your site isn't behind Cloudflare's proxy, the dashboard settings do nothing.
Layer 2: Bot Fight Mode
On the same Security → Bots page you'll find Bot Fight Mode (free plan) — a broader defense that challenges traffic exhibiting machine-like behavior. It won't specifically target AI bots the way the AI toggle does, but it catches the undeclared scrapers: the no-name crawlers harvesting content for smaller AI projects, the ones that never bothered to give themselves an honest user-agent.
Enable both. The AI toggle handles the known AI bots precisely; Bot Fight Mode mops up the suspicious remainder. If you run a site where every millisecond of latency matters (say, a game portal or a real-time tool), test Bot Fight Mode first — the challenges it issues can occasionally affect unusual-but-legitimate traffic patterns.
Layer 3: Custom WAF rules for specific bots
The toggle covers known bots, but sometimes you want surgical control — blocking one specific crawler while allowing another, or blocking a bot only from certain sections of your site. Cloudflare's WAF custom rules (available on free plans with some limits) let you match on the user-agent string:
- Go to Security → WAF → Custom rules and create a rule.
- Use the expression builder:
http.user_agent contains "GPTBot"— combine multiple withor. - Set the action to Block (or Managed Challenge if you'd rather test the waters).
- Deploy, then check Security → Events to confirm the rule is firing.
A practical example — block training bots but allow AI search bots site-wide, while keeping your API endpoints bot-free:
(http.user_agent contains "GPTBot"
or http.user_agent contains "ClaudeBot"
or http.user_agent contains "CCBot"
or http.user_agent contains "Bytespider")
or (http.request.uri.path contains "/api/")Two cautions. First, match on the bot token (GPTBot), not the full user-agent string — version numbers change and full-string matches break silently. Second, anyone can claim to be GPTBot; user-agent rules alone can't verify identity. For verification against spoofing, cross-check suspicious IPs against the vendor's published IP ranges in your server logs.
Layer 4: Rate limiting
Some scrapers don't identify as bots at all — they just request pages at inhuman speed. Rate limiting caps how many requests an IP can make in a window:
- Go to Security → WAF → Rate limiting rules.
- Create a rule for your content paths — for example,
/blog/*or/articles/*. - Set a threshold real users won't hit, like 30 requests per minute per IP, with a Block or Challenge action for a cooldown period.
Legitimate readers almost never trip these limits; scrapers always do. Start generous and tighten gradually — an overly aggressive limit that blocks real users costs more than the bots it stops. Note that rate limiting rules have usage quotas on the free plan, so apply them to your most-scraped paths first.
Layer 5: See what's actually crawling you
Cloudflare's analytics show you bot traffic broken down by type, so you're not guessing. Check Analytics for automated-traffic breakdowns, and Security → Events to see exactly which rules fired against which requests. Cloudflare also offers an AI Audit-style view distinguishing AI bot categories — useful when you want to allow AI search bots (which cite you) while blocking pure training scrapers.
Do this audit before you block aggressively: you might discover that a bot you were about to block is actually a monitoring service you pay for, or that most of your "AI bot problem" is one misconfigured scraper from a single network.
Combining Cloudflare with robots.txt
These layers complement robots.txt rather than replacing it. Keep your robots.txt accurate for the compliant bots — it's still the documented, standard way to declare policy, and it creates your paper trail of non-consent. Think of it this way:
- robots.txt = your public declaration. Honest bots (GPTBot, ClaudeBot) obey it. Generate a clean one with a robots.txt generator and verify it with a robots.txt tester.
- Cloudflare = your enforcement. It stops the bots that ignore the declaration, identified by fingerprint and behavior rather than self-reported user-agents.
One gotcha: if you write WAF rules that block bots from fetching your /robots.txt itself, compliant bots can't read your policy — make sure your rules allow requests to /robots.txt and your sitemap.
What Cloudflare can't do
Stay realistic. Cloudflare can't stop user-triggered fetches — when a person asks ChatGPT or Claude to read your page, that request looks like legitimate traffic and should be treated that way. It can't stop someone from copying your content manually or through a residential proxy network that mimics human behavior perfectly. And aggressive bot blocking always carries a small risk of false positives — monitor your Security Events log weekly, especially after changing rules, and whitelist anything legitimate you catch.
FAQ
Is the AI bot blocker really free?
Yes. Cloudflare's "AI Scrapers and Crawlers" toggle under Security → Bots is available on all plans including the free tier. Some advanced features — deeper bot management scoring, higher rate-limiting quotas — are paid, but the core AI blocking costs nothing.
Will blocking AI bots hurt my Google rankings?
No. Cloudflare's AI blocking distinguishes AI scrapers from search engine crawlers — Googlebot and Bingbot are unaffected. Just don't write your own custom rules so broadly that they catch legitimate crawlers; test every rule in the Events log first.
Does the toggle block ChatGPT from citing my site?
It blocks AI bots Cloudflare identifies as scrapers/crawlers. User-triggered fetches — a person asking an AI assistant to read your specific page — are a different category and generally aren't blocked, since they represent real user interest, similar to a browser visit.
Do I still need robots.txt if I use Cloudflare?
Yes. Robots.txt remains the standard, documented declaration of your crawling policy, and compliant bots honor it without needing enforcement. Cloudflare handles the non-compliant remainder. Also make sure your Cloudflare rules don't block bots from reading robots.txt itself.
How do I know the rules are working?
Check Security → Events in your Cloudflare dashboard — every blocked or challenged request appears there with the rule that fired. Compare your origin server logs before and after enabling the toggle; AI bot traffic should drop sharply within days.
Can bots get around Cloudflare's blocking?
Determined scrapers can rotate IPs, mimic browsers, and use residential proxies — no defense is perfect. But Cloudflare's fingerprinting catches far more than user-agent matching alone, and each additional layer (rate limiting, challenges) raises the cost of scraping your site until most scrapers move on to easier targets.
The takeaway
Start with the one-click toggle — it takes ten seconds and it's free. Add Bot Fight Mode for the undeclared scrapers, write custom WAF rules only where you need surgical precision, and put rate limits on your most-scraped content paths. Keep robots.txt as your public declaration underneath it all, and check your Security Events log weekly. That's a complete, layered AI bot defense most sites can set up in under an hour.
