There are three main ways to block AI bots: robots.txt, .htaccess rules, and Cloudflare. Robots.txt asks bots to stay away. .htaccess refuses them at your server. Cloudflare stops them at the network edge before they ever reach you. The right choice depends on what you are protecting and who you are protecting it from — and for most sites, the best answer is a combination, not a single method.
This guide compares all three on the dimensions that actually matter: whether they enforce or merely request, which bots they stop, how hard they are to set up, what they cost, and what can go wrong. By the end, you will know which layer (or layers) your site needs.
The Three Methods at a Glance
- Robots.txt — a text file at your site root that tells crawlers which paths they may fetch. Voluntary. Free. Every site should have one.
- .htaccess — server configuration rules (on Apache and LiteSpeed servers) that match a request's user-agent string and return a 403 Forbidden before the page is served. Enforced at your server. Free. Powerful, but a bad rule can break your site.
- Cloudflare — a toggle in the Cloudflare dashboard ("AI Scrapers and Crawlers") plus optional WAF rules, enforced at Cloudflare's edge network. Free tier available. Catches bots robots.txt cannot, including some that fake their user agent.
The short version: robots.txt is your policy, .htaccess is your bouncer, and Cloudflare is your perimeter fence. A policy alone does not stop anyone who ignores it. A bouncer alone is useless if nobody told the guest list. The fence alone cannot express the fine print. Layer them, and each one covers the others' gaps.
Method 1: Robots.txt — The Policy Layer
Robots.txt is the oldest and most universal crawler control. It is a plain text file served at yourdomain.com/robots.txt, written in a standard format defined by RFC 9309. A typical AI-blocking block looks like this:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /
What it is good at
- It is the official opt-out. Every major AI provider — OpenAI, Google, Anthropic, Meta, Perplexity, Common Crawl — documents robots.txt tokens for opting out of training or crawling. When you block GPTBot in robots.txt, you are using OpenAI's own published mechanism.
- It is free and universal. Every crawler knows how to read it. It works on every platform and every hosting setup.
- It is precise. You can block one bot, allow another, block a bot from one folder, or allow it everywhere except one path. No other method expresses per-bot, per-path policy as cleanly.
- It costs nothing in performance. Serving a small text file is trivial. It adds no load to your server.
Where it falls short
- It is voluntary. Robots.txt is a request, not a lock. Honest crawlers obey it. Scrapers that ignore it will fetch your pages anyway, and robots.txt gives you no way to stop them.
- It cannot identify liars. A scraper can claim to be a normal browser, or claim to be Googlebot, and robots.txt has no way to check.
- It has limits. Google stops reading a robots.txt file larger than 500 KiB, so an enormous block list can silently truncate. And
Crawl-delay— a line many sites still write — is not part of the standard: Google ignores it entirely, and support among other crawlers is inconsistent. - It does not remove past collection. Robots.txt controls future crawling only. It cannot pull your content out of a training set that already has it.
Verdict: every site needs a robots.txt AI policy, because it is the mechanism providers officially honor. But on its own, it only controls the honest bots. If you need enforcement, keep reading.
Method 2: .htaccess — The Server Layer
.htaccess is a configuration file used by Apache (and LiteSpeed) web servers. Unlike robots.txt, it does not ask — it acts. A rule can inspect each incoming request's user-agent string and return a 403 Forbidden before your page is ever served:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC]
RewriteRule .* - [F,L]What it is good at
- It enforces. A matched request gets a 403. The bot never sees your content. That is a real difference from robots.txt, which relies on the bot's goodwill.
- It stops dishonest bots too — as long as they identify themselves honestly. A scraper that uses the GPTBot user-agent string but ignores robots.txt will still be blocked by a .htaccess rule matching that string.
- It is free and immediate. No new service, no DNS change. Save the file and the rule is live.
- It can protect more than AI bots. The same mechanism blocks spam bots, scrapers, and bad actors of every kind.
Where it falls short
- It only works on Apache and LiteSpeed. If your server runs Nginx, .htaccess does nothing — Nginx uses a different configuration system, and the equivalent rules go in the server block instead.
- A bad rule can break your site. An overly broad pattern can block real visitors, search engines, or your own monitoring tools. Always keep a backup of the working .htaccess file, test after every change, and make sure you can restore it quickly. One typo in .htaccess can return a 500 error for your entire site.
- It cannot catch user-agent spoofing. If a scraper pretends to be Chrome, your .htaccess rule sees "Chrome" and waves it through. User-agent matching is only as reliable as the user agent itself.
- It runs on every request. The overhead is small, but on a high-traffic site, a long list of rewrite conditions adds up. Keep the list focused on the bots you actually need to block.
- It is invisible to providers. Unlike robots.txt, a .htaccess block is not an official opt-out signal. A provider's crawler that gets a 403 may retry, and the block says nothing about your policy to anyone reading your site.
Verdict: .htaccess is the right enforcement layer for Apache/LiteSpeed sites that need to stop bots that ignore robots.txt. Use it alongside robots.txt, not instead of it — the policy and the enforcement should agree. Our .htaccess blocking guide has tested rules, and the htaccess generator builds them for you.
Method 3: Cloudflare — The Edge Layer
Cloudflare sits between your visitors and your server. Because it sees every request before your server does, it can block bots at the network edge — the request never reaches your site at all. Its main control is a single toggle: in the Cloudflare dashboard, go to Security > Bots and turn on AI Scrapers and Crawlers. Cloudflare announced this one-click block as available to all customers, including the free tier, and says it updates the underlying signatures automatically as new AI bot fingerprints appear.
What it is good at
- It enforces, at the edge. Blocked requests never touch your server. That saves bandwidth and server load — a real benefit when bot traffic is heavy.
- It catches bots robots.txt cannot. Cloudflare uses behavioral signals and fingerprinting, not just the user-agent string. That is how it can block scrapers that spoof a browser user agent — something neither robots.txt nor .htaccess can do.
- It stays current without your help. New AI bots appear constantly. Cloudflare updates its managed signatures for you, so your protection does not rot the way a hand-maintained block list does.
- WAF rules give per-bot control. Beyond the toggle, Cloudflare's WAF lets you write custom rules for individual user agents — block this bot, allow that one, challenge anything suspicious. That is the granularity a bulk toggle cannot provide.
- It is free to start. The AI bot toggle is available on Cloudflare's free plan.
Where it falls short
- It requires routing your DNS through Cloudflare. That is a real change: you point your domain's nameservers at Cloudflare, and all traffic flows through its network. For most sites this is straightforward, but it is a bigger step than editing a text file.
- It is a bulk control by default. The toggle blocks AI bots as a category. If you want "block GPTBot but allow PerplexityBot," you need custom WAF rules, which take more setup.
- It is a third-party dependency. Your bot protection now depends on Cloudflare's service and its classification decisions. If Cloudflare has an outage or misclassifies a bot, your rules inherit that.
- It does not replace the policy. Cloudflare's toggle is not a published opt-out in the way robots.txt is. Keep your robots.txt policy in place so providers see your stated position.
Verdict: Cloudflare is the strongest single enforcement layer, and the best answer for sites on platforms that do not allow robots.txt editing (like Squarespace) or server config (like most hosted platforms). Read our Cloudflare AI bot blocking guide for the full setup.
Head-to-Head Comparison
Here is how the three methods stack up on the questions site owners actually ask:
- Stops honest bots: all three work. An honest bot obeys robots.txt, gets a 403 from .htaccess, and is blocked by Cloudflare.
- Stops bots that ignore robots.txt: .htaccess and Cloudflare do; robots.txt does not.
- Stops bots that fake their user agent: only Cloudflare has a real chance, through behavioral fingerprinting. .htaccess and robots.txt both rely on the user agent being truthful.
- Per-bot, per-path precision: robots.txt is best. .htaccess can match per-bot but per-path rules get complicated. Cloudflare's toggle is bulk; per-bot needs WAF rules.
- Official opt-out signal: only robots.txt. Providers document robots.txt tokens as the way to opt out.
- Setup effort: robots.txt is easiest, .htaccess is moderate (and risky if you are careless), Cloudflare requires a DNS change.
- Cost: all three are free at the basic level.
- Server load: Cloudflare reduces it (blocked requests never arrive). .htaccess adds a small per-request check. Robots.txt adds nothing.
Which Combination Should You Use?
- Personal blog or small site: robots.txt plus the Cloudflare toggle. Two free steps, five minutes each, and you have both the policy and enforcement. You do not need .htaccess unless you see bots ignoring robots.txt in your logs.
- Publisher worried about training data: all three. Robots.txt states your opt-out, .htaccess enforces it at the server, Cloudflare catches spoofers at the edge. Content is your product — protect it in depth.
- E-commerce or high-traffic site: robots.txt plus Cloudflare. Server load matters to you, and Cloudflare keeps bot requests off your infrastructure entirely. Add .htaccess rules only for specific bots you see in logs.
- Squarespace, Wix, or another hosted platform: you cannot edit robots.txt freely and you have no server config, so use the platform's built-in toggle plus Cloudflare. Cloudflare is doing the heavy lifting here.
- You want per-bot granularity: robots.txt for the policy, plus Cloudflare WAF rules (or .htaccess rules on Apache) for enforcement. That is the only combination that lets you say "block this bot, allow that one" and make it stick.
Setup Checklist
- Write your robots.txt policy. Decide which bots to block and which to allow, then generate the file with the robots.txt generator and test it with the robots.txt tester before publishing. See our guide to testing robots.txt if you are unsure.
- Add enforcement if you need it. On Apache/LiteSpeed, add .htaccess rules matching the same bot list. Or enable Cloudflare's AI Scrapers and Crawlers toggle.
- Check for conflicts. Your robots.txt, .htaccess, and Cloudflare rules should tell the same story. A bot allowed in robots.txt but blocked in .htaccess creates confusion and makes debugging painful.
- Verify in logs. After a week, check server logs (or Cloudflare analytics) for the blocked tokens. If visits continue, look for a typo, a conflicting rule, or a scraper spoofing a user agent.
- Review quarterly. New AI crawlers appear every few months. Revisit your list, check for new tokens in your logs, and confirm your Cloudflare signatures are still enabled.
FAQ
Is robots.txt enough to block AI bots?
It is enough for honest bots from major providers, because they officially honor robots.txt. It is not enough for scrapers that ignore robots.txt or fake their user agent. If enforcement matters to you, add .htaccess or Cloudflare.
Does .htaccess work on Nginx?
No. .htaccess is an Apache/LiteSpeed feature. On Nginx, equivalent user-agent blocking goes in the server block configuration. If you are on Nginx and cannot edit server config, Cloudflare is the easier enforcement layer.
Does Cloudflare's AI bot toggle block Googlebot?
No. The toggle targets AI scrapers and crawlers, not search engines. Googlebot and Bingbot keep crawling and indexing normally. Cloudflare distinguishes AI bots from search crawlers.
Can I use all three methods at once?
Yes — and for content businesses, you should. Robots.txt states the policy, .htaccess enforces it at the server, and Cloudflare enforces it at the edge. Just make sure all three agree on which bots are blocked and which are allowed.
Will blocking AI bots hurt my SEO?
No, as long as you block AI tokens only — GPTBot, ClaudeBot, CCBot, and similar — and leave Googlebot and Bingbot alone. AI training crawlers are separate from search crawlers. Blocking them does not affect your search rankings.
What about Crawl-delay?
Crawl-delay is not part of the robots.txt standard (RFC 9309). Google ignores it entirely. Some other crawlers honor it, but support is inconsistent, so do not rely on it to control crawl rate. If a bot is hammering your server, block it or rate-limit it at the server or edge instead.
Bottom line: robots.txt, .htaccess, and Cloudflare are not rivals — they are layers. Start with robots.txt, because it is the official opt-out every provider honors. Add .htaccess if you run Apache and need server-level enforcement. Add Cloudflare if you want edge enforcement, spoof detection, and automatic updates. One layer is a request. Two layers are a policy with teeth. Three layers are a system.
