Block AI Crawlers With .htaccess: Copy-Paste Rules (2026)

You can block AI crawlers and bad bots at the server level with a few lines in your .htaccess file. A mod_rewrite rule matching crawler user-agents returns a 403 before the bot ever loads a page — harder enforcement than robots.txt, which relies on bots behaving. This works on Apache and LiteSpeed servers only, never on Nginx or hosted platforms like Shopify. Back up the file first: one syntax error can take your whole site down.

robots.txt is a request. .htaccess is a wall. If you have asked politely in robots.txt and the scrapers kept coming — or you run a site where bandwidth bills and server load matter — server-level blocking is the next step. It stops the request at the web server, returns an HTTP 403 Forbidden, and never lets the bot touch your content, your database, or your analytics.

This guide gives you copy-paste .htaccess rules for the AI crawlers site owners block most (GPTBot, ClaudeBot, CCBot, Bytespider, and friends), explains the two standard methods, and — just as important — tells you which bots to leave alone so you don't accidentally de-index yourself from Google. Generate the matching robots.txt layer with the free robots.txt generator first, then add .htaccess as the enforcement layer on top.

How .htaccess blocking differs from robots.txt

These two tools answer different questions. robots.txt answers "may I?" — it is a voluntary protocol. Well-behaved crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, Applebot) honor it. Bad actors, aggressive scrapers, and many data-harvesting bots ignore it completely. If a bot ignores your robots.txt, the file has done all it can do.

.htaccess answers "can you?" — it is enforced by your own server. When a request arrives, Apache checks the rules before serving anything. A blocked user-agent gets a 403 and nothing else: no page HTML, no images, no database queries. The practical differences:

Enforcement: robots.txt relies on bot cooperation; .htaccess relies on your server. Scrapers that ignore robots.txt still hit the .htaccess wall.
Resource cost: a robots.txt-blocked bot still downloads your robots.txt and may probe pages. A .htaccess-blocked bot gets a tiny 403 response — far less bandwidth and CPU.
Granularity: robots.txt controls crawling politely per path; .htaccess can block by user-agent, IP, referrer, request rate, and more.
Risk: robots.txt mistakes are easy to spot and fix. .htaccess mistakes (a bad regex, a missing flag) can 500-error your entire site instantly.

The right setup for most sites is both: robots.txt as the polite, standards-based declaration (which good bots honor and which documents your intent), and .htaccess as enforcement for the bots you've decided shouldn't be there at all. For the robots.txt side, see the copy-paste robots.txt template for AI crawlers.

Before you touch your .htaccess file

Three non-negotiable precautions:

Back up first. Download the current .htaccess from your site root (public_html or www) via FTP or your host's file manager. Keep the copy on your computer. If anything breaks, restoring is a 30-second upload.
Confirm you're on Apache or LiteSpeed. .htaccess is an Apache-family feature; LiteSpeed honors it too. It does nothing on Nginx, and you cannot use it on Shopify, Wix, Squarespace, or most managed platforms. Ask your host if unsure — getting this wrong means editing a file that silently does nothing, which is its own kind of trap.
Edit with a plain-text editor. Not Word, not Google Docs — something like VS Code, Notepad++, or the file manager's built-in editor. Curly quotes or rich-text formatting will corrupt the file.

Also: changes take effect immediately on save. There is no staging. Test every change (more on testing below) before you walk away.

Method 1: Block by user-agent with mod_rewrite (most common)

This is the method most guides use and the one to reach for first. It checks the request's User-Agent header against a pattern and returns 403 for matches.

# Block AI training crawlers and known bad bots
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|ClaudeBot|Claude-User|CCBot|Bytespider|Applebot|FacebookBot|Meta-ExternalAgent|Amazonbot|Diffbot|SemrushBot|AhrefsBot|MJ12bot|DotBot) [NC]
RewriteRule ^ - [F,L]
</IfModule>

How to read this: RewriteCond tests the user-agent string; [NC] makes the match case-insensitive; the (a|b|c) groups are alternation (match any one); RewriteRule ^ - [F,L] means "for any request (^ matches every path), do nothing (-) and return Forbidden (F), and stop processing rules (L)."

Two judgment calls in that list deserve explanation. ChatGPT-User and Claude-User are live-fetch agents — they visit pages when a user explicitly asks the assistant to look at a URL. Blocking them doesn't stop AI training; it stops the assistants from reading your pages on demand, which can reduce the accuracy of answers about your site. Many publishers block only the training crawlers (GPTBot, ClaudeBot, CCBot) and leave the user agents alone. Decide based on whether you want AI assistants to be able to fetch your content when asked.

SEO tool crawlers (SemrushBot, AhrefsBot, MJ12bot, DotBot) are in the list because they consume significant crawl budget on some sites and exist to feed competitor-analysis tools — but they are not AI trainers. If you use those SEO tools yourself, their crawlers are how your data gets in. Remove any you rely on.

Method 2: Block with SetEnvIf (cleaner on Apache 2.4)

If your host runs Apache 2.4, the mod_setenvif + Require approach is arguably cleaner: it sets an environment flag for matching user-agents and denies them in one place. It's also easier to extend with IP-based rules later.

# Flag bad-bot user agents
SetEnvIfNoCase User-Agent "GPTBot" bad_bot
SetEnvIfNoCase User-Agent "ClaudeBot" bad_bot
SetEnvIfNoCase User-Agent "CCBot" bad_bot
SetEnvIfNoCase User-Agent "Bytespider" bad_bot
SetEnvIfNoCase User-Agent "Applebot" bad_bot
SetEnvIfNoCase User-Agent "facebookexternalhit" bad_bot
SetEnvIfNoCase User-Agent "Meta-ExternalAgent" bad_bot

<RequireAll>
  Require all granted
  Require not env bad_bot
</RequireAll>

One line per bot, which makes the file self-documenting: six months from now you'll know exactly what's blocked and why. To also block by IP — useful for a scraper that rotates user-agents — add lines like SetEnvIf Remote_Addr "^203\.0\.113\." bad_bot using the offending IP range from your server logs.

Do not use both methods at once for the same bots. Pick one. Running both is redundant and makes debugging harder.

Which bots to block — and which to leave alone

The most expensive .htaccess mistake is blocking a bot you need. Search crawlers are how you get traffic; answer-engine crawlers are how you get cited. A quick decision table:

Block if you oppose AI training use: GPTBot, ClaudeBot, CCBot (Common Crawl feeds many models), Bytespider, Applebot-Extended (note: this is a robots.txt token, not a crawler user-agent — it has no user-agent to block; handle it in robots.txt), Google-Extended (same — robots.txt token only).
Think twice: Applebot (powers Siri/Spotlight/Safari results — blocking it removes you from Apple search entirely), FacebookBot (controls how your links preview when shared), PerplexityBot (sends referral traffic; blocking it removes you from Perplexity answers).
Never block: Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, OAI-SearchBot, Claude-SearchBot. These are search and answer-engine crawlers. Blocking them doesn't protect your content from training — it deletes you from search results and AI answers, which is the opposite of what most sites want. The -Extended training tokens (Google-Extended, Applebot-Extended) are separate from the search crawlers and have no search effect when blocked.

When in doubt, block the narrowest thing that solves your problem. A rule blocking all of facebookexternalhit because one scraper annoyed you also breaks your link previews on Facebook and Instagram.

Performance notes: keep the list short

Every rule in .htaccess is evaluated on every single request to your site — every page, every image, every CSS file. A 200-line user-agent blocklist with complex regex is a small tax on each request, and on high-traffic sites it adds up. Practical guidance:

Keep the alternation list to bots you have actually decided about — dozens, not hundreds.
Put bot-blocking rules near the top of the file so blocked requests exit early via the [L] flag.
Anchor patterns where you can: GPTBot as a substring match is fine, but avoid greedy patterns like .*bot.* that match far more than intended (including, potentially, parts of legitimate user-agents).
If your blocklist grows past ~50 entries, consider moving enforcement to the firewall/CDN layer (Cloudflare bot management, for example) instead of .htaccess.

Testing your rules (do this before you trust them)

Confirm your site still loads. Visit your homepage and a few inner pages in a normal browser immediately after saving. If you see a 500 error, restore your backup — you have a syntax error.
Simulate a blocked bot. Use curl with a spoofed user-agent: curl -A "GPTBot/1.0" -I https://yoursite.com/ should return HTTP/2 403. Then curl -A "Mozilla/5.0" -I https://yoursite.com/ should return 200. If both return 403, your pattern is too broad.
Check you didn't block Googlebot. Run curl -A "Googlebot/2.1" -I https://yoursite.com/ — it must return 200. Also verify your live robots.txt separately with the free robots.txt tester, since the two layers interact.
Watch your logs for a day. A sudden drop in all bot traffic is expected; a drop in human traffic means something is wrong. Check Search Console's crawl stats after a week to confirm Googlebot is still fetching normally.

For a deeper walkthrough of validation, see how to test robots.txt before publishing — the same discipline applies here.

When .htaccess isn't an option

Plenty of sites can't use .htaccess at all. Your alternatives, in order of effectiveness:

Nginx servers: the equivalent lives in the server block config (if ($http_user_agent ~* (GPTBot|ClaudeBot) ) { return 403; }), but most shared hosts won't give you access — ask your host or move the logic to a CDN.
Cloudflare (any platform): Cloudflare's one-click "Block AI bots" toggle plus custom WAF rules by user-agent work in front of any origin — WordPress, Shopify, static sites. This is the most practical hard-blocking route for sites without server access. Cloudflare documents the approach on its blog for blocking AI bots, scrapers, and crawlers.
WordPress plugins: security plugins can block by user-agent at the application layer — later in the request than .htaccess (WordPress has to boot first), but workable. See the WordPress AI-crawler blocking guide.
Shopify / Wix / Squarespace: no server-level blocking exists; robots.txt customization is your lever (Shopify exposes it via robots.txt.liquid).

Whichever layer you use, keep robots.txt accurate. A bot you hard-block at the server but allow in robots.txt creates confusing signals; a bot you allow in robots.txt but block at the server wastes its crawl budget on 403s. The two files should tell the same story. Build the robots.txt side with the .htaccess generator companion workflow: declare in robots.txt, enforce in .htaccess, verify with the tester.

FAQ

Will blocking bots in .htaccess hurt my SEO?

Only if you block the wrong bots. Blocking training crawlers (GPTBot, ClaudeBot, CCBot) has no effect on Google or Bing rankings — those are separate crawlers with separate rules. Blocking Googlebot or Bingbot in .htaccess will remove you from search results, so always test with a Googlebot user-agent string after making changes.

Does .htaccess blocking stop scrapers that ignore robots.txt?

Yes, for scrapers that use the user-agents you block — the server returns 403 before serving any content, regardless of what your robots.txt says. It does not stop scrapers that rotate user-agents or residential IPs; for those you need IP/rate-based rules or a bot-management service like Cloudflare.

Can I use .htaccess bot blocking on Nginx hosting?

No. .htaccess is Apache/LiteSpeed only and is silently ignored on Nginx. On Nginx you need server-block rules (usually requiring host support), or move blocking to a CDN/WAF layer like Cloudflare in front of your site.

Should I block AI crawlers in both robots.txt and .htaccess?

Yes — they serve different purposes. robots.txt is the standards-based declaration that well-behaved bots honor and that documents your intent. .htaccess is enforcement for bots that ignore the declaration. Keep both consistent so you're not telling two different stories.

How do I undo a block if something breaks?

Restore your backed-up .htaccess via FTP or file manager — changes are instant. If you didn't keep a backup, delete or comment out the block you added (prefix lines with #). This is why the backup step before editing is non-negotiable.

Block AI Crawlers and Bad Bots With .htaccess: Copy-Paste Rules (2026)