How to Block AI Crawlers in WordPress: 3 Methods That Work

WordPress powers over 40% of the web, which makes it the single biggest source of AI training data on the internet — and the easiest platform on which to take control. You can block AI crawlers on WordPress three ways: editing robots.txt through your SEO plugin, blocking at the server or CDN level, or both. This guide walks through each method, explains which bots to block and which to keep, and shows you how to verify it worked. No code required for the first two methods.

First: decide what you are blocking (and why it matters)

Before touching any setting, understand the split that trips up most WordPress owners. AI companies run separate bots for training and for search answers:

• Training bots (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, Meta-ExternalAgent) — collect content to train models. Blocking these is a licensing and consent decision.
• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) — index content so it can be cited in AI answers. Blocking these removes you from ChatGPT, Claude, and Perplexity answers.

Most publishers want to block training while staying citable in AI search. A few want everything blocked. Almost nobody wants to block search bots while allowing training bots — yet that is exactly what sloppy one-click "block AI" toggles sometimes do. If you have not read it, our pros-and-cons guide will help you decide which camp you are in before you configure anything.

Method 1: robots.txt via your SEO plugin (easiest, recommended first step)

WordPress generates a virtual robots.txt by default. Your SEO plugin lets you override it with custom rules — no FTP, no file editing.

Rank Math

1. Go to Rank Math SEO → General Settings → Edit robots.txt (enable the robots.txt module under Rank Math → Dashboard if you do not see it).
2. Add a block for each training bot you want to stop. For the standard "block training, keep AI search" setup, paste:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

3. Save. Rank Math serves the merged file at yourdomain.com/robots.txt immediately.

Yoast SEO

1. Go to Yoast SEO → Tools → File editor. (If the file editor is missing, your host has disabled file editing — use Method 2 or ask your host.)
2. Click Create robots.txt file if none exists, then add the same bot blocks above your existing rules.
3. Save changes.

Important robots.txt mechanics for WordPress owners

• One group per bot. A bot obeys only the group that names it. If you already have a User-agent: * group, your new User-agent: GPTBot group overrides it for GPTBot — the wildcard rules will not apply to it. This is correct behavior, not a bug.
• Do not version-match. Write GPTBot, not GPTBot/1.4. Version numbers change and your rule would silently stop matching.
• Blocking Google-Extended does not hurt Google rankings. Google states the opt-out has no effect on Search inclusion or ranking — it only governs Gemini and Vertex AI training use.
• Expect a ~24-hour lag. OpenAI and Perplexity note it takes about a day for their systems to pick up robots.txt changes.

Method 2: Block at the CDN — Cloudflare's AI bot toggle

robots.txt is voluntary — well-behaved crawlers honor it, but scrapers and some aggressive trainers ignore it. If you use Cloudflare (many WordPress hosts include it), you can enforce blocking at the edge:

1. Open your Cloudflare dashboard → select your domain → Security → Bots.
2. Find the AI crawlers / AI bots setting (Cloudflare maintains its own bot verification list) and enable blocking.
3. For finer control, create a WAF custom rule: If User-Agent contains "GPTBot" (or ClaudeBot, CCBot, etc.), then Block. This lets you block training bots while explicitly allowing OAI-SearchBot and PerplexityBot — something the blunt toggle cannot do.

One caution: CDN-level blocks can override your robots.txt. If you allow OAI-SearchBot in robots.txt but your Cloudflare toggle blocks all AI bots at the edge, ChatGPT search cannot reach your pages. When AI tools report they cannot read your site despite a permissive robots.txt, the CDN dashboard is the first place to look. Our full Cloudflare AI-bot blocking guide covers the WAF rule syntax in detail.

Method 3: Server-level blocking (.htaccess for Apache hosts)

Most shared WordPress hosting runs Apache, which means you can block bots in .htaccess — enforced at the server, invisible to the application, and effective against bots that ignore robots.txt:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule ^.*$ - [F,L]

This returns a 403 Forbidden to the named bots on every request. Place it near the top of your .htaccess, above the WordPress rewrite block, and back the file up first — a syntax error here can take down the whole site. If hand-editing server files makes you nervous, generate the rules with the free .htaccess generator instead.

Nginx hosts (Kinsta, many managed hosts) cannot use .htaccess — you would add equivalent if ($http_user_agent ~* ...) rules in the server block, which usually requires host support or a control-panel tool. On managed WordPress hosting, ask support before editing server config.

Which method should you use?

• robots.txt alone — enough if you trust the major labs to honor it (OpenAI, Anthropic, Google, Apple, and Meta all document compliance). This is the right default for most blogs and business sites.
• robots.txt + Cloudflare WAF — the practical maximum for most WordPress owners. Policy plus enforcement, with per-bot control.
• Add .htaccess — when you are not on Cloudflare, or when log analysis shows bots ignoring your robots.txt (Bytespider is the usual suspect).

Layer them rather than choosing one: robots.txt declares your policy to every compliant crawler on the internet, while the CDN or server layer enforces it against the non-compliant ones.

What about WordPress plugins that promise "block AI crawlers"?

Several plugins offer one-click AI blocking. They work — but audit what they actually block before activating. Some block only the training tokens (good), some block every known AI user agent including the search bots (which silently removes you from ChatGPT and Perplexity answers), and a few inject rules you cannot easily inspect. The methods above give you the same result with full visibility into exactly which tokens are affected. If you do use a plugin, verify its block list against the community-maintained AI crawler registry and confirm search bots like OAI-SearchBot are not on it.

Verify it worked

1. Check the file. Visit yourdomain.com/robots.txt in a browser — you should see your new bot groups. Validate the syntax with the robots.txt tester.
2. Fetch as a bot. From a terminal: curl -A "Mozilla/5.0; compatible; GPTBot/1.0" https://yourdomain.com/robots.txt — confirm the file is served with HTTP 200.
3. Watch the logs. After ~24 hours, check your access logs (or your host's visitor logs, or a security plugin like Wordfence's live traffic) for the GPTBot, ClaudeBot, and CCBot user-agent strings. Hits should drop toward zero for blocked bots.
4. Check Search Console. Your Google Search Console coverage report should show no change in indexed pages — confirming the blocks did not leak into Googlebot's rules.

Troubleshooting: when the block does not seem to work

You set everything up, but the bots keep showing in your logs. Work through this checklist before assuming the worst:

• You checked too soon. OpenAI and Perplexity need roughly 24 hours to pick up robots.txt changes. Re-check tomorrow before changing anything.
• You are reading cached logs. Some hosts and security plugins show delayed traffic data. Confirm the timestamps on the hits you are seeing.
• A CDN rule is overriding robots.txt. If Cloudflare's AI-bot toggle is off but a WAF rule or "Bot Fight Mode" is misconfigured, edge behavior and file policy can disagree. The edge always wins — check Security → Bots first.
• The hits are from a different token. Site owners often block GPTBot and then panic at OAI-SearchBot hits in the logs. That is a different bot doing a different job (search indexing), and if you want AI search visibility, those hits are exactly what you want to see.
• A caching plugin is serving a stale robots.txt. Purge all caches (plugin, host, and CDN) after editing robots.txt, then fetch the file in an incognito window to confirm the live version contains your rules.
• The bot is not who it claims to be. Scrapers sometimes spoof AI user-agent strings. If a "GPTBot" ignores your rules but hits at inhuman speed from a datacenter IP outside OpenAI's published ranges, it is likely a spoofed scraper — block it by IP or behavior, not by user-agent token.

When in doubt, test directly: fetch your robots.txt with the bot's user agent via curl (Method section above) and confirm the correct group is served. If the file is right and the bot is genuine, give it the full 24-hour propagation window.

Do not forget the other half: declare what AI systems should know

Blocking is only half of AI crawler management. The other half is making sure the AI systems you do welcome understand your site correctly. Add an llms.txt file at your domain root pointing models and answer engines at your best content in clean Markdown — and keep your XML sitemap fresh so search crawlers keep finding new posts. Control who may read, then guide those you welcome: that is the complete WordPress AI-crawler playbook.

FAQ

Will blocking AI crawlers slow down my WordPress site?

The opposite — AI training crawls can be surprisingly heavy, and blocking them reduces bot traffic and server load. The rules themselves (a few robots.txt lines or WAF conditions) add negligible overhead.

Does WordPress have a built-in AI crawler block?

No. WordPress core has no AI-crawler settings as of 2026. You configure it through robots.txt (via your SEO plugin), your CDN, or server rules as described above.

Will this affect my Google rankings?

No — provided you only block AI-specific tokens. Googlebot follows its own rules and is unaffected by groups addressed to GPTBot or ClaudeBot. Double-check that you have not accidentally disallowed Googlebot, and confirm in Search Console.

Can I block AI crawlers on WordPress.com (hosted) sites?

On WordPress.com hosted plans you cannot edit .htaccess or server config, but you can still manage robots.txt through SEO plugin settings on eligible plans and use Cloudflare in front of the site for edge blocking.

What if a bot ignores my robots.txt?

Move to enforcement: Cloudflare WAF rules or .htaccess user-agent blocks return 403 regardless of what the bot claims to honor. Check logs to identify the offender — persistent ignore-lists usually point at lesser-known scrapers rather than the major labs.

Should I block AI crawlers on a new blog?

Usually not. New sites need every discovery channel available — including AI search citations. Blocking makes sense once you have valuable original content worth protecting, or a clear editorial policy against training use. Start permissive; restrict deliberately.