Yes, you can block AI crawlers with Nginx — and it is the most reliable way to do it. A robots.txt file only asks bots to stay away. An Nginx rule answers their request with a 403 Forbidden before your app ever sees it. If you run your own server or VPS, this guide shows you exactly how to set it up: the recommended map method, the simpler if-directive method, how to test it with curl, and the mistakes that break sites.
Here is the short version. Nginx reads the User-Agent header of every incoming request. AI crawlers identify themselves with names like GPTBot, ClaudeBot, CCBot and PerplexityBot. You match those names in your Nginx config and return a 403 status. Legitimate search crawlers like Googlebot keep working, real visitors never notice, and the training bots stop eating your bandwidth and your content.
Why robots.txt alone is not enough
Robots.txt is the first layer, and you should keep it. Most well-known AI crawlers do respect it: OpenAI says GPTBot honors robots.txt, Anthropic says the same for ClaudeBot, and Common Crawl respects it for CCBot. A simple stanza — User-agent: GPTBot, Disallow: / — stops the honest bots within a day or two.
But robots.txt has three hard limits. First, it is a request, not a lock. Any bot can ignore it, and the bots you most want to stop are the ones most likely to ignore it. Second, it does nothing about bandwidth: a bot that ignores your file still downloads every page it wants. Third, it cannot stop user-triggered agents. When a person asks ChatGPT or Claude to fetch a page, the agent that visits (ChatGPT-User, Claude-Web) may ignore robots.txt entirely, because the providers treat that as acting on a user's instruction rather than bulk crawling.
That is why server-level blocking matters. Nginx sits in front of your application. A rule there rejects the request at the door, so no PHP, Node, or database work happens at all. The bot gets a 403 and your server spends almost nothing on the visit.
How Nginx blocking actually works
Every HTTP request carries a User-Agent header — a short string that says what software is making the request. A normal browser sends something like Mozilla/5.0 with Chrome or Firefox details. GPTBot sends a string that contains GPTBot/1.2 and a link to OpenAI's crawler page. ClaudeBot, CCBot, PerplexityBot and the rest do the same with their own names.
Nginx can inspect that header with the $http_user_agent variable and match it against a pattern. If it matches one of your blocked names, Nginx returns a status code of your choice — usually 403 Forbidden — and stops processing. The request never reaches your app.
Two things to understand before you touch any config. First, this only stops bots that identify themselves honestly. A scraper that pretends to be Chrome will walk straight past a User-Agent rule. That is a real limit, and it is why this is one layer of defense, not the whole wall. Second, a bad rule can block real users or take your site offline. Always test with nginx -t before reloading, and always test with curl afterwards.
Method 1: the map directive (recommended)
The map directive is the clean way to block a list of bots. You define it once in the http block of your Nginx config — usually in /etc/nginx/nginx.conf or a file under /etc/nginx/conf.d/ — and it sets a variable to 1 whenever the User-Agent matches a blocked bot. Your server blocks then just check that variable.
Here is a working example:
http {
map $http_user_agent $blocked_bot {
default 0;
~*GPTBot 1;
~*ChatGPT-User 1;
~*OAI-SearchBot 1;
~*ClaudeBot 1;
~*Claude-Web 1;
~*anthropic-ai 1;
~*CCBot 1;
~*PerplexityBot 1;
~*Perplexity-User 1;
~*Bytespider 1;
~*Amazonbot 1;
~*Applebot-Extended 1;
~*Google-Extended 1;
~*Meta-ExternalAgent 1;
~*meta-webindexer 1;
~*DuckAssistBot 1;
~*cohere-ai 1;
~*MistralAI-User 1;
~*YouBot 1;
~*Applebot 1;
}
server {
listen 80;
server_name yoursite.com;
if ($blocked_bot) {
return 403;
}
# the rest of your config goes here
}
}A few notes on what this does. The ~* means case-insensitive regex matching, so GPTBOT and gptbot match too. The default 0 line means every visitor not on the list gets through. Each line is one bot, so adding or removing a bot is a one-line edit. The if ($blocked_bot) check uses a plain variable test, which is the one safe use of if inside a server block — it only returns a status, it does not rewrite anything.
Why is this better than a long if with a giant regex? Performance and safety. Nginx evaluates the map once per request, and the pattern list is easy to read and maintain. With twenty bots in one regex, a single typo can break the whole expression — or worse, match something you never intended, like a real browser string.
Method 2: the if directive (simple cases)
If you only want to block one or two bots and you want the smallest possible change, the if directive inside a server block works:
server {
listen 80;
server_name yoursite.com;
if ($http_user_agent ~* (GPTBot|ClaudeBot|CCBot)) {
return 403;
}
# the rest of your config goes here
}This is fine for a short list. The well-known warning about Nginx if — often summarized as 'if is evil' — applies to rewrite logic and complex rules inside location blocks, where if can behave unpredictably. A simple return 403 based on a header check is the accepted safe pattern, and it is the example Nginx's own documentation community uses. Still, once your list grows past a handful of bots, switch to the map method above. It is easier to maintain and easier to review.
Which bots should you block?
Do not block everything that moves. Your goal is to stop training and bulk-collection crawlers while keeping the bots that bring you traffic. Here is a practical split.
Block these if you do not want your content used for AI training: GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl dataset), Google-Extended (Google AI training), Applebot-Extended (Apple training), Meta-ExternalAgent (Meta training), Bytespider (ByteDance training), Amazonbot (Amazon training), cohere-ai (Cohere training), and MistralAI-User (Mistral's user-triggered agent).
Think twice before blocking these: OAI-SearchBot powers citations in ChatGPT search, so blocking it can remove your pages from ChatGPT answers. PerplexityBot drives Perplexity citations and referral traffic. DuckAssistBot feeds DuckDuckGo's AI answers. Claude-Web and ChatGPT-User are user-triggered agents: they visit because a real person asked for a specific page, and blocking them can frustrate your own readers. If your goal is training opt-out only, leave the search and answer bots alone.
Never block these: Googlebot (Google Search), Bingbot (Bing), Applebot (Apple Search and Siri), facebookexternalhit (link previews when someone shares your URL), and Twitterbot (link previews on X). Blocking any of these hurts your search visibility or breaks your social sharing cards. Note the difference between Applebot and Applebot-Extended: the first is Apple's search crawler, the second is its AI training crawler. They are separate user agents with separate jobs, and only the second one belongs on a training blocklist.
A good starting point for most sites is to block the pure trainers — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot — and leave the answer-engine bots alone until you have data on what they bring you. You can always add more later.
Where the config goes, and how to apply it
On a typical Ubuntu or Debian server, your site config lives in /etc/nginx/sites-available/yoursite with a symlink in /etc/nginx/sites-enabled/. The map block must go in the http context — the main nginx.conf file, or a file in /etc/nginx/conf.d/, which is included inside the http block. The if ($blocked_bot) check goes inside your server block.
The workflow is always the same:
- Edit the config file.
- Run
sudo nginx -tto test the syntax. If it reports an error, fix it before doing anything else — a broken config can stop Nginx from starting. - Run
sudo systemctl reload nginxto apply the change without dropping connections. Use reload, not restart, so live visitors are not interrupted. - Test with curl, as described below.
If you use a control panel like RunCloud, Ploi, or a managed VPS, the file locations differ, but the directives are the same. Paste the map into your global Nginx config area and the check into your site config, then reload through the panel.
Testing your rules with curl
Never trust a config you have not tested. Curl lets you pretend to be a bot and see exactly what your server returns:
curl -I -A 'Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)' https://yoursite.com/If your rule works, the first line of the response should read HTTP/2 403. Then test a normal browser user agent and confirm you still get a 200:
curl -I -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36' https://yoursite.com/Test each bot on your list, and test your homepage plus one inner page. A rule that works on the homepage but not on subpages usually means the check was placed inside a single location block instead of the server block — move it up a level.
You can also watch your access log in real time to see the blocks landing:
sudo tail -f /var/log/nginx/access.log | grep -i gptbotOver the following days, the GPTBot lines in that log should drop to near zero, and the ones that remain should show 403 instead of 200.
Common mistakes that break sites
Forgetting to reload. Editing a config file changes nothing until Nginx reloads it. If your curl test still shows 200 after an edit, the reload probably did not happen, or you edited a file that is not included in the active config.
Putting the map inside a server block. The map directive only works in the http context. Nginx will refuse to start, and nginx -t will tell you so. Read the error; it names the exact line.
Matching too broadly. A pattern like bot matches Googlebot, Bingbot, and every other bot — including the ones you need. A pattern like GPT could match unrelated strings. Always match the full bot token, and test with a normal browser user agent afterwards.
Blocking Googlebot by accident. Google-Extended and Googlebot are different user agents. If you write a pattern that matches Googlebot, your Google rankings will suffer. Double-check every line against the 'never block' list above.
Using 444 instead of 403 without understanding it. Nginx's non-standard 444 status closes the connection without sending any response. It saves a little bandwidth, but it confuses debugging tools and some monitoring services. Stick with 403 unless you have a specific reason to use 444 and you understand the tradeoff.
Blocking empty user agents. Some guides suggest blocking requests with no User-Agent header at all. That sounds tough, but it also blocks legitimate tools, health checks, and some feed readers. Leave empty user agents alone unless you have evidence of abuse from them.
Forgetting robots.txt. Nginx blocking and robots.txt are complementary, not alternatives. Keep your robots.txt rules in place: they stop honest bots before they even send a page request, and they are the mechanism the bot operators officially document. Nginx is the enforcement layer for the bots that ignore the request.
What Nginx blocking cannot do
Be honest about the limits. A User-Agent rule only stops bots that tell the truth about who they are. A scraper that spoofs a Chrome user agent will pass straight through. For those, you need rate limiting, behavioral detection, or a managed firewall — Nginx's limit_req module can slow down aggressive IPs, and Cloudflare's bot management can catch more, but neither is a complete answer on its own.
IP verification is the stronger check for the big crawlers. OpenAI publishes its GPTBot IP ranges in a JSON file, and Google publishes its crawler IPs too. You can verify that a request claiming to be GPTBot really comes from OpenAI's network by checking the IP against the published list or doing a reverse DNS lookup. That is more work, but it is the only way to catch impostors using a real bot's name.
Also, blocking at the server level does not remove content that is already in a training dataset, and it does not stop a bot that already downloaded your pages from using that copy. It stops future collection. If your content is already widely copied, a block is damage control for tomorrow, not a recall of yesterday.
Layering Nginx with robots.txt and Cloudflare
The strongest setup uses three layers. Robots.txt declares your policy in the standard format every honest bot reads. Nginx enforces it at the server level for the bots that ignore the file. And if your site sits behind Cloudflare, its AI Crawl Control — available on all plans — lets you allow or block AI crawlers by name with a toggle, plus shows you which bots visit and which ones break your robots.txt rules. Cloudflare's rules run before traffic reaches your server at all, so they save the most bandwidth.
Use our free tools to build each layer without hand-writing files. Generate a correct robots.txt with our robots.txt generator, check it before publishing with the robots.txt tester, and if your site runs Apache anywhere in the stack, build matching rules with the .htaccess generator. For the full picture, read our guide to blocking AI crawlers with robots.txt and our Cloudflare AI bot blocking guide.
FAQ
Does blocking GPTBot in Nginx hurt my Google rankings?
No. GPTBot is OpenAI's training crawler, not a search crawler. Googlebot is a completely separate user agent, and as long as your rules do not match Googlebot, your Google rankings are unaffected. The same applies to Bingbot.
Will blocking GPTBot remove my site from ChatGPT answers?
Not by itself. ChatGPT search citations come from OAI-SearchBot, a different crawler. If you block GPTBot but allow OAI-SearchBot, your pages can still be cited in ChatGPT answers. Block both only if you want out of ChatGPT entirely.
Should I use the map directive or the if directive?
Use the map directive if you are blocking more than a few bots. It is faster, easier to read, and easier to maintain. Use the if directive only for a short list of one to three bots where you want the smallest possible change.
How do I know the block is working?
Test with curl using a fake GPTBot user agent and look for a 403 response. Then watch your Nginx access log for GPTBot entries: they should fall sharply within 24 to 48 hours, and the remaining hits should show 403 instead of 200.
What if I make a mistake and block real visitors?
Remove or comment out the rule, run nginx -t, reload Nginx, and test with a normal browser user agent. Because a reload takes seconds and does not drop connections, a bad rule is quick to undo — which is another reason to test with curl before you call the job done.
Do I still need robots.txt if I block bots in Nginx?
Yes. Robots.txt is the standard, documented way to declare your policy, and honest bots read it before they crawl. Nginx is the enforcement layer for bots that ignore the file. Keep both, and keep them consistent with each other.
The bottom line
Nginx gives you real enforcement where robots.txt only asks politely. Start with the map directive, block the training crawlers, leave the search and answer bots alone unless you have a reason to block them, and test every rule with curl before you move on. Pair the server rules with a clean robots.txt and you have a setup that stops honest bots at the file, stops dishonest ones at the server, and leaves your search traffic untouched.
