Blocking GPTBot takes one robots.txt block — but doing it right means understanding what you're actually blocking. GPTBot is OpenAI's crawler for collecting training data for its AI models. It honors robots.txt, so a single Disallow rule opts your content out of future training datasets. This guide shows the exact configuration, how to verify it works, and the mistakes that make blocks silently fail.
What GPTBot Is (and Isn't)
- It's a training crawler, not a search crawler. Blocking it has zero effect on your Google rankings or your visibility in ChatGPT's search answers — those come from
OAI-SearchBot, a separate crawler with a separate robots.txt token. - It honors robots.txt. OpenAI documents this; a disallow is treated as an opt-out signal for training data.
- Blocking is forward-looking only. It signals that future content should be excluded from training datasets. Content collected before your block may already be in existing models.
Its user-agent contains the token GPTBot, typically Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot). Always match on the token, never the full string — version numbers change.
The Basic Block
Add this to your robots.txt at the domain root:
User-agent: GPTBot
Disallow: /That's it for a full block. GPTBot will stop crawling your content once it re-fetches your robots.txt — typically within days, not minutes, since crawlers cache the file.
Partial Blocking (Block Some Sections, Allow Others)
User-agent: GPTBot
Disallow: /blog/
Disallow: /research/
Allow: /press/
Allow: /about/Rules to remember: the most specific matching rule wins for a given path, paths are case-sensitive, and each subdomain needs its own robots.txt.
The Most Popular Setup: Block Training, Keep ChatGPT Search
Most public sites want the training opt-out without losing ChatGPT citations. Since the two crawlers use separate tokens, you can have both:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Verifying the Block Actually Works
1. Syntax check. One typo can void a rule. Paste your file into our free robots.txt tester and test your key URLs against the GPTBot user-agent — they should show as blocked. (Note: Google removed its own robots.txt tester from Search Console in 2023, so third-party testers like ours are now the way to do this.)
2. Log check. Search your server access logs:
grep -i "gptbot" /var/log/apache2/access.logAfter a correct block, you should see GPTBot hits drop to robots.txt fetches only. Rather than hand-writing the file, generate it with our robots.txt generator.
Verifying It's Really GPTBot (Not an Impersonator)
Anyone can put "GPTBot" in a user-agent string — scrapers do it to look legitimate. OpenAI publishes the IP ranges GPTBot crawls from at openai.com/gptbot.json. Take the source IPs from your logs and check them against that list: matching token + matching IP = genuine; matching token + non-matching IP = impersonator, which you should block at the server/firewall level.
When robots.txt Isn't Enough: Server-Level Blocking
robots.txt is a voluntary convention. If you need enforcement — against impersonators or aggressive re-crawling — block at the server level. For Apache, our .htaccess generator can build user-agent blocking rules; Cloudflare users can add a WAF rule matching the GPTBot user agent.
Common Mistakes
- Blocking GPTBot and expecting ChatGPT to go blind. Search citations come from OAI-SearchBot. Blocking GPTBot alone changes nothing about ChatGPT answers citing you.
- Using a blanket
User-agent: * / Disallow: /. This blocks Googlebot too. Be specific. - Expecting retroactive removal. The block affects future collection, not models already trained.
- Putting the file in a subfolder. It must be
yourdomain.com/robots.txt. - Forgetting subdomains.
blog.example.comneeds its own file.
Blocking Other OpenAI Crawlers Too
- OAI-SearchBot — powers ChatGPT search citations. Block it only if you want out of ChatGPT answers entirely.
- ChatGPT-User — fetches pages when a specific person asks ChatGPT to look at a URL. Blocking it stops real-time, user-requested fetches.
- OAI-AdsBot — reviews pages submitted as ChatGPT ads for safety. Its data isn't used for training.
# Full OpenAI opt-out (training + search + user fetches)
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /Should You Block GPTBot? The Honest Trade-Off
Reasons to block: you don't want your content feeding future AI models without compensation; licensing constraints; or GPTBot hammering your server.
Reasons to allow: blocking costs you nothing in search visibility, but allowing costs you nothing either — GPTBot sends no traffic either way. If you have no strong view, the default (allow) is less configuration to maintain.
Troubleshooting: Block Not Working?
Work through this order: 1) view your live robots.txt in a private window — is the rule there and spelled GPTBot exactly? 2) Test with our robots.txt tester. 3) Clear caching plugin/CDN cache. 4) Wait — crawlers cache robots.txt for hours. 5) Check whether the "GPTBot" hits are genuine via IP verification.
Frequently Asked Questions
Does blocking GPTBot hurt my Google rankings?
No. GPTBot has no role in search indexing or ranking. Your Google visibility is governed by Googlebot, a completely separate crawler.
How long until the block takes effect?
Once GPTBot re-fetches your robots.txt — typically within days. It won't be instant.
Can I block GPTBot but allow OAI-SearchBot?
Yes — they use separate robots.txt tokens, so the rules are independent. This is the most common configuration for public sites.
Does the block remove my content from existing ChatGPT models?
No. It signals that future content should be excluded from training datasets. Previously collected content may already be in existing models.
What if GPTBot keeps crawling after I blocked it?
First re-check your syntax with a tester, then verify the traffic is genuinely OpenAI's via the published IP ranges. Impersonators ignore robots.txt by definition.
