Quick answer: Meta runs several web crawlers with separate robots.txt tokens: Meta-ExternalAgent for AI training, Meta-WebIndexer for search indexing, and Meta-ExternalFetcher for user-triggered fetches. To opt out of Meta AI training, add User-agent: Meta-ExternalAgent with Disallow: / to your robots.txt. But blocking the wrong token can break Facebook and Instagram link previews — so you need to know which bot does what before you touch the file.
Meet Meta's crawlers: five bots, five jobs
Most blocklists treat "Meta" as one bot. It isn't. Meta's own developer documentation lists five crawlers, and the separation between them is the whole point — each carries its own robots.txt token, so you can permit indexing while refusing training, or the reverse. Here they are:
- Meta-ExternalAgent — collects data for training Meta's foundation models and for indexing content directly. This is the token you block to opt out of Meta AI training.
- Meta-WebIndexer — feeds Meta's search index. If you want your content findable in Meta's search surfaces, leave this one alone.
- Meta-ExternalFetcher — retrieves individual links at a user's request, powering agentic features. Meta says this one may bypass robots.txt because the fetch was initiated by a user.
- FacebookExternalHit — the classic link-preview fetcher that grabs titles, images, and descriptions when someone shares your URL on Facebook, Instagram, or Threads. It may bypass robots.txt when running security or integrity checks.
- Meta-ExternalAds — crawls for advertising and other business products.
Note that Meta documents the tokens in lowercase (meta-externalagent). Robots.txt user-agent matching is case-insensitive per the standard, so Meta-ExternalAgent works the same — use whichever casing your team finds readable.
How to block Meta AI training with robots.txt
If your goal is simple — "don't use my content to train Meta's AI models" — this is the snippet:
# Opt out of Meta AI training
User-agent: Meta-ExternalAgent
Disallow: /That's it. This tells Meta's training crawler not to use your pages, while Meta-WebIndexer keeps indexing you, FacebookExternalHit keeps rendering your link previews, and Meta-ExternalFetcher keeps fetching links users share. This is the configuration most publishers want: training opted out, everything else untouched.
Meta notes that it can take up to 24 hours for robots.txt changes to take effect, because its crawlers may cache the file. Don't expect instant results — and don't read an afternoon log as evidence of non-compliance on the morning you publish the rule.
How to block ALL Meta crawlers (and why you probably shouldn't)
Some sites block the whole Meta family:
# Block all Meta crawlers
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-WebIndexer
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: Meta-ExternalAds
Disallow: /
User-agent: FacebookExternalHit
Disallow: /Think twice before pasting this. Blocking FacebookExternalHit means links shared on Facebook, Instagram, and Threads render as bare URLs with no title, image, or description — killing the social traffic those previews generate. Blocking Meta-WebIndexer removes you from Meta's search index. For most sites, the targeted approach (block only Meta-ExternalAgent) is strictly better.
The uncomfortable truth about Meta-ExternalFetcher
Meta states openly that Meta-ExternalFetcher may bypass robots.txt because it performs fetches requested by a user. This is the same pattern as OpenAI's ChatGPT-User and Perplexity's Perplexity-User: user-triggered fetching sits in a gray zone where robots.txt "may not apply."
What does that mean in practice? If someone asks Meta's AI assistant about your page, the fetcher may still retrieve it even with a Disallow rule in place. Robots.txt establishes your stated preference — which matters legally and for the well-behaved crawlers — but it is not a technical barrier. If you have content that absolutely must not be retrieved under any circumstances, authentication (a login or paywall) is the only real answer. See our guide on protecting content from AI scraping for the full layered approach.
Verifying your Meta crawler rules
- Check the file is live. Visit
yourdomain.com/robots.txtand confirm your Meta rules appear. Test them with our free robots.txt tester. - Wait 24 hours. Meta explicitly asks for this window because of robots.txt caching.
- Test link previews separately. Share a test URL in a private Facebook post or message and confirm the preview still renders — if you only blocked Meta-ExternalAgent, it should.
- Check server logs. Grep your access logs for the Meta tokens to confirm the training crawler backed off while FacebookExternalHit still visits:
grep -i 'meta-external' /var/log/nginx/access.log | cut -d'"' -f6 | sort | uniq -cMeta vs the other labs: how the opt-outs compare
Every major AI lab now offers a robots.txt-based opt-out, but the shapes differ:
- OpenAI:
GPTBot(training) is separate fromOAI-SearchBot(ChatGPT search answers) andChatGPT-User(user-triggered). Clean separation, like Meta's. - Anthropic:
ClaudeBot(training) vsClaude-SearchBotandClaude-User. Anthropic is unusually strict — it says all its bots honor robots.txt, including the user-triggered one. - Google:
Google-Extendedis a permission token with no crawler of its own; it doesn't affect Search at all. Blocking it costs you nothing in rankings. - Meta:
Meta-ExternalAgentfor training, but the fetcher and link-preview bots may bypass robots.txt in specific cases.
The pattern is consistent: block the training token, allow the answer/index tokens, and treat user-triggered fetchers as best-effort. Our complete list of AI crawlers covers every token with its exact purpose, and the robots.txt generator builds a file with the Meta presets included.
How to verify a Meta crawler hit is really Meta
Anyone can fake a user-agent string — scrapers spoof meta-externalagent precisely because site owners whitelist or monitor it. Before you act on a log entry, verify the visitor is genuinely Meta's. Meta doesn't publish a simple JSON list of crawler IPs the way Google does; instead, it directs operators to the RADb routing registry and the address space of its autonomous system, AS32934. In practice, verification looks like this:
- Reverse DNS check. Take the IP from your log and look up its PTR record. Genuine Meta crawler traffic resolves to hostnames under Meta's infrastructure.
- Forward confirm. Resolve that hostname back to an IP and confirm it matches the original. This two-way check defeats simple spoofing.
- Expect rotation. Site owners report Meta's crawlers arriving from many rotating IPv6 addresses at once (large allocations under
2a03:2880::/32are commonly observed). That's why token-based robots.txt rules beat IP blocklists for Meta — the addresses shift too often to maintain by hand.
The takeaway: use robots.txt tokens for policy (they're stable), and use IP-level data only for verification, not for blocking.
Meta-ExternalAds: the token everyone forgets
Of the five Meta crawlers, Meta-ExternalAds gets the least attention — and it's the one most likely to be crawling your site right now without you knowing. It crawls for Meta's advertising and business products: checking landing pages behind ads, verifying advertiser domains, and gathering data for ad-related features.
Should you block it? For most sites, no. If you run Meta ads, blocking Meta-ExternalAds can interfere with ad review and landing-page checks — the exact systems that decide whether your ads get approved. If you don't advertise on Meta at all and want minimal Meta footprint, adding it to your blocklist is harmless:
# Block Meta's ad crawler too (only if you don't run Meta ads)
User-agent: Meta-ExternalAds
Disallow: /The rule of thumb: block the crawlers tied to outcomes you don't want (training, if you opt out), and leave alone the crawlers tied to outcomes you do want (previews, indexing, ad approval).
Putting it together: a sensible Meta policy for most sites
After all the detail, here's what the robots.txt of a typical content site should say about Meta in 2026:
# Meta policy: opt out of AI training, keep everything else working
User-agent: Meta-ExternalAgent
Disallow: /
# Everything else (Meta-WebIndexer, FacebookExternalHit,
# Meta-ExternalFetcher, Meta-ExternalAds) is intentionally allowed.Five lines, one decision, zero broken previews. If your situation changes — you start licensing content, or you launch a paywalled product — revisit the other tokens then. And if you're building a full AI-crawler policy rather than just handling Meta, generate the whole file with our robots.txt generator instead of assembling tokens by hand.
Instagram, Threads, and WhatsApp: does blocking affect them?
A common worry: "If I block Meta's crawlers, will my Instagram posts or Threads links break?" Short answer: no — as long as you block only Meta-ExternalAgent.
Here's why. Your Instagram profile, your Threads posts, and your WhatsApp messages live inside Meta's apps — they're not fetched from your website by a crawler at all. The only interaction between your website and Meta's social apps is the link preview: when someone pastes your URL into Instagram DMs, a Threads post, or a Facebook share, FacebookExternalHit fetches your page to build the preview card. That crawler has its own token, and blocking Meta-ExternalAgent doesn't touch it.
The one scenario to watch: Meta's AI assistant features can fetch web content when users ask about it (that's Meta-ExternalFetcher's job). If you block the fetcher too, links shared inside Meta AI chats may not expand into rich previews. For most sites that's a minor cosmetic loss — but it's a good example of why the targeted approach beats the "block everything Meta" blocklist. Block the training crawler, leave the user-facing fetchers alone, and your social presence works exactly as before. When in doubt, test a shared link in a private message after publishing your robots.txt changes — it takes thirty seconds and removes all guesswork.
Frequently asked questions
What is the robots.txt user agent for Meta AI?
Meta's AI training crawler uses the token Meta-ExternalAgent (documented as meta-externalagent; matching is case-insensitive). To opt out of Meta AI training, add User-agent: Meta-ExternalAgent with Disallow: / to your robots.txt.
Does blocking Meta-ExternalAgent break Facebook link previews?
No. Link previews are fetched by FacebookExternalHit, a separate crawler with its own token. Blocking only Meta-ExternalAgent leaves previews working normally. Only blocking FacebookExternalHit itself would break previews.
Does Meta honor robots.txt?
Meta states that its crawlers honor robots.txt, with two documented exceptions: Meta-ExternalFetcher may bypass it for user-requested fetches, and FacebookExternalHit may bypass it for security or integrity checks. Meta-ExternalAgent and Meta-WebIndexer are in the compliant category.
How long does it take for Meta to respect new robots.txt rules?
Meta asks you to allow up to 24 hours, because its crawlers may cache your robots.txt file. If the crawler keeps visiting after a day or two, re-check your syntax — a misplaced rule is a far more common cause than non-compliance.
Can I block Meta AI but stay in ChatGPT and Perplexity answers?
Yes — these are completely independent systems. Blocking Meta-ExternalAgent affects only Meta's training. ChatGPT answers depend on OAI-SearchBot and Perplexity answers on PerplexityBot, each controlled by their own robots.txt tokens.
Will blocking Meta-ExternalAgent hurt my Facebook page or Instagram traffic?
No. Your Facebook page, Instagram profile, and the link previews for shared URLs are unaffected. Meta-ExternalAgent only covers web crawling for training and direct content indexing — not Meta's social products.
