Applebot is Apple's web crawler: it powers Siri suggestions, Spotlight search, Safari, and Apple News. Applebot-Extended is not a crawler at all — it is a robots.txt-only token that opts your content out of training Apple's AI models. Blocking Applebot-Extended has no effect on your search visibility or rankings; blocking Applebot itself removes you from Apple search entirely. Decide which one you actually mean before you write the rule.
Of all the AI-era crawler questions site owners ask, Apple's is the most confused — because Apple gave two very similar names to two very different things. One is a crawler that decides whether a billion iPhone users can find your content. The other is a permission switch for AI training that crawls nothing. Mix them up and you either train models you meant to opt out of, or vanish from Siri results you wanted to keep.
This guide separates the two, shows the exact robots.txt rules for each goal, explains how to verify real Applebot traffic in your logs, and gives you a decision framework for whether blocking makes sense for your site. If you want the broader field first, see the complete list of AI crawlers.
What Applebot actually does
Applebot is the crawler behind Apple's on-device and services search: results surfaced in Siri, Spotlight, Safari's start page and search suggestions, Apple News, and parts of Apple Maps and Stocks. When someone asks Siri a question or types in Spotlight, the answer often comes from pages Applebot fetched and indexed. Apple has been crawling the web for this purpose for years — long before the current AI wave made crawler-blocking fashionable.
Practically, this means Applebot is a search crawler, not a training crawler. It exists so Apple products can answer questions and show results. Content it fetches can also inform Apple's retrieval for newer AI features — but the training-data question is handled separately, by the Extended token below.
Applebot identifies itself with a user-agent along the lines of Applebot/0.1; +http://www.apple.com/go/applebot, honors robots.txt, and follows standard meta robots tags (noindex, nofollow) and nosnippet. It is, by crawler standards, well-behaved: it respects the rules you publish.
Applebot vs Applebot-Extended: the crucial difference
This is the distinction the whole article turns on, so here it is plainly:
- Applebot — a real crawler. Fetches your pages. Powers Siri, Spotlight, Safari, Apple News. Blocking it with
Disallow: /removes your content from Apple search surfaces. - Applebot-Extended — not a crawler. It never fetches a page. It is a robots.txt token Apple reads as a permission signal for using already-crawled content to train its generative AI models (Apple Intelligence and related features). In Apple's own words, pages that disallow Applebot-Extended can still be included in search results, and its rules are not considered in search ranking.
The parallel is Google's Google-Extended: same design, same logic. A -Extended token is a training-permission switch, not a visibility switch. Blocking it changes whether your content trains models; it does not change whether Siri or Spotlight surface and cite you. If you've read our Google-Extended opt-out guide, the mechanics are identical — only the vendor differs.
One more consequence worth stating: because Applebot-Extended doesn't crawl, there is no user-agent to block at the server level and no IP range to firewall. It exists only as lines in robots.txt. Anyone selling you "Applebot-Extended blocking" at the firewall layer is selling nothing.
How to opt out of Apple AI training (keep your search visibility)
This is what most site owners actually want: out of training, still in Siri and Spotlight. Add a dedicated block to your robots.txt:
User-agent: Applebot-Extended
Disallow: /That's it. Two lines, in their own User-agent group, separate from your Applebot rules. Leave your regular Applebot rules (or the default allow) untouched. Apple documents this in its About Applebot support article: Applebot-Extended determines how data collected by the Applebot user agent may be used for training Apple's general-purpose foundation models.
A partial opt-out is also possible: Disallow: /members/ under Applebot-Extended keeps your public content trainable (if you consider that a contribution) while excluding private or paywalled sections. The token follows standard robots.txt path matching.
Generate the full multi-bot version of this file with the free robots.txt generator — it includes Applebot-Extended alongside GPTBot, ClaudeBot, CCBot, and Google-Extended presets — then verify the result with the robots.txt tester before deploying.
How to block Applebot completely (and why you probably shouldn't)
Full blocking looks like this:
User-agent: Applebot
Disallow: /Understand what you're giving up: every Apple search surface. Siri answers, Spotlight results, Safari suggestions, Apple News inclusion. For most sites this is pure loss — Apple users are a large, high-value audience, and there is no training-data benefit beyond what the Extended opt-out already gives you, since Applebot itself is a search crawler.
The rare cases where full blocking makes sense: staging or internal sites that should never appear in any search surface; sites with licensing constraints that prohibit any Apple indexing; or publishers who've decided Apple's ecosystem delivers them no value and want minimal crawl load. Everyone else should use the Extended opt-out and leave Applebot crawling.
Meta tags Applebot honors
For page-level control (rather than site-wide robots.txt rules), Applebot follows the standard tags:
<meta name="Applebot" content="noindex">— keeps the specific page out of Apple search results while leaving the rest of the site indexed.<meta name="Applebot" content="nosnippet">— allows indexing but limits how Apple uses the page's content in snippets and retrieval context.- Standard
robotsmeta tags (noindex,nofollow) are also honored.
Page-level tags are the right tool when the decision varies by page — for example, keeping your marketing pages in Siri while excluding user-generated or thin content from Apple's index.
How to verify real Applebot traffic
Before blocking anything, check what's actually visiting. In your server logs, look for the Applebot user-agent. Then verify it's genuine — user-agents are trivially faked. Apple publishes reverse-DNS verification: legitimate Applebot requests resolve to hostnames under *.applebot.apple.com (for example, 17-58-101-179.applebot.apple.com).
The check: take the IP from your log, run a reverse DNS lookup (host <ip> on macOS/Linux or an online rDNS tool), and confirm the hostname ends in applebot.apple.com. Then optionally forward-resolve that hostname to confirm it maps back to the same IP. This is the same verification discipline you'd apply to Googlebot or Bingbot — never trust the user-agent string alone, because scrapers routinely impersonate major crawlers.
What to look for in the logs: crawl frequency, which sections get hit hardest, and the ratio of Applebot to total bot traffic. Applebot is generally a light crawler compared to Googlebot; if it's hammering your server, check whether something (infinite calendar URLs, faceted filters) is generating crawl traps before blaming the bot.
Should you allow or block? A decision framework
Run through these in order:
- Do Apple users matter to you? If your audience skews iPhone/iPad/Mac — US/UK consumer, media, local business, e-commerce — Apple search surfaces are distribution you don't want to lose. Default: allow Applebot, decide separately on training.
- Do you object to AI training use? If yes, disallow Applebot-Extended. Costs you nothing in visibility — Apple says so explicitly. This is the highest-value, lowest-risk rule in the whole Applebot topic.
- Is your content the product? Publishers, stock-photo sites, and data businesses have the strongest case for the training opt-out, since their content is exactly what models want. The opt-out doesn't reduce piracy or scraping — it only governs Apple's own training use — but it's the lever Apple offers.
- Do you want AI assistants citing you? Siri's answers and Apple's retrieval features draw on the standard crawl. If being cited in Apple answers matters (local businesses, publishers, reference content), keep Applebot allowed and keep your content answer-friendly: clear headings, direct answers, structured data.
Notice the pattern: for nearly every site, the answer is "allow Applebot, disallow Applebot-Extended." Full blocking is the exception, not the rule. And whatever you decide, say it in robots.txt — Applebot honors the file, so the polite declaration is also the effective one. For the template covering all major AI crawlers in one file, see the robots.txt AI-crawler template.
Putting it together: a complete Apple robots.txt block
Here's how the Apple rules sit alongside the other AI-crawler blocks most sites run. Order of User-agent groups doesn't matter to parsers, but keeping vendors grouped keeps the file readable:
# Apple: allow search crawler, opt out of AI training
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
# Other training opt-outs (same pattern, one block each)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /The explicit Allow: / under Applebot is optional — crawlers default to allowed — but it documents your intent for the next person who edits the file, and it guards against a future wildcard Disallow accidentally catching Applebot. After deploying, confirm the live file with the robots.txt tester: Applebot should resolve to allowed, Applebot-Extended to disallowed.
FAQ
Does blocking Applebot-Extended hurt my rankings or Siri visibility?
No. Apple states that Applebot-Extended does not crawl webpages, that pages disallowing it can still appear in search results, and that its rules are not considered in search ranking. It only controls whether your content is used to train Apple's generative AI models.
What is the difference between Applebot and Applebot-Extended?
Applebot is Apple's real web crawler — it fetches pages for Siri, Spotlight, Safari, and Apple News. Applebot-Extended is not a crawler; it's a robots.txt token Apple reads as a permission signal for AI training use of already-crawled content. One controls visibility, the other controls training.
Can I block Applebot-Extended at the firewall or in .htaccess?
No — there's nothing to block. Since Applebot-Extended never makes HTTP requests, it has no user-agent string or IP range. It exists only as a robots.txt directive. Server-level blocking can only target the real Applebot crawler.
How do I verify that Applebot traffic in my logs is genuine?
Do a reverse DNS lookup on the visitor's IP and confirm the hostname ends in applebot.apple.com (optionally forward-resolve it back to the same IP). Don't trust the user-agent string alone — scrapers frequently impersonate major crawlers including Applebot.
Should I also block Google-Extended and other training tokens?
Apply the same logic per vendor: each -Extended token (Google-Extended, Applebot-Extended) is an independent training-permission switch with no search effect. If your policy is "no AI training use," disallow each vendor's token individually while leaving their search crawlers allowed.
