Short answer: nobody can prove it. OpenAI, Anthropic, and Perplexity have not said publicly whether their crawlers fetch sitemap.xml. Google is the one crawler we know reads sitemaps for sure, because Google's own documentation says so.
But "undocumented" is not the same as "useless." A sitemap.xml is the simplest way to hand any crawler a complete list of your pages — including bots that follow your internal links poorly or not at all. If you want AI crawlers to find your site, a clean sitemap announced in your robots.txt is the cheapest step you can take. Here is what we actually know, and how to set yours up.
Which AI bots are crawling your site right now?
Check your server logs and you will probably find some of these user agents. Here are the names that matter, and what each one does:
- GPTBot — OpenAI's crawler, used to collect training data for its AI models. OpenAI says it identifies itself with the GPTBot user agent and respects robots.txt.
- OAI-SearchBot — OpenAI's crawler that feeds live search results in ChatGPT, separate from the training-data bot.
- ChatGPT-User — fetches pages on demand when a ChatGPT user asks it to look something up. It only visits a page someone requested, so it will never be your biggest traffic source.
- CCBot — the crawler behind Common Crawl, the nonprofit web crawl whose petabytes of data feed the training pipelines of many AI companies. If a crawler's output trains models, this one matters more than most.
- ClaudeBot — Anthropic's crawler. Anthropic documents its crawler types, including ClaudeBot and Anthropic-AI, and says they honor robots.txt rules.
- PerplexityBot — Perplexity's crawler, feeding its AI answer engine. Perplexity documents the bot and says it respects robots.txt.
- Bytespider — ByteDance's crawler, widely reported as being used for both search indexing and AI training data collection.
- Google-Extended — not a crawler at all, but a token you can put in robots.txt that controls whether Google may use your pages to train its AI models. Googlebot itself (which definitely reads sitemaps) is a separate thing.
Notice the pattern: every one of these companies documents that their bots identify themselves and honor robots.txt. None of them documents whether the bot fetches sitemap.xml. That is the honest state of public knowledge, and everything else in this post is built on that line.
What AI crawlers actually do to discover pages
Strip away the AI branding and these are ordinary web crawlers. They start with a set of seed URLs, download pages, and follow links. Common Crawl publishes the raw crawl output — the HTML, the HTTP headers, the link graph — and researchers and AI labs use that data to train models. URL discovery happens through the link graph: a page that nothing links to is a page that crawlers struggle to find.
A sitemap.xml hands a crawler the answer directly: a list of URLs plus the date each one was last modified (lastmod). Google's documentation says sitemaps help crawlers discover pages that are not well linked internally and help them crawl large sites more efficiently. That logic is not Google-specific. Any crawler that downloads the file gets the same benefit.
So here is the realistic chain of events. AI bots fetch robots.txt — site owners regularly report seeing GPTBot and CCBot request it. Your robots.txt can contain a Sitemap: line pointing at your sitemap's full URL. A bot that reads robots.txt can therefore find your sitemap without guessing. Whether it then downloads and parses the sitemap is the part nobody outside the AI companies can confirm. You control the first two links in that chain; the third is out of your hands.
One more reality check: a sitemap helps discovery, not usage. Getting a URL into a crawl is one thing. Whether that URL's content ends up in training data or in an AI answer is governed by robots.txt rules, terms of use, and the AI company's own policies — not by your sitemap.
How to tell if AI bots are reading your sitemap
You do not have to guess. Your server's access logs record every request, including the user agent and the URL requested. Search your logs for the bot names above paired with requests for your sitemap file:
grep -i "gptbot" /var/log/nginx/access.log | grep "sitemap"If you see entries showing GPTBot or CCBot requesting /sitemap.xml or /robots.txt, that bot is at least aware of your sitemap. On shared hosting without log access, check your CDN or security plugin's bot analytics — many show bot hits broken down by user agent.
A few notes on reading the results. First, absence of evidence is not evidence of absence: a bot may fetch your sitemap rarely, outside your log window. Second, treat hits on robots.txt as the baseline — if a bot never even requests robots.txt, it is not following the standard discovery path at all. Third, do not trust user agents blindly; anyone can spoof "GPTBot" in a request header. Real crawler traffic comes from the IP ranges or verification methods each company publishes — OpenAI, for example, documents that you can verify GPTBot requests with reverse DNS. Log analysis tells you what is happening on your site specifically, which is more useful than any general claim about what these bots "usually" do.
How to optimize your sitemap for AI discovery
Everything below is standard, documented sitemap practice. It is verified to help with Googlebot, and it is the best available bet for AI crawlers too.
Step 1: Generate a complete, valid sitemap
Your sitemap must contain every indexable URL you want discovered, with absolute URLs in each <loc> tag, truthful <lastmod> dates, and a single <urlset> root element. Keep each sitemap file under 50,000 URLs and 50 MB uncompressed — those are the hard limits in the sitemap protocol, and oversized files get truncated or skipped. If building one by hand sounds painful, use our free sitemap generator — paste your pages in, and it produces a valid sitemap.xml in seconds.
Step 2: Announce the sitemap in robots.txt
Add a Sitemap: line to your robots.txt with the full URL:
User-agent: *
Sitemap: https://example.com/sitemap.xmlThis is the one discovery mechanism that is part of the sitemap protocol itself, which means any bot author who implements sitemap support will look for it exactly here. Since every major AI bot reads robots.txt, this is your best shot at getting AI crawlers to notice the sitemap. Use a robots.txt generator to get the syntax right, and if the directives confuse you, read our robots.txt directives explained guide before you touch the file.
Step 3: Keep lastmod honest
AI companies are obsessed with fresh data — training cutoffs and freshness signals both depend on it. A lastmod tag that claims every page changed today, when nothing changed, teaches crawlers to ignore your dates entirely. Update lastmod only when the content actually changed in a meaningful way. Truthful dates make crawlers (any crawlers) spend their limited crawl budget on your genuinely new pages instead of re-downloading identical ones.
Step 4: Split big sites with a sitemap index
If your site is anywhere near the 50,000-URL or 50 MB limits, do not cram everything into one file. Create per-section sitemaps (blog, products, help docs) and list them all in a sitemap index file using <sitemapindex>, then point the robots.txt Sitemap: line at the index. Crawlers handle small, clean files far more reliably than one giant dump.
Step 5: Test before you rely on it
Fetch your sitemap URL yourself and check three things: it returns HTTP 200, the content type is application/xml (or text/xml), and the XML parses without errors. Then confirm the Sitemap: line is visible in your robots.txt — our robots.txt tester shows you exactly what crawlers see. A sitemap that 404s or serves broken XML helps nobody, human or bot.
Step 6: Don't accidentally block the bots you want
Dig up your robots.txt and look for blanket blocks you added during last year's bot panic. If you disallowed GPTBot, CCBot, or PerplexityBot back then, those bots cannot fetch your sitemap or anything else — make that a deliberate decision, not a forgotten one. And keep Google-Extended straight: disallowing it stops Google from using your content to train AI models, but it does not stop Googlebot from crawling you for search. Search visibility and AI Overviews still depend on normal crawling and indexing.
What won't help (myths to drop)
- Tuning
priorityandchangefreq: Google has said for years that it ignorespriority, andchangefreqis at best a weak hint. No AI company has claimed to use either. Spend your time onlastmodinstead. - Manually pinging search engines: the old ping endpoints are long gone or deprecated. For Google, submit the sitemap in Search Console and keep the robots.txt
Sitemap:line in place. - Treating llms.txt as a sitemap replacement: llms.txt is an emerging convention for describing your content to language models, but it is not a URL discovery format. Sitemap.xml lists your URLs; llms.txt describes them. They do different jobs — run both.
Frequently asked questions
Do GPTBot and PerplexityBot actually read sitemap.xml?
Neither company has confirmed it publicly. What is confirmed: both bots respect robots.txt, and robots.txt is where sitemaps are announced via the Sitemap: directive. So a sitemap is cheap insurance for AI discovery — plausibly helpful, not guaranteed. Build it for the crawlers you can verify (Googlebot), and treat AI discovery as the bonus.
Should I list my sitemap in robots.txt?
Yes. The Sitemap: directive is part of the sitemap protocol itself, so it is the standard way crawlers find your sitemap without guessing its URL. One line, zero downside.
Do AI crawlers respect robots.txt?
The major ones do. OpenAI, Anthropic, Perplexity, and Common Crawl each document that their crawlers identify themselves with their own user agents and honor robots.txt rules. If you want an AI bot off your site, a robots.txt disallow aimed at its user agent is the documented route.
Does blocking Google-Extended remove my site from AI results?
No. Google-Extended only controls whether Google may use your content to train its AI models (including Vertex AI). Crawling for Google Search is handled by Googlebot and governed separately. Your visibility in search results and AI Overviews depends on being crawled and indexed normally — blocking Google-Extended does not take you out of the index.
How often should I update my sitemap?
Regenerate it (ideally automatically) whenever pages are added, removed, or meaningfully updated, and keep every lastmod date truthful. A stale sitemap with fake dates erodes crawler trust; an accurate one focuses crawl budget on what is actually new.
The bottom line: no AI company publishes whether its crawler reads sitemap.xml, so build the sitemap for the crawlers you can verify, announce it in robots.txt where every honest bot looks, and treat AI discovery as a bonus. It costs you an afternoon, and it is the only sitemap work that matters.
