Do AI Bots Actually Read llms.txt? Four Log Studies Say Almost Never

No. Almost never — and there is now enough independent log evidence to say that plainly. Across five separate studies published between April and October 2026, AI crawlers fetched llms.txt either zero times on sites that had it, or a handful of times next to thousands of robots.txt requests. If you paid for an llms.txt generator expecting ChatGPT to start reading your site differently, you bought a bet, not a channel. That said, the file still earns its thirty minutes — just not for the reasons the ads promised. Here is what the logs actually show, why the hype got ahead of the evidence, and where your effort belongs instead.

The Numbers: Five Studies, One Pattern

None of these are CrawlGuards' own logs. They are third-party studies from independent researchers, an SEO tool vendor, and two site operators who published their raw counts. Taken together, the pattern is unmistakable.

GenSight: 1,687 AI-crawler visits, zero llms.txt fetches

GenSight published its own crawler logs for a 36-day window, from 6 September to 11 October 2026. In 1,687 AI-crawler visits, the file robots.txt was fetched 346 times, sitemap files 96 times, and llms.txt — not once. Their conclusion was blunt enough that they are now reducing llms.txt's weight in their own AI-visibility scoring: the file, in their words, is "a cheap bet on how things might develop rather than a fix for visibility now." That line is the most honest single sentence written about this file all year.

Prosopo: IP-verified logs, same result

Prosopo has published an llms.txt file since May 2026 — so any crawler that wanted the file had five months to fetch it. Across four days of IP-verified logs (meaning the bot identities were checked, not just assumed from user-agent strings), AI crawlers requested robots.txt 120 times and the sitemap 43 times. The llms.txt count was zero. Same site, same crawlers, same time window: the bots that bothered to check the rules file never touched the supposedly AI-optimized one.

Ahrefs: 137,000 sites, 97% silence

The June 2026 Ahrefs study was the biggest measurement yet: 137,000 sites that had llms.txt files. Ninety-seven percent of those files received zero requests in the preceding month. Of the requests that did arrive, only 1.1% came from AI retrieval bots — the rest were humans, SEO tooling, and unrelated crawlers poking at a filename they had seen mentioned online. In other words, the overwhelming majority of llms.txt files sit on servers being read by nobody, and even the few requests that exist are mostly curiosity traffic from the SEO industry itself.

EZY Research: OpenAI read robots.txt 3,990 times, llms.txt 7

In July 2026, EZY Research looked at 83 websites and compared how often OpenAI's crawlers requested the two files. Robots.txt was fetched 3,990 times. The llms.txt file was fetched 7 times — all of them by OpenAI's search-facing crawlers rather than its training crawlers. A ratio of roughly 570 to 1. That is not a reading habit; that is a rounding error.

Unsourced: 1,227 requests, zero frontier-lab crawlers

An independent analysis published 8 September 2026 tracked llms.txt-family requests across roughly 900 domains from September 2025 to April 2026. It logged 1,227 requests in total — and exactly zero of them came from verified frontier-lab crawlers: not GPTBot, not ClaudeBot, not PerplexityBot, not Google-Extended. About two-thirds of the requests came from commercial data aggregators, about a third from human browsers, and a handful from security scanners. The bots everyone is hoping to influence simply did not show up.

How llms.txt Got Mis-sold as an AI-Visibility Lever

The file itself is not the problem. It was proposed in September 2024 by Jeremy Howard (of Answer.AI, at llmstxt.org) as a Markdown index that lives at the root of a website — a compact summary of a site's pages, meant to help AI coding assistants read documentation without burning thousands of tokens. For that job, it works well. Developer tools genuinely do read it.

The mislabeling started later, when parts of the SEO industry picked up the file and renamed it, in effect, "the new robots.txt for AI." Vendors began selling llms.txt generators, audits, and "GEO packs" on the implication that adding the file would improve visibility in ChatGPT answers, AI Overviews, and Perplexity results. The claim was always speculation: no frontier lab has ever documented reading llms.txt, and the proposal was never designed as a crawl-control file. It controls nothing and blocks nothing — it is not a robots.txt-style directive, and it cannot keep a crawler out or pull one in.

To be fair, the file's own spec never promised any of this. The hope was understandable: site owners had been given real control through robots.txt for thirty years, and they wanted something equivalent for the AI era. But the industry relabeled a developer convenience as a marketing lever, and the logs above are the receipt.

What Google Actually Says About llms.txt

Google has been unusually direct about this file. In May 2026 guidance mythbusting AI-search rumors, Google stated that you do not need machine-readable files like llms.txt to appear in generative AI features. Search Central's position is that Google ignores AI text files including llms.txt for search, and it has no plans to support the file. This is not a quiet FAQ entry — it is an on-the-record statement of current and planned behavior. If you want the full breakdown of Google's documented position, we covered it in our guide to Google's stance on llms.txt.

Google's John Mueller put it even more plainly: llms.txt is "not done for search," and he described it as a "temporary crutch" for AI coding tools that need to parse developer documentation. Note the framing — the file exists for tools that consume docs, not for search engines that rank pages. When the person whose team builds the crawler tells you the file is not for search, believe him.

One more distinction worth keeping straight: robots.txt is the control mechanism every AI vendor documents. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and the rest all check robots.txt before crawling, and they publish how to block or allow them there. No vendor has published equivalent documentation for llms.txt, because reading it is not part of their retrieval pipeline.

Why It's Still Worth Thirty Minutes

Here is where the honest contrarian take differs from the "rip out your llms.txt" take: keep the file. Just stop treating it as a channel.

First, it is cheap. Generating a decent llms.txt takes about half an hour, most of which is copy-pasting your key pages into a generator and sanity-checking the links. The cost of skipping it is zero, but the cost of adding it is near-zero too — and the file is a bet on how things might develop. If even one major lab adds llms.txt reading to its retrieval pipeline next year, early adopters with a good file are ahead. GenSight's "cheap bet" framing is exactly right: a lottery ticket with a thirty-minute price tag is not a bad purchase, as long as you know it is a lottery ticket.

Second, developer-tool readers are real. AI coding assistants that read project documentation do respect the convention — Howard designed it for them, and they are the one audience the logs consistently show up for. If your site has technical docs, that alone may justify the file.

Third, there is a mild hygiene effect. New agentic auditing tools are starting to check for llms.txt as part of AI-readiness scores. The score itself is marketing, but a missing file in an audit can create a pointless argument with a client or a boss. Having the file filed and forgotten is simpler.

The rule is proportion. Give llms.txt the effort of a small chore — thirty minutes, once, then a quick check when your site structure changes — not the effort of a marketing channel.

The Priority Stack: Where Your GEO Hours Should Actually Go

If you have five hours a week for AI-search work, spend them in this order. This is the honest hierarchy the log data supports.

1. Robots.txt — the file every AI crawler actually reads. This is the single file the studies agree on: 346 fetches at GenSight, 120 at Prosopo, 3,990 at EZY. Every documented AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider — checks robots.txt, and the vendors publish exactly how to block or allow them there. A clean, deliberate robots.txt with separate stanzas for training crawlers and search-facing crawlers is the highest-leverage AI-search file you can maintain. Test it properly before you publish it.

2. Your XML sitemap — AI retrieval bots do read sitemaps. Sitemaps got 96 requests at GenSight and 43 at Prosopo, against zero for llms.txt. Sitemaps are a standard every crawler implements; they tell bots what exists and what changed. Keep yours accurate, current, and referenced from robots.txt. That convention is three decades old and every AI crawler honors it.

3. Plain, citable HTML pages. Models cite pages, not index files. The single biggest driver of appearing in AI answers is having clear, factual, well-structured pages that answer real questions — the same work that wins featured snippets and PAA boxes. Every hour spent rewriting a thin page into a genuinely useful one beats every hour spent polishing an llms.txt that no bot fetches.

4. Deliberate bot allowlists. Decide which AI crawlers you actually want. Blocking GPTBot (training) while allowing OAI-SearchBot (ChatGPT citations) is a real, documented strategy — they are separate user agents with separate jobs. The same logic applies to ClaudeBot vs. Anthropic's search crawlers and PerplexityBot. This is control you can verify in your own server logs; llms.txt gives you no equivalent lever.

5. llms.txt — last, as a cheap lottery ticket. Only after the four above are done. Thirty minutes, a generator, a spec check, and done. Our free llms.txt generator builds the file in a couple of minutes if you want to check this box without overthinking it.

When You Can Skip It Entirely

Not every site needs even the thirty minutes. If your site is tiny — a handful of static pages — a bot does not need an index file; it can crawl the whole site in seconds. If your content goes stale quickly and nobody updates the file, an out-of-date llms.txt is worse than none at all, because it points tools at dead links. And if you have bigger, verified gaps — no sitemap, a broken robots.txt, pages that fail to get indexed at all — every minute spent on llms.txt is a minute stolen from the thing actually hurting you. Do the real fixes first, in the order above.

Adding One Properly Anyway (Once, Then Forget It)

If you have decided the lottery ticket is worth it, do it right once so you never have to think about it again.

Generate the file from your actual site structure rather than writing it by hand — a generator pulls your real page titles and URLs, which is the part humans get wrong. Upload the result to your domain root, so it is served at yoursite.com/llms.txt; the location is the entire convention, and a file buried in a subfolder is not a real llms.txt. WordPress users can follow our WordPress walkthrough for the exact steps.

Then validate it against the spec before you trust it. There is no official validator and no search console that checks the file, so validation is a manual checklist: one H1 with the site name, a summary blockquote, H2 sections of properly formatted Markdown links, and every link absolute and actually working. Our twelve-point llms.txt validation checklist walks through the whole test in about fifteen minutes.

A minimal file looks like this:

# Example Site
> Example Site is a free tool site for SEO and AI-crawler control.
## Tools
- [Robots.txt Generator](https://example.com/tools/robots-generator): Build a robots.txt with AI-crawler rules.
- [llms.txt Generator](https://example.com/tools/llms-generator): Create an llms.txt file for AI assistants.
## Guides
- [How to Block AI Crawlers](https://example.com/blog/how-to-block-ai-crawlers): Step-by-step robots.txt rules for GPTBot, ClaudeBot, and PerplexityBot.

After that, put it on a maintenance trigger, not a schedule: revisit the file whenever your navigation or core pages change, and ignore it the rest of the time. A file that reflects your site as it is today will be there waiting if the day ever comes when a frontier lab starts reading it.

FAQ

Do AI bots read llms.txt at all?

Effectively no, at scale. Five independent studies found either zero fetches (GenSight, Prosopo, Unsourced's frontier-lab count) or negligible numbers (EZY's 7 OpenAI fetches against 3,990 robots.txt requests; Ahrefs' 97% of files with zero requests). A handful of requests do exist — Ahrefs found 1.1% of requests came from AI retrieval bots — but at the scale of real crawler traffic, the file is unread.

Does llms.txt help SEO?

Google says no, explicitly: you do not need machine-readable files like llms.txt to appear in generative AI features, and Google ignores the file for search with no plans to support it. Any benefit is indirect and speculative — a future possibility, not a current lever.

Is llms.txt a replacement for robots.txt?

No, and it was never designed to be one. llms.txt controls nothing and blocks nothing; it is a Markdown summary, not a directive. Robots.txt remains the only file every AI vendor documents and honors for crawl control.

Should I still add llms.txt to my site in 2026?

Yes — if you have thirty spare minutes and your real foundations are done. Treat it as a cheap lottery ticket: low cost, possible future upside, no current downside. If your robots.txt is broken or your sitemap is missing, fix those first; they are the files bots actually read.

Which files do AI crawlers actually read?

Robots.txt first (every documented AI crawler checks it), then XML sitemaps, then your HTML pages themselves. That is the reading order the log studies show, and it is where all of your GEO effort should go before you touch llms.txt.