Documentation sites are the one place where llms.txt makes obvious, immediate sense. The file was invented for docs: a short, structured index that lets an AI model find the right page without chewing through your whole site. If you run API docs, a developer portal, or a help center, an llms.txt file is a cheap, low-risk way to make your docs usable by the coding assistants your readers already use. This guide explains what to include, how to write one by hand, and how to keep it fresh as your docs change.
The short version: put a Markdown file called llms.txt at your site root. Start with an H1 naming your project, add a short blockquote that says what the docs cover, then list your most important pages under H2 sections, each link with a one-line description. Add an llms-full.txt with the full text of your key pages if your docs are compact enough. That is the whole format — and for docs sites, it is genuinely useful even though no search engine has promised to use it for rankings.
Why docs sites are the natural home of llms.txt
llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024. The problem he described was narrow and practical: language models have limited context windows, and converting a cluttered documentation website into clean text a model can use is slow and error-prone. His reference implementation was FastHTML, a Python web framework, and the intended reader was a developer — or a coding agent working for a developer — looking for fast, accurate access to API documentation.
That origin matters, because it tells you where the file actually helps. A docs site is a bounded, structured knowledge base: getting-started guides, API references, tutorials, changelogs. A model that can read a clean index of that material can answer a developer's question with a link to the exact page, instead of guessing from a half-remembered training snapshot. A marketing site with the same file gets far less, because its pages are not the kind of reference material a coding assistant needs.
Adoption followed the docs platforms, not the marketing blogs. In November 2024, Mintlify — a platform that hosts developer documentation — began generating llms.txt automatically for every site it hosts. Within weeks, documentation sites for Anthropic, Stripe, Cursor and Zapier all had one, because their docs ran on a platform that shipped the feature by default. That is still where adoption is highest: developer docs, where the structure maps cleanly onto the format and the readers are the people most likely to paste a docs URL into an AI assistant.
What llms.txt is — and what it is not
Before you build one, get the expectations right, because the honest picture is mixed. Google has said publicly that it does not use llms.txt: Gary Illyes confirmed at a Google Search Central event in 2025 that Google does not support the file and has no plans to, and John Mueller has said no AI system currently uses it for search. Log studies back up the skepticism for general websites — one widely cited check found that 97 percent of sites with an llms.txt file saw zero requests for it.
None of that makes the file useless for docs. It makes it a documentation convenience, not a ranking lever. Coding assistants and developer tools do fetch it: Cursor uses llms.txt for project context, and documentation platforms generate it because their users ask for it. The value is in the workflow — a developer pastes your docs URL into an assistant, the assistant reads a clean index instead of scraping messy HTML — not in a citation boost you can measure in Search Console.
So write the file for the developer who will actually use it, not for a search engine. If it helps one developer find your authentication guide in ten seconds instead of ten minutes, it has done its job. Do not promise your team that it will improve rankings or AI citations, because the evidence does not support that claim.
The format, exactly as the spec defines it
The specification lives at llmstxt.org and it is deliberately small. A valid llms.txt has four parts:
- An H1 title naming the project or site.
- A blockquote summary — one or two sentences saying what the docs cover and who they are for.
- Optional H2 sections grouping links by topic, such as Getting Started, API Reference, or Guides.
- Link lists under each section: each line is a Markdown link with a short description of what the page contains.
There is also an extended variant, llms-full.txt, which inlines the full text of your pages instead of just linking to them. For a docs site with a compact knowledge base, the full file is the more useful of the two: a model can read the whole thing in one request. For a large docs site, the full file can get enormous, so the short index is the practical choice.
Here is a complete example for a fictional API product:
# Acme API Documentation
> Complete documentation for the Acme API: authentication, endpoints,
> webhooks, and SDKs for developers building billing integrations.
## Getting Started
- [Quickstart](https://docs.acme.com/quickstart): Create your first API key and make your first request in five minutes.
- [Authentication](https://docs.acme.com/auth): API keys, OAuth 2.0, and how to rotate credentials safely.
- [Errors and retries](https://docs.acme.com/errors): Every error code, what it means, and when to retry.
## API Reference
- [Customers](https://docs.acme.com/api/customers): Create, list, and update customer records.
- [Invoices](https://docs.acme.com/api/invoices): Issue invoices, apply discounts, and handle failed payments.
- [Webhooks](https://docs.acme.com/api/webhooks): Event types, signature verification, and replaying missed events.
## SDKs and Examples
- [Python SDK](https://docs.acme.com/sdks/python): Installation and the ten most common calls.
- [JavaScript SDK](https://docs.acme.com/sdks/javascript): Browser and Node usage with code samples.
## Optional
- [Changelog](https://docs.acme.com/changelog): Breaking changes and new endpoints, newest first.Notice what the file does not contain: no marketing copy, no navigation chrome, no footer links, no duplicate pages. Every link earns its place by answering a question a developer actually asks. That curation is the whole point — a model reading this file gets your docs' table of contents, not a sitemap dump.
What to include — and what to leave out
Start with the pages developers search for most. Your analytics will tell you: the quickstart, the authentication page, the API reference for your most-used endpoints, the errors page, and the webhook guide. Those five or six pages cover the majority of real questions. Then add the pages that prevent support tickets: rate limits, idempotency, sandbox versus production, and the changelog.
Leave out pages that do not help a developer do a task. Pricing pages, blog posts, careers pages, legal pages, and landing pages do not belong in a docs index. Neither do near-duplicate pages: if you have three pages that explain the same endpoint in slightly different words, link the canonical one and fix the duplication separately.
Write each link description as if it were the answer to 'what will I find on this page?' Keep it to one line, name the concrete thing — 'signature verification' beats 'everything about webhooks' — and use the same words your docs themselves use, so a model matching a question to a link is matching the vocabulary your readers already know.
How to generate one for your docs stack
The easiest path depends on where your docs live. If you host on Mintlify, you already have one: the platform generates llms.txt and llms-full.txt automatically for every project, and the files stay synced as your docs change. Check that yours exists by visiting your docs domain followed by /llms.txt.
For self-hosted docs, there are practical routes for the common frameworks. Docusaurus sites can generate the file with a community plugin or a small build script that walks your sidebar and emits the Markdown. MkDocs and static-site docs can do the same with a script that reads your navigation config. For WordPress-based docs, Yoast SEO added llms.txt generation in version 25.3, and it updates the file weekly.
If none of those fit, generate it with a tool and maintain it by hand. Our free llms.txt generator builds a spec-compliant file from your key pages — paste in your most important URLs with one-line descriptions, and it produces a file you can drop at your site root. For a docs site under fifty pages, a hand-maintained file is entirely realistic: update it whenever you ship a new guide or rename a page, the same way you would update a sidebar.
Whatever route you take, the file must live at the root: https://docs.yoursite.com/llms.txt, not buried under /assets/ or /files/. Models and tools look for it at the standard location, and a file in the wrong place is the same as no file at all.
llms-full.txt: when the full file is worth it
The short file points at pages; the full file contains them. llms-full.txt inlines the complete Markdown of every page you list, so a model can read your entire docs set in a single request without fetching anything else. For a small docs site — a few dozen short pages — that is a genuinely better experience, and it is the variant Anthropic publishes for its own developer docs.
It has two costs. The first is size: a large docs site can produce a full file of several megabytes, which exceeds what most models will read in one go and becomes slow to serve. The second is freshness: every docs edit must flow into the full file, or the model will read stale instructions. Mintlify handles both automatically for its hosted sites. If you generate the file yourself, automate the build step or accept that the full file will drift out of date — and a stale full file is worse than no full file, because it teaches the model the wrong answers.
A reasonable rule: ship the short llms.txt for every docs site, and add llms-full.txt only if your docs are compact (under roughly a hundred short pages) and you can regenerate it on every deploy. If you cannot automate it, skip the full file and keep the index accurate instead.
Keeping it fresh as your docs change
An llms.txt file rots faster than a sitemap, because it contains human-written descriptions that go stale when pages are renamed or rewritten. Build the update into your docs workflow rather than treating it as a one-off task.
The simplest discipline is a checklist item in your docs pull-request template: 'If this change adds, renames, or removes a page, update llms.txt.' That catches the common case — a new guide that never gets linked, or a renamed authentication page that leaves a dead link in the index. A monthly link check catches the rest: fetch every URL in the file and confirm it still returns a 200. Our robots.txt tester is for robots files, but the same habit — verify before you publish — applies here: a dead link in your llms.txt tells a model your docs are unmaintained.
Also keep the file in sync with your robots.txt policy. If your docs block AI training crawlers in robots.txt — a reasonable choice covered in our guide to blocking AI crawlers with robots.txt — understand the distinction: robots.txt governs crawling, while llms.txt is a file you publish for models to read. The two are not contradictory. Many docs teams block bulk training crawlers while still publishing an llms.txt for the coding assistants their own users invoke. Decide the policy deliberately, document it, and do not assume one file cancels the other. For the broader picture of how the two files relate, read our comparison of llms.txt vs robots.txt.
Measuring whether anyone uses it
Do not expect Search Console to tell you anything — Google does not report llms.txt fetches. Your server logs do. Search your access logs for requests to /llms.txt and /llms-full.txt and look at the user agents: you will see a mix of coding assistants, documentation tools, and curious humans. Track the count monthly. For most docs sites the number is small, and that is fine — the file serves the developers who fetch it deliberately, not a mass audience.
A more useful signal is qualitative: ask in your docs feedback channel whether anyone pastes your docs URL into an AI assistant, and whether the answers improved after you shipped the file. If your support tickets about 'where is the auth guide' drop, the file is working. If nobody uses it after three months, the file cost you an afternoon and you can leave it in place — it is not hurting anything.
FAQ
Does llms.txt help my docs rank in Google?
No evidence supports that. Google has said it does not use llms.txt for search, and no provider has confirmed using it for citations. Treat it as a developer convenience, not an SEO tactic.
Should a docs site publish llms-full.txt too?
Yes, if your docs are compact and you can regenerate the file on every deploy. For large docs sites, the short index alone is the practical choice — a multi-megabyte full file is slow to serve and quick to go stale.
Does llms.txt replace my sitemap?
No. A sitemap tells search engines which pages exist so they can crawl and index them. llms.txt tells AI models which pages matter and what each one covers. They serve different readers, and a docs site should have both.
What if my docs are on Mintlify, Docusaurus, or ReadMe?
Mintlify generates both files automatically — just verify yours exists at /llms.txt. Docusaurus and similar frameworks need a plugin or a small build script; the output format is the same either way, so the guidance in this article still applies.
Can I include internal or draft docs in llms.txt?
Do not list anything you would not want an AI model to quote back to a user. The file is public by design. Internal runbooks, unreleased API endpoints, and draft pages belong behind authentication, not in a public index.
How often should I update llms.txt?
Update it in the same change that adds or renames a docs page, and run a link check monthly. A file with dead links is worse than no file, because it teaches models — and the developers reading their output — that your docs are out of date.
The bottom line
Docs sites are where llms.txt earns its keep. Write a short, curated index of the pages developers actually need, keep it accurate as your docs change, and add the full-text variant only if you can keep it fresh automatically. Skip the ranking promises — nobody has shown the file moves search visibility — and judge it by the only metric that matters for docs: whether developers find the right page faster.
