llms.txt is a short Markdown index that lists your site's most important pages with descriptions and links. llms-full.txt is its optional companion: a single file containing the full text of those pages, pasted in one place so an AI agent can read everything without following links. If your site is large or changes often, publish llms.txt. If your content is compact and rarely changes, you can publish both. This guide explains the difference, shows working samples of each, and helps you decide which your site actually needs.
What llms.txt is
llms.txt is a proposed web convention, put forward in September 2024 by Jeremy Howard (Answer.AI, fast.ai), with the spec maintained at llmstxt.org. The idea is simple: web pages are built for humans, wrapped in navigation, ads, and JavaScript. When an AI agent — a coding assistant fetching library docs, a chat assistant researching a product — needs to understand your site, converting HTML back into clean text is slow and lossy. Context windows are bigger than they used to be, but still too small for most whole websites, and every wasted token costs time and money.
So instead of asking agents to crawl your site like a search engine, you hand them a short Markdown file that says: here is what this site is about, here are the pages that matter, and here is what each one covers.
The format is deliberately strict. The current (v2) spec describes the structure in this order: an H1 with the project or site name (the only required section), a blockquote with a short summary, zero or more detail sections without headings, and then H2 sections containing file lists — bullet entries in the shape - [Title](URL): short description. There is a convention for a final H2 called "Optional" for links an agent may skip when it wants a shorter context. Links in the file should point to LLM-friendly content, ideally clean Markdown versions of your pages.
One honest caveat: llms.txt is a proposed convention, not an official standard. It was never ratified by the W3C or IETF, and no major AI platform treats it as a ranking signal. There is no verified evidence that publishing one improves your visibility inside ChatGPT, Perplexity, or Google AI Overviews. It is curation and orientation — a courtesy for agents — not a magic visibility lever, and definitely not an opt-out mechanism (that is what robots.txt is for). Adoption is early but real: thousands of sites now publish one, platforms like Mintlify, GitBook, Wix, Yoast, and AIOSEO generate it automatically, and Chrome's Lighthouse audits sites for it as part of its agentic browsing checks.
What llms-full.txt is
llms-full.txt came out of the original 2024 proposal: while /llms.txt serves as the navigation aid, /llms-full.txt was defined as an optional companion containing the full content of the site's key pages concatenated into a single Markdown document. Instead of an index of links, the LLM gets the entire readable substance in one request — no following links, no second round trip.
Here is where people get confused, so let me be precise. The current v2 specification at llmstxt.org focuses on llms.txt plus Markdown versions of individual pages (the .md page convention with rel="alternate" discovery). The v2 spec page no longer describes llms-full.txt. That does not mean the file is wrong to publish — plenty of generators and site owners still create it, and some tooling produces both variants automatically — but it does mean llms-full.txt is a practical convention rather than part of the current written spec. Treat it as an optional extra, not a requirement.
In practice, llms-full.txt shines in specific situations: small documentation sites, knowledge bases, or FAQ pages where the whole useful content fits comfortably in one file; agents operating offline or in restricted environments where they cannot follow outbound links; and tools that want to load your site into a context window in a single fetch. For a large blog, e-commerce catalog, or news site, a single concatenated file would be enormous and pointless — the index form is the right choice.
llms.txt vs llms-full.txt: side by side
| llms.txt | llms-full.txt | |
|---|---|---|
| Purpose | Curated index: tells an agent which pages exist and what each covers | Full content: the actual text of key pages in one file |
| Size | Small — dozens of links, fits easily in any context window | Large — grows with every page you include |
| Format | H1 title, blockquote summary, H2 sections with - [Title](URL): description entries | Same header structure, but each entry is followed by the full page content in Markdown |
| Maintenance | Low — only update when pages are added, removed, or repurposed | Higher — must be regenerated every time page content changes |
| Agent behavior | Agent reads the index, then fetches only the relevant pages | Agent reads everything at once, no link-following needed |
| Status | Described in the current v2 spec at llmstxt.org | From the original 2024 proposal; practical convention, not in the v2 spec text |
| Required? | The file to start with — this is the standard form | Optional companion; publish only if you have a use case for it |
A realistic example: llms.txt
Below is a realistic llms.txt for a fictional bike shop, Northwind Cycles. (The domain uses the reserved .example suffix, so these links are illustrative, not real.) Notice how short it is, how every link has a plain-language description, and how the "Optional" section holds secondary pages.
# Northwind Cycles
> Independent bike shop in Portland selling commuter, gravel, and kids' bikes, with an on-site repair workshop and free 30-day tune-ups.
Northwind Cycles is a single-store retailer. Prices include free assembly. All repairs are done in-house with same-week turnaround.
## Bikes
- [Commuter bikes](https://northwindcycles.example.com/bikes/commuter.md): Step-through and flat-bar models for city riding, priced $499-$1,299.
- [Gravel bikes](https://northwindcycles.example.com/bikes/gravel.md): Drop-bar adventure bikes with tire clearance up to 45mm.
- [Kids' bikes](https://northwindcycles.example.com/bikes/kids.md): Balance bikes through 24-inch wheel models, with trade-in program details.
## Repair workshop
- [Repair services and pricing](https://northwindcycles.example.com/repair.md): Full tune-up, brake bleed, wheel truing, and e-bike service prices.
- [Book a repair](https://northwindcycles.example.com/repair/booking.md): Online booking form, current lead times, and drop-off hours.
## Optional
- [About the shop](https://northwindcycles.example.com/about.md): History of the store and the team.
- [Return policy](https://northwindcycles.example.com/returns.md): 30-day returns on bikes, 14 days on parts.
That whole file is under 200 words. An agent can read it in a fraction of a second, decide which one or two pages matter for the user's question, and fetch only those. That is the entire point of the design.
The same site as llms-full.txt
Now the same content as llms-full.txt. The header is the same, but instead of descriptions pointing elsewhere, each section carries the complete page text inline:
# Northwind Cycles
> Independent bike shop in Portland selling commuter, gravel, and kids' bikes, with an on-site repair workshop and free 30-day tune-ups.
## Bikes
### Commuter bikes
Our commuter range covers step-through and flat-bar models built for daily city riding. Every commuter bike is sold fully assembled and includes a free 30-day tune-up. Prices run from $499 for the Cityline 1 (single-speed, steel frame) to $1,299 for the Cityline 7 (7-speed internal hub, dynamo lights). All models accept full fenders and rear racks; pannier bundles ship free.
Sizing: measure your inseam and match it to the chart below. Between sizes? We recommend sizing down for city riding, since frequent stops favor a lower standover.
- Cityline 1 — $499 — single speed — best for flat routes
- Cityline 3 — $749 — 3-speed hub — best all-rounder
- Cityline 7 — $1,299 — 7-speed hub — hilly commutes
### Gravel bikes
[full gravel page content would continue here — specs, tire clearance, geometry chart]
## Repair workshop
### Repair services and pricing
[full repair pricing page content would continue here — service menu, lead times, booking notes]
Two things to notice. First, the file is much longer and grows every time you add or edit a page — on a real site with 50 pages this could run to tens of thousands of words. Second, it is a curated dump, not a raw copy of your database: you pick the pages an agent would actually need, convert them to clean Markdown, and leave out the shopping cart, account pages, and legal boilerplate. Pasting raw HTML or dumping your entire sitemap into one file is the most common way people do this wrong — it bloats the file and teaches the agent nothing.
Which one should your site use?
Start with llms.txt. It is the spec'd format, it is cheap to maintain, and it works for sites of any size. Publish it at /llms.txt at your domain root — the same placement convention as robots.txt — served as plain Markdown. No special headers are required, though you can declare it with a rel="describedby" link so agents can discover it from any page.
Add llms-full.txt when
- Your key content is genuinely compact. A docs site with 20 short pages, a SaaS product's help center, a restaurant's menus and policies — anything where the full text stays under roughly 50,000 words.
- Agents cannot follow your links. Offline tools, sandboxed assistants, or pipelines that ingest one file per source benefit from the single-file form.
- Your generator produces it for free. Some static-site plugins (for example Hugo modules and docs frameworks) emit both files at build time. If it costs you nothing, publishing it hurts nothing.
Skip llms-full.txt when
- Your site is large. Concatenating 500 blog posts helps nobody; the index form exists precisely for this case.
- Content changes daily. News sites and busy blogs would need to regenerate the file constantly. The index stays accurate far longer.
- You are following the v2 spec closely. The current spec's recommended pattern is llms.txt linking to Markdown page versions (
page.html.mdstyle). That pattern already gives agents clean full content on demand, which is what llms-full.txt was trying to achieve.
The honest rule of thumb: llms.txt is the map; llms-full.txt is a photocopy of the whole territory. Most sites need the map. Only hand out the photocopy when the territory is small enough to carry.
How to build both files in minutes
You do not need to hand-write these files. The fastest path is a generator: crawl your key pages, write human descriptions, and output spec-shaped Markdown. If you want to check what your site currently exposes to AI agents, start with the free llms.txt generator, which builds a properly structured file from your pages.
A practical workflow that works well:
- List candidate pages — pull your sitemap (the sitemap generator can build one if you lack it) and pick the 20-60 pages an agent would genuinely need: product overviews, pricing, docs, FAQs, policies.
- Write one-line descriptions — for each page, one sentence a stranger could understand. "Subscription tiers and feature comparison" beats "Pricing information."
- Generate the file — group links under H2 sections, add your H1 and blockquote summary, and put secondary pages under an "Optional" H2.
- Verify AI crawlers can actually reach it — use the robots.txt tester to confirm your robots rules do not accidentally block the paths your llms.txt points to. An llms.txt that links to disallowed pages is self-defeating.
- Publish at /llms.txt and spot-check monthly — fetch the file in a browser, confirm it renders as Markdown, and update descriptions when pages change purpose.
If you decide you also want llms-full.txt, generate it from the same page list: strip the navigation and chrome from each page, keep the Markdown body, and concatenate under matching H2/H3 headings. Regenerate it on the same schedule you update your docs — a stale full dump is worse than none, because agents will quote outdated prices and policies as if they were current.
Common mistakes that make these files useless
- Dumping raw HTML. The whole point is clean Markdown. An llms-full.txt full of div soup costs the agent tokens and yields confusion.
- Listing every URL. llms.txt is not a sitemap. If it contains 2,000 links, it is not an index anymore — curate the 5% of pages that answer 95% of questions.
- Using it as a blocking tool. llms.txt does not stop crawlers or opt you out of AI training. It is orientation, not access control.
- Wrong filenames. The convention is
llms.txtandllms-full.txtat the path they describe (usually the root). Variants likellm.txtare not the canonical names and agents will not look for them. - Vague descriptions. "Click here for more info" teaches an agent nothing. Each link needs a description that lets the agent decide whether to fetch it.
- Inventing adoption claims. Do not write on your site that "all major AI models read llms.txt" — that is not verifiable. The file is low-cost future-proofing, not a guaranteed pipeline into any model.
FAQ
Is llms-full.txt required, or is llms.txt enough?
llms.txt alone is enough for most sites. llms-full.txt is an optional companion from the original 2024 proposal, and the current v2 spec does not describe it at all — it recommends linking to Markdown page versions instead. Publish llms.txt first; add llms-full.txt only if your content is compact or your agents cannot follow links.
Does llms.txt stop AI companies from training on my content?
No. llms.txt is not an access-control or opt-out mechanism. Crawling and training are governed by robots.txt rules, user-agent blocks, and each platform's policies. llms.txt is curation — it tells cooperative agents where your best content is — not a fence. For blocking, see our guide to controlling AI crawlers with robots.txt.
Will adding llms.txt improve my rankings in ChatGPT, Perplexity, or Google AI Overviews?
There is no verified evidence that it does. No major answer engine has announced that llms.txt influences which sources get cited. The file's value is practical: when an agent does visit your site, it can find the right page faster and with fewer wasted tokens. Treat it as good infrastructure, not a ranking tactic.
My whole site is just 15 pages. Should I publish both files?
Yes, that is the ideal case for both. With 15 pages the full dump stays small, and some agents will happily ingest it in one request. Keep both files in sync: every time you edit a page, regenerate llms-full.txt and re-check that the llms.txt descriptions still match.
What content type should I serve these files with?
Serve them as Markdown — text/markdown is the ideal content type — but plain text/plain works fine in practice. What matters is the file being clean Markdown at a predictable URL, not the header. Avoid serving them as HTML pages with site chrome around them.
Should the links in my llms.txt point to .md versions of my pages?
Yes, when you can. The v2 spec recommends providing clean Markdown versions of pages (same URL with .md appended or the extension swapped) and advertising them with rel="alternate" type="text/markdown". Pointing your llms.txt links at those Markdown versions means the agent gets clean text on the first fetch — which is exactly the convenience llms-full.txt was trying to provide, without the maintenance burden of a second giant file.
The bottom line: llms.txt is the short index, llms-full.txt is the full-content companion, and most sites only need the index. Build the index first, keep it curated and honest, and add the full dump only when your site is small enough — or your tooling free enough — to justify it. Check the official specification at llmstxt.org when you want the exact format, and read Google's robots.txt documentation to keep straight which file does the blocking and which does the guiding.
