llms.txt is a simple Markdown file that tells AI systems what your site is about — nothing more, nothing less. Proposed by Jeremy Howard (founder of Answer.AI and fast.ai) in September 2024, with the specification published at llmstxt.org and revised to v2 in August 2026, it's a community proposal — not a web standard — for giving AI agents a clean, curated summary of your content. This guide covers the real format, what it can and can't do, and how to implement it honestly.
What llms.txt Actually Is
Documentation and content sites are built for humans: navigation menus, sidebars, ads, scripts, cookie banners. An AI agent that wants to understand your site has to download all that HTML, strip the chrome, and hope the result still makes sense. llms.txt proposes a shortcut: a single Markdown file at yourdomain.com/llms.txt that lists the documents worth reading.
- It's a proposal, not a standard. No IETF RFC, no W3C blessing. Adoption is voluntary and uneven.
- It grants nothing and denies nothing. It is not a permission system — access control stays in robots.txt.
- It's Markdown, not a config format. No YAML frontmatter, no "Permissions" sections, no machine-readable policy fields in the spec.
The Real Format (Spec v2)
Per the specification, the only required element is an H1 with the site or project name:
# Acme Widgets
> Acme Widgets sells industrial fasteners with same-day shipping across the US.
## Products
- [Product catalog](https://example.com/products): full catalog with specifications
- [Shipping policy](https://example.com/shipping): delivery times and rates
## Optional
- [Blog archive](https://example.com/blog): older posts, lower priority- H1 (required): your site or project name. The one thing every valid file must have.
- Blockquote: a one-or-two-line summary of what the site is.
- H2 sections: group important links by topic, in the form
- [name](url): one-line description. ## Optional: lower-priority links that agents short on context can skip.
What llms.txt Cannot Do (Read This Before the Hype)
- It cannot block crawlers. Writing "do not crawl /private/" in llms.txt accomplishes nothing. Access rules go in robots.txt.
- It is not a ranking factor. Google has told site owners plainly they don't need new machine-readable AI text files or markup to appear in AI Overviews or AI Mode.
- It does not guarantee AI citations. Many of the most-cited sites have no file at all.
- It has no Permissions section. Templates with "Training Allowed: No" blocks are invented, not spec. Usage policy belongs in ai.txt or your terms.
Who Actually Benefits
Most of llms.txt's real consumers today are coding agents — tools like Cursor and Claude Code index it during project setup. For documentation sites, developer tools, and API references, that alone justifies the ten minutes it takes. For a local bakery's brochure site, the payoff is thin — but the cost is also near zero.
Writing a Good llms.txt: Practical Rules
- Curate, don't dump. Link your 5–20 most important pages. A thousand-link file defeats the purpose.
- Write descriptions for agents. Good: "Troubleshooting guides for installation errors, organized by error code." Bad: "Our blog."
- Keep it small. A few kilobytes. If it needs pagination, it's too long.
- Keep it fresh. A stale file pointing at dead URLs is worse than none.
- Serve it correctly. At the domain root as
text/plainortext/markdown.
Implementation Across Platforms
- WordPress: create the file at your site root via your host's File Manager or FTP/SFTP. No plugin needed.
- Static sites (Next.js, Gatsby, Hugo): drop
llms.txtin the public/static folder. - Shopify: add it as a root-level file through theme code editing or an app that serves static files.
Rather than hand-writing it, our free llms.txt generator builds a spec-shaped file from your site's key pages in minutes. Pair it with a correct robots.txt and optionally ai.txt.
llms.txt vs robots.txt vs ai.txt: Which Does What
- robots.txt — access control. Tells crawlers which paths they may fetch. The bouncer at the door.
- ai.txt — usage policy. A machine-readable declaration of how AI systems may use your content.
- llms.txt — orientation. Tells AI systems what the site is and which pages matter most. The welcome brochure inside.
Common llms.txt Mistakes
- Dumping the whole sitemap in. Curate ruthlessly — agents have limited context.
- Inventing a Permissions section. Not in the spec; put usage policy in ai.txt instead.
- Writing it once and forgetting it. Review quarterly alongside your robots.txt.
- Expecting SEO miracles. It's a discoverability courtesy. The ranking work is still content quality and technical SEO.
Frequently Asked Questions
Is llms.txt an official web standard?
No. It's a community proposal by Jeremy Howard, published at llmstxt.org. Useful and growing, but not an IETF or W3C standard.
Will llms.txt help me rank in Google or get cited by ChatGPT?
There's no documented evidence it does either. Google says you don't need new AI-specific files for AI Overviews.
Can llms.txt block AI crawlers or declare training permissions?
No. It has no blocking capability and no Permissions section in the spec. Use robots.txt for access control.
What's the difference between llms.txt and llms-full.txt?
llms.txt is the curated shortlist (spec). llms-full.txt is a companion convention holding a site's full text in one Markdown file — practical only for small sites.
Should small sites bother?
If AI agents are a plausible audience for your content — yes, it takes minutes. Otherwise it's an optional nice-to-have.
