llms.txt vs robots.txt: What Each File Does (and Why You Probably Need Both)

robots.txt and llms.txt solve opposite problems. robots.txt tells crawlers where they are not allowed to go — it's a 30-year-old access convention. llms.txt tells AI systems what your site is — it's a 2024 proposal for a Markdown file that gives AI agents a clean, curated summary of your content. Neither replaces the other. If you run a site in 2026, you probably need robots.txt for crawler control and should consider llms.txt as a cheap discoverability upgrade.

What Is robots.txt?

robots.txt is one of the oldest conventions on the web. Proposed by Martijn Koster in 1994 and eventually standardized by the IETF as RFC 9309 in 2022, it is a plain-text file served at your domain's root (yourdomain.com/robots.txt) that tells crawlers which parts of your site they may or may not fetch.

A typical robots.txt looks like this:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Three things every site owner should understand about it:

• It is advisory, not a lock. Well-behaved crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot) honor it. Malicious scrapers ignore it completely. Anything truly private needs authentication.
• Disallow is not the same as noindex. Blocking a page in robots.txt stops crawlers from fetching it, but the URL can still appear in search results. If you want a page out of search results, use a noindex tag, not robots.txt.
• It is per-subdomain. Rules on example.com/robots.txt do not apply to blog.example.com — each host needs its own file.

robots.txt answers the question: "Who may crawl what?" It says nothing about what your content means.

What Is llms.txt?

llms.txt is much younger. It was proposed by Jeremy Howard (founder of Answer.AI and fast.ai) in a September 2024 post, and the specification — published at llmstxt.org — was revised to v2 in August 2026.

The problem it addresses is simple: documentation and content sites are built for humans, so an AI agent that wants to understand your site has to download HTML, strip navigation, ads, and scripts, and hope the result still makes sense. llms.txt proposes a shortcut: a single Markdown file at /llms.txt that lists the documents worth reading.

A basic llms.txt looks like this:

# Acme Widgets

> Acme Widgets sells industrial fasteners with same-day shipping across the US.

## Products

- [Product catalog](https://example.com/products): Full catalog with specifications
- [Shipping policy](https://example.com/shipping): Delivery times and rates

Per the spec, the only required element is the H1 with the site or project name. The recommended structure adds a blockquote summary and H2 sections containing link lists in the form - [name](url): notes.

llms.txt answers the question: "What is this site, and what should an AI read first?" It grants nothing, denies nothing, and enforces nothing.

llms.txt vs robots.txt: The Full Comparison

• Age: robots.txt dates to 1994 (standardized 2022); llms.txt was proposed in 2024
• Status: robots.txt is an IETF standard (RFC 9309); llms.txt is a community proposal, not a standard
• Location: both live at the domain root — /robots.txt and /llms.txt
• Format: robots.txt uses its own directive syntax; llms.txt is Markdown
• Audience: robots.txt speaks to crawlers; llms.txt speaks to AI models and agents
• Job: robots.txt says "don't fetch these paths"; llms.txt says "here is what matters on this site"
• Enforcement: robots.txt is honored by legitimate crawlers by convention; llms.txt enforces nothing at all
• Effect on search rankings: robots.txt directly controls crawling; llms.txt has no documented effect on rankings

Why People Confuse the Two

Both are .txt files at your domain root, both are "for bots," and both showed up in the same wave of AI-crawler anxiety. Some site owners even assume llms.txt is a newer replacement for robots.txt. It isn't — it answers a different question, for a different consumer, at a different stage of the pipeline.

Think of it this way: robots.txt is the bouncer at the door. llms.txt is the welcome brochure inside. The bouncer decides who gets in; the brochure helps guests who are already inside find what they need.

What robots.txt Cannot Do

• It cannot stop determined scrapers. Bad actors simply ignore it. If bandwidth theft is the problem, you need server-level blocking, a WAF, or a CDN bot-management layer.
• It cannot keep a URL out of Google's index by itself. A disallowed URL can still be indexed without content. Use noindex for removal.
• It cannot make your content AI-friendly. Access control and content clarity are separate jobs.

What llms.txt Cannot Do

• It cannot grant or deny access. It is not a permission system.
• It is not a ranking factor. Google has told site owners plainly that they do not need to create new machine-readable AI text files or markup to appear in AI Overviews or AI Mode.
• It does not guarantee AI citations. Many of the most-cited sites have no file at all.
• Most of its real consumers today are coding agents. Tools like Cursor and Claude Code index llms.txt during project setup.

Using Both Together: A Practical Recipe

Step 1 — robots.txt: decide who may crawl. Allow search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) so you stay visible in both classic and AI search. Generate the file with our free robots.txt generator instead of hand-writing it.

Step 2 — llms.txt: summarize what the site is. Write an H1 with your site name, a one-line blockquote summary, and H2 sections linking to your 5–20 most important pages with one-line descriptions. Build it in minutes with our llms.txt generator.

Step 3 — Optional: declare content policy with ai.txt. Our ai.txt generator creates that file too.

Common Mistakes to Avoid

• Replacing robots.txt with llms.txt. They do different jobs. Dropping robots.txt leaves your crawler policy undefined.
• Putting access rules in llms.txt. Writing "do not crawl /private/" inside llms.txt accomplishes nothing. Access rules go in robots.txt.
• Dumping your whole sitemap into llms.txt. The file is meant to be a curated shortlist, not a second sitemap.
• Blocking everything in robots.txt "until the AI thing settles." A blanket disallow also blocks the search crawlers that drive your traffic. Decide per crawler instead.

Frequently Asked Questions

Do I need llms.txt if I already have robots.txt and a sitemap?

Strictly speaking, no. robots.txt and your XML sitemap cover crawler access and URL discovery, and no search engine requires llms.txt. It is an optional, low-cost extra that helps AI agents understand your site quickly.

Will llms.txt help me rank in Google or get cited by ChatGPT?

There is no documented evidence that it does. Google says you don't need new AI-specific files or markup to appear in AI Overviews. Treat it as a discoverability courtesy, not a ranking tactic.

Can llms.txt block AI crawlers?

No. llms.txt has no blocking capability whatsoever. Crawler access is controlled by robots.txt and by server-level controls.

Where exactly does llms.txt go?

At your domain root: yourdomain.com/llms.txt. Serve it as text/plain or text/markdown.

Should small sites bother with either file?

robots.txt: yes if you have anything worth controlling — and it costs nothing to have a correct minimal one. llms.txt: worthwhile if AI agents are a plausible audience for your content; otherwise optional.