agents.md is a Markdown file published at your site's root that gives AI agents plain-language instructions about your site — what it offers, how to use it, what is off-limits. It borrows its name from AGENTS.md, the established coding-agent convention, but no official website standard exists in 2026: adoption is informal and no major agent reliably reads one yet.
A new file keeps showing up in AI-SEO conversations: agents.md. Tool vendors bundle generators for it. Competitor scanners are starting to look for agent-facing files. And site owners are asking the obvious question: is this another llms.txt I need to publish, or noise?
The short version is in the answer box above. The longer version matters, because agents.md is a case of a good idea wearing a borrowed name. The name comes from a real, adopted standard — but that standard was built for code repositories, not websites. Everything after that is proposals, products, and honest disagreement. This guide separates the two, shows you exactly what AI agents verifiably look for on your site today, and gives you a practical template if you decide to publish one anyway.
Where agents.md comes from
AGENTS.md (usually uppercase) started as a repository file for coding agents, not a website file. When agentic coding tools took off in 2025, developers ran into a mundane problem: every coding assistant needed the same project context — build commands, test steps, code conventions, things not to break — and each tool read its instructions from a different file. AGENTS.md became the shared answer: a plain-Markdown file at the repository root that any coding agent reads when a session starts. The official spec site, agents.md, shows exactly what it looks like — environment tips, testing instructions, pull-request rules. Nothing about websites; everything about working on code.
Adoption grew fast because the tools cooperated. OpenAI's Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, and others read AGENTS.md natively; Claude Code reads its own CLAUDE.md, which many projects shrink to a one-line pointer at AGENTS.md. One industry write-up puts adoption at 60,000+ repositories, and stewardship now sits with the Agentic AI Foundation under the Linux Foundation. For coding, this is a real, maturing standard — not a fad.
The website version is where the name jumped the fence. Site owners and tool vendors started using "agents.md" (lowercase) for a file served from the website root that would tell visiting AI agents how to treat the site. The jump happened without a standards body behind it. As of 2026, no major model provider or consortium has published a specification for agents.md on websites. What exists instead is a loose family of proposals and products that share the name but not the format:
- Generator bundles: aiwebpageseo.com ships an agents.md generator as part of an AI-discovery file bundle — the closest thing to a working product definition, but a vendor feature, not a standard.
- agent.json (Agent Web Protocol): a JSON file at
/.well-known/agent.jsondeclaring what agents can do — actions with typed inputs and outputs, auth requirements, error recovery, dependencies. Machine-first, not prose. - agents.json proposals: separate drafts place a JSON manifest at
/.well-known/agents.jsoncovering data endpoints, actions, rate limits, and policies. - Adjacent standards: Google's ai-catalog.json (the ARD draft spec) and the A2A agent card at
/.well-known/agent-card.jsondeclare capabilities to agents, while skills.txt and WEBSITE.md drafts cover interaction instructions — each from different authors, none a winner yet.
So when someone says "add an agents.md to your website," they are invoking one of these ideas, not one thing. The name is shared; the format is not. That is the single most important thing to understand before you write one.
agents.md vs robots.txt vs llms.txt vs agent.json
These files get confused constantly because they all sit at well-known paths and all talk to machines. They solve different problems. Here is each one, plainly:
robots.txt — Plain text at /robots.txt. Declares access control: which crawlers may fetch which paths. The established standard, honored by every search bot since the 1990s. It says nothing about how to understand your content or what agents may do with it. If you want to allow or block AI crawlers, this is still the file that matters — test any change with the free robots.txt tester before it goes live.
llms.txt — Markdown at /llms.txt. Guides language models to your most important content: a short summary plus links to key pages. An informal standard proposed in 2024 that crawlers, RAG pipelines, and increasingly AI agents doing research know to look for. It is about understanding, not acting — it tells models what your site says, not what they may do on it. Build one with the llms.txt generator if you have not already.
agent.json — JSON at /.well-known/agent.json. Declares capabilities: what actions agents can take, with typed inputs and outputs, auth rules, and error recovery. Strictly machine-first. The most credible real-world deployment: Cloudflare publishes a live agent.json at cloudflare.com/.well-known/agent.json, describing a read-only content service that points agents at its llms.txt and Markdown snapshots of its pages. That is a major vendor shipping the concept in production — but note it is agent.json, not agents.md.
agents.md — Markdown at /agents.md. Aims to be plain-language behavior instructions for visiting AI agents: what the site is, how to interact with it, rate limits, identification expectations, what is off-limits. Human-readable by design. Informal, with multiple competing drafts and no official spec. For a head-to-head with the older files, see llms.txt vs robots.txt.
ai-catalog.json — JSON at /.well-known/ai-catalog.json. Google's ARD draft specification (v0.9, May 2026, co-developed with a Linux Foundation working group) for declaring callable capabilities — APIs, MCP servers, agents — to agent registries. A registry-discovery layer, not an instruction layer; and as of mid-2026, no public registry an agent could actually query had been documented. For the full picture, see ai-catalog.json explained.
The one-line version: robots.txt controls access, llms.txt aids understanding, agent.json and ai-catalog.json declare capabilities, and agents.md tries to declare behavior — but it is the least defined of the five.
What AI agents actually look for on your site today
Here is the claim to be careful with: there is no evidence that any major AI browser agent fetches /agents.md as part of its standard behavior today. Not ChatGPT's agent mode, not Claude's agent capabilities, not the shopping or research agents shipping from browser vendors. If you publish the file, the main readers right now are curious humans and readiness scanners — not agents. That may change; today it is the truth.
What agents and agent-readiness tools do verifiably check, drawn from published scanner checklists and the files real deployments serve:
- robots.txt with explicit AI-crawler rules. Training bots and search bots are separate tokens with separate rules —
GPTBotvsOAI-SearchBot,ClaudeBotvsClaude-SearchBot— and scanners flag files that conflate them. - An XML sitemap that resolves. Reachability of
/sitemap.xmlis a basic agent-readiness check; it is still the canonical machine-readable inventory of your pages. - llms.txt. The informal standard agents know to look for when they need to understand a site quickly. If you publish one agent-facing file, the evidence says make it this one.
- An MCP server card at
/.well-known/mcp/server-card.json. Readiness scanners check this path. If you expose tools to AI applications through the Model Context Protocol, this is how agents find the front door. - An API catalog at
/.well-known/api-catalog. Another documented discovery surface for callable endpoints. - Structured data. Schema markup on your pages is machine-readable meaning that every crawler and agent already consumes — the oldest "agent instruction" on the web, hiding in plain sight.
- Server-rendered content. Agents fetch raw HTML. If your key content only appears after JavaScript runs, many agents see an empty page. Nothing in any instruction file fixes that.
Notice the pattern: agents discover your site through the same machine-readable surfaces the web has been building for decades, plus the new protocol layer. Which brings us to the protocol that actually matters for agent interaction.
MCP — the Model Context Protocol — is the open standard for connecting AI applications to external systems. The official documentation at modelcontextprotocol.io describes it as a USB-C port for AI applications: a standardized way for assistants like Claude or ChatGPT to reach data sources, tools, and workflows. If your site offers something agents should use — a booking API, a product search, a calculator — exposing it through an MCP server is the verified, adopted path. Markdown instruction files are the suggestion layer; MCP is the action layer. An agents.md that points at a real MCP server is ten times more useful than one that just describes your homepage in friendlier words.
How to write an agents.md for your site
If you have decided to publish one — for early positioning, for policy clarity, or because your agents.md generator output needs a human review — keep it short, honest, and machine-friendly. The steps:
- Decide what you actually offer agents. If nothing on your site is callable — no API, no booking flow, no tool — your agents.md is a statement of intent, not an interface. Say so plainly instead of implying capabilities you do not have.
- Keep it to one screen. The coding convention works because it is a cheat sheet, not a manual. Aim for under 100 lines. Point at machine-readable sources (sitemap, llms.txt, JSON feeds, your MCP server) instead of duplicating them in prose.
- Cover the six things agents need: what the site is, the machine-readable entry points, how to identify themselves, rate limits and retry behavior, what requires authentication, and what is off-limits (form submissions, account actions, admin paths).
- Defer access authority to robots.txt. Your agents.md should name robots.txt as the source of truth for crawl permissions, not restate or contradict it. Two files disagreeing about access is worse than one file.
- Serve it as plain Markdown at
/agents.mdover HTTPS with a 200 status, no JavaScript required. If agents ever do look for it, this is where the convention points. - Keep it in sync. An instruction file that describes endpoints you retired is a trust problem. Review it whenever you change your robots.txt or site structure.
The example below is an illustrative starting template, not a specification — adapt every line to what your site actually offers, and delete sections that do not apply:
# agents.md — Example Store (starting template)
This file gives AI agents plain-language guidance for interacting with
this site. For crawl permissions, robots.txt is authoritative.
## What this site is
Example Store sells refurbished laptops. Public catalog pages are open
to everyone; checkout and account pages require authentication.
## Machine-readable entry points
- Sitemap: https://example.com/sitemap.xml
- Product feed (JSON): https://example.com/api/products.json
- Content summary for LLMs: https://example.com/llms.txt
## How to interact
- Identify your agent with a contactable user-agent string.
- Stay under 60 requests/minute; back off on 429 responses.
- For API access, request a key at https://example.com/developers.
## Off-limits
- Do not create accounts, place orders, or submit forms on behalf of
users without explicit user confirmation.
- Do not access /admin or any path disallowed in robots.txt.
## Contact
Agent operators with questions: agents@example.comOne more caution: if you used a generator to produce your file, read every line before publishing. Generators in this space tend to invent endpoints, rate limits, and capabilities that do not exist on your site. An agents.md that lies about your site is worse than none — it teaches every agent that reads it to distrust your declarations.
Should you add one in 2026? Honest take
Publish one if you run a site where agents will plausibly transact — commerce, bookings, tools, public APIs. For you it is a one-hour policy document that states the rules before the traffic arrives, and it pairs well with the JSON formats (agent.json, ai-catalog.json) that machines actually prefer. If you expose an MCP server, mention it in the file; the instruction file and the protocol reinforce each other.
Skip it — for now — if you run a content site. Your highest-leverage agent-facing work in 2026 is, in order: a correct robots.txt with separate training and search rules, a good llms.txt, schema markup, and server-rendered content. No major agent fetches /agents.md today, it carries no SEO signal (Googlebot and Bingbot do not read it), and a file that promises interaction your site cannot deliver is theater.
The honest framing: agents.md for websites is a bet on a convention that has not settled yet. The coding original earned its status through years of tool cooperation; the website version has not had that yet. A short, truthful file costs you an hour and positions you early. A long, aspirational one costs you credibility. If you publish, publish the short truthful one — and keep your llms.txt excellent, because that is the discovery file that works today.
FAQ
Is agents.md an official standard?
No. AGENTS.md (uppercase) is the established, foundation-stewarded standard for coding agents, with its spec at agents.md. The lowercase agents.md for websites is an informal convention with no official specification — several vendors and proposals use the name for different formats, and no major model provider has endorsed one.
Does publishing agents.md help my SEO?
Not directly. It is not read by Googlebot or Bingbot and carries no ranking signal. It is also not currently fetched by major AI agents as standard behavior, so do not expect traffic or citations from it. Treat it as future-facing infrastructure, not an SEO tactic.
What is the difference between agents.md and llms.txt?
llms.txt describes your content so language models can understand and cite it — a summary plus links to key pages. agents.md aims to describe behavior: how AI agents should interact with your site, what is allowed, and what is off-limits. One is about understanding; the other is about conduct.
Will ChatGPT or Claude agents read my agents.md?
There is no public evidence that they fetch /agents.md automatically today. Both assistants can read the file if a user points them at it, and readiness scanners may check for agent-facing files — but as a discovery channel, it is a forecast, not a finding.
Where exactly do I put the file?
At https://yoursite.com/agents.md, served as plain Markdown over HTTPS with a 200 status and no JavaScript required. Keep robots.txt as the authority on crawl permissions, keep the file short, and review it whenever your site structure or robots.txt changes.
