Quick answer: A robots.txt file uses a small set of directives to tell crawlers what they may and may not access. Disallow blocks paths, Allow opens them back up, Sitemap points crawlers at your XML sitemap, and Crawl-delay asks aggressive crawlers to slow down. Everything else — Noindex, Host, most "block AI" tokens — is either unsupported or misunderstood. This guide explains each directive exactly as the 2022 standard (RFC 9309) defines it, so you can stop guessing and write a file that does what you mean.
The two lines that matter most: User-agent and Disallow
Every robots.txt file is built from groups. A group starts with one or more User-agent lines (who the rules are for) followed by rules (what they may do). A crawler reads the file, finds the group whose user-agent token matches it best, and follows only that group's rules.
User-agent: Googlebot
Disallow: /private/
Disallow: /tmp/
User-agent: *
Disallow: /admin/Here Googlebot is kept out of /private/ and /tmp/; every other crawler is kept out of /admin/. Two details people constantly get wrong:
- User-agent matching is case-insensitive.
User-agent: googlebotandUser-agent: Googlebotmean the same thing. - A crawler follows only the most specific matching group. If your file names GPTBot explicitly and also has a
User-agent: *group, GPTBot ignores the wildcard group entirely and follows its own. You cannot "stack" rules across groups.
You can also group several bots under one set of rules by listing multiple User-agent lines before the rules — fully valid per the standard, and it keeps long AI-crawler blocklists readable:
# Block the main AI training crawlers in one group
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Disallow: what it blocks, and the traps
Disallow tells the crawler which paths it must not fetch. Disallow: / blocks the whole site for that group. An empty value — Disallow: — means "disallow nothing," which is the same as allowing everything.
The value is a path prefix. Disallow: /images/ blocks /images/cat.jpg but not /photos/images/. A few classic traps:
Disallow: /with no trailing care blocks the entire site. This is the single most common catastrophic robots.txt error, and it usually comes from leaving a staging-file setting enabled after launch.- Robots.txt does not stop indexing. Disallowed pages can still appear in search results if Google finds links to them — it just can't fetch the page to read the content, so the listing shows without a description. If you want a page out of the index, use a
noindexmeta tag or HTTP header instead, and allow crawling of the page so Google can see the tag. - Listing a path makes it public. Every crawler — and every curious human — can read your robots.txt. Never put genuinely secret paths in Disallow; protect them with a login.
Allow: the override (and why it's rarely needed)
Allow re-opens a path inside a group that also disallows it. When an Allow and a Disallow both match a URL, the longest match wins — the more specific rule takes priority.
User-agent: *
Disallow: /blog/
Allow: /blog/public-guide/Here crawlers stay out of everything under /blog/ except /blog/public-guide/, because the Allow line is longer and more specific. You rarely need Allow: / on its own — an absent rule already means allowed — but it is useful for documenting intent ("yes, we deliberately allow this bot") and for carving exceptions out of broad blocks.
Wildcards: * and $ done right
Two special characters work inside Disallow and Allow values:
*matches any sequence of characters.Disallow: /*?blocks every URL containing a question mark — the classic way to keep crawlers out of faceted/filtered parameter pages.$means "end of URL."Disallow: /*.pdf$blocks only URLs that end in .pdf, not ones like/pdf-guide/.
# Keep crawlers off parameter junk and printable versions
User-agent: *
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*/print$Be careful with *: Disallow: /*.php without the $ also matches /php-info/, because the wildcard keeps matching past the ".php" part.
Sitemap: the one directive everyone should use
The Sitemap directive tells crawlers where your XML sitemap lives. It takes an absolute URL, and it's independent of user-agent groups — put it anywhere in the file, usually at the top or bottom:
Sitemap: https://example.com/sitemap.xmlAll major search engines read it. A common misconception: the Sitemap line does not force crawling of the listed URLs, and it does not let you limit a sitemap to one crawler — every crawler that reads the file sees every Sitemap line. If you maintain several sitemaps, list each on its own line.
Pairing your robots.txt with a complete sitemap is the simplest thing a small site can do to get new pages discovered faster. If you don't have one yet, our free sitemap generator builds one from any URL in seconds.
Crawl-delay: the polite slowdown (with limits)
Crawl-delay asks a crawler to wait a number of seconds between requests:
User-agent: ClaudeBot
Crawl-delay: 10This is a non-standard extension — it is not part of RFC 9309. Support varies: Bing and Yandex honor it; Google's crawlers ignore it entirely. Anthropic's ClaudeBot supports a Crawl-delay line if you want to slow it down rather than block it. Treat it as a request, not a guarantee, and if server load is the real problem, rate-limiting at your firewall or CDN is the reliable fix.
Directives that don't work the way people think
Noindex in robots.txt is dead
Google announced in 2019 that it would no longer support Noindex in robots.txt, and it stopped reading the directive that September. If you see Noindex: / in old tutorials, delete it — modern crawlers ignore it, and the page stays indexable.
Host is not a real standard
The Host directive (telling crawlers your preferred domain version) was only ever honored by a few crawlers and is not part of the standard. Use canonical tags and 301 redirects to consolidate www vs non-www instead.
Google-Extended is a permission token, not a crawler
Blocking User-agent: Google-Extended opts your content out of training future Gemini models. It has no separate user agent string, does not affect your Google Search rankings, and cannot hurt your indexation. The same is true of Applebot-Extended for Apple intelligence features. Confusing these tokens with Googlebot itself is an expensive mistake — never disallow Googlebot by accident.
A complete, safe robots.txt template for 2026
Here is a starting point for a typical content site. It allows normal search crawlers, blocks admin and parameter junk, opts out of AI model training, and declares the sitemap. Test any file before publishing with our free robots.txt tester.
# Allow standard search engines
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Opt out of AI model training (does NOT affect Google Search)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Keep crawlers off admin areas and parameter junk
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xmlNotice the structure: the AI-training bots get their own explicit groups so they never fall through to the wildcard group, and Allow: / is written explicitly only where it documents intent.
Five rules to live by
- One crawler, one group. Crawlers ignore the wildcard group once you name them — put every rule for that bot in its own group.
- Longest match wins. When Allow and Disallow collide, the more specific path wins. Use that deliberately.
- Disallow is not noindex. Blocked pages can still show in search results. Use noindex for pages that must leave the index.
- Test before you publish. A single stray
Disallow: /can de-list your site. Verify with a tester every time you edit the file. - Non-standard directives are requests, not contracts. Crawl-delay, training opt-outs, and wildcards depend on crawler cooperation — pair them with server-side controls when it matters.
If you'd rather not hand-write the file, our robots.txt generator builds one with the right AI-crawler presets, and our AI-crawler guide at robots.txt template for AI crawlers shows the exact tokens for every major bot.
How crawlers actually read the file: the order of operations
It helps to know the exact sequence a crawler follows, because most "why isn't this working" mysteries come from misunderstanding it:
- Find the groups. The crawler scans for every group whose
User-agentline matches its token (case-insensitive, and the token may be a substring of the crawler's full user-agent string). If none match by name, it falls back to theUser-agent: *group. - Pick one group. If several groups match, the crawler uses the group with the most specific (longest) matching token. It reads only that group — rules in other groups are invisible to it.
- Apply the longest-match rule. Within the group, the crawler compares every Allow and Disallow against the URL it's about to fetch. The longest matching rule wins, regardless of whether it's an Allow or a Disallow.
A few mechanical details the standard pins down: the file must live at the domain root (example.com/robots.txt — nowhere else counts), it must be UTF-8 encoded, and blank lines plus # comments are ignored. Google truncates files larger than about 500 KB, so keep yours lean — a hand-written file will never come close to that, but auto-generated ones stuffed with thousands of parameter rules sometimes do.
One more subtlety: a crawler caches your robots.txt (typically up to 24 hours), so changes aren't instant. When you test, you're testing the file as published; when a bot "ignores" your new rule an hour later, it's usually reading the cached copy.
Frequently asked questions
What is the difference between Disallow and Noindex?
Disallow stops a crawler from fetching a page; Noindex (as a meta tag or HTTP header) stops a search engine from indexing it. They are different tools. A disallowed page can still appear in search results without a description, while a noindexed page requires the crawler to fetch it first — so you must allow crawling of noindexed pages.
Does Disallow: / block my whole site?
Yes — for the user-agent group it appears in. Disallow: / inside a User-agent: * group blocks every crawler from everything. Always check for stray Disallow lines after migrating, staging, or installing plugins that write robots.txt.
Is Crawl-delay supported by Google?
No. Google ignores Crawl-delay entirely. Bing, Yandex, and Anthropic's ClaudeBot honor it. If Googlebot is hammering your server, adjust the crawl rate in Google Search Console instead.
Can I use wildcards in robots.txt?
Yes. * matches any sequence of characters and $ anchors to the end of the URL. Both are supported by Google and the other major crawlers, even though they were only formalized in the standard in 2022.
Should the Sitemap line go inside a user-agent group?
It works either way — the Sitemap directive is independent of groups, so every crawler sees every Sitemap line. Convention is to place it at the top or bottom of the file, outside all groups, where humans can find it.
Does blocking Google-Extended hurt my Google rankings?
No. Google states explicitly that Google-Extended is a standalone training-permission token with no effect on Search inclusion or ranking. Blocking it only affects whether your content may be used to train future Gemini models.
