Two small characters cause more robots.txt accidents than any other feature in the file: the asterisk and the dollar sign. Used well, they let you block a whole class of URLs with a single line — every PDF, every search-results page, every URL with a tracking parameter. Used carelessly, they can block your entire site, or block nothing at all while looking correct. This guide explains exactly how robots.txt wildcards and pattern matching work under RFC 9309, with examples you can test before you publish.
The short answer: in robots.txt, the asterisk matches any sequence of zero or more characters, and the dollar sign marks the end of the URL path. So a rule blocking every path ending in .pdf is written as a disallow for a pattern ending in .pdf followed by the dollar sign. A rule blocking every URL that contains a question mark starts with a pattern containing a question mark preceded by the asterisk. Everything else in this guide is about the details that make those two rules behave the way you expect — and the traps that make them behave differently.
The Standard Behind the Symbols: RFC 9309
Robots.txt began as an informal agreement in 1994. For almost thirty years, different crawlers implemented it slightly differently, and the wildcard behavior was one of the biggest sources of disagreement. In September 2022, the Internet Engineering Task Force published RFC 9309, which formalized the Robots Exclusion Protocol as a standard. That standard defines the file format, the parsing rules, and the two wildcard characters.
RFC 9309 says, in its opening section, that these rules are not a form of access authorization. That sentence matters for everything that follows: robots.txt is a request that well-behaved crawlers honor, not a lock. Wildcards do not make it a lock either. They make the request more precise, so a compliant crawler blocks exactly what you meant and nothing else. They do nothing at all to a crawler that ignores the file.
The file itself has strict basics that wildcards do not change. It must live at the root of the site, at yourdomain.com/robots.txt, and be served as plain text in UTF-8. Rules are grouped under user-agent lines. Path matching is case-sensitive, even though the directive names themselves are not — so /Admin/ and /admin/ are different paths. Google also documents a 500 KB size limit for robots.txt files; beyond that, crawlers may stop reading, which means a rule at the bottom of a very large file may never be seen.
The Asterisk: Match Anything, Including Nothing
The asterisk matches any sequence of zero or more characters. The "including nothing" part is the detail people forget, and it causes real bugs. A pattern like /blog/*.pdf does not only match /blog/guide.pdf. Because the asterisk can match nothing, it also matches /blog/.pdf. Because it can match many characters, it also matches /blog/2026/01/deep/nested/guide.pdf — the asterisk crosses slashes without stopping, unlike wildcards in some other systems where it stops at a directory boundary. In robots.txt, there is no directory boundary for the asterisk. It matches everything until the rest of the pattern matches or the path ends.
That greediness is useful. A single rule can block every PDF on your site, every URL containing a session identifier, or every filtered version of a category page. It is also dangerous. A rule written as a disallow for a pattern starting with an asterisk and a slash can match far more than intended if the author meant to target only one directory. Before publishing any wildcard rule, read it as a crawler would: start at the beginning of the path, let the asterisk swallow as much as it needs to, and ask what else could satisfy the pattern. If the answer includes pages you want indexed, narrow the pattern.
The Dollar Sign: Anchor to the End of the Path
The dollar sign matches only the end of the URL path. It turns a loose pattern into an exact ending. Without it, a rule blocking .pdf would also block a path like /guide.pdf/preview, because the pattern matches the beginning of that path. With the dollar sign at the end, only paths that actually end in .pdf are blocked, and /guide.pdf/preview stays open.
A common pair of examples makes the difference concrete. A disallow for /product blocks /product, /product/details, and /products, because all three start with the pattern. A disallow for /product followed by the dollar sign blocks only /product itself, and leaves /product/details and /products alone. If your goal is to block one exact page, the dollar sign is not optional decoration; it is the difference between blocking one page and blocking a whole section.
Note that the dollar sign only works at the end of a pattern. Putting it in the middle of a pattern does not anchor anything in the middle; it is simply part of the pattern that must match literally, which in practice means the rule will rarely match anything. Keep the dollar sign at the very end, after the last character you want to match.
How Crawlers Choose Between Conflicting Rules
Most real robots.txt files contain rules that overlap. A common case: a broad rule blocks a whole directory, and a narrower rule re-opens one file inside it. RFC 9309 resolves this with a simple principle: the most specific matching rule wins, where "most specific" means the longest matching pattern. So a disallow for /archive/ and an allow for /archive/public/ together mean the public file is crawlable and the rest of the directory is not, because the allow pattern is longer and therefore more specific.
This longest-match rule is also how wildcards interact with plain rules. A wildcard pattern counts by its length as written, including the asterisk character itself. That can produce surprises: a short wildcard rule can lose to a longer plain rule, or win against a shorter one, in ways that are not obvious from reading the file top to bottom. Order does not decide the winner in RFC 9309; length does. If two rules tie in length, the allow rule wins under Google's implementation, which is the behavior most site owners should assume when a tie is possible. The safest practice is to avoid ties entirely: when you write an exception, make it clearly longer and more specific than the rule it overrides.
There is a second layer to rule selection that matters just as much: which group applies. A crawler uses only the groups that name it, plus the catch-all group for all crawlers if no specific group matches. Groups for the same crawler are combined. So if you write a wildcard rule under the catch-all group and a specific rule under a named group, a named crawler follows its own group, not the catch-all. A rule that looks universal in the file may not apply to the crawler you care about, because that crawler has its own group elsewhere in the file.
Practical Patterns You Can Use Safely
The patterns below cover the jobs site owners most often want wildcards for. Each one includes the trap to check before publishing.
Block every file of one type
To block every PDF, use a disallow pattern that ends with .pdf and the dollar sign. The asterisk at the start lets the pattern match PDFs in any directory, and the dollar sign stops it from matching paths that merely contain .pdf in the middle. Check that you do not have PDFs you want indexed, such as public reports or white papers, because this pattern does not distinguish between them.
Block search results and filtered pages
Internal search pages and faceted filter URLs are a classic crawl-budget drain. A disallow pattern with an asterisk followed by a question mark blocks every URL containing a query string, which covers search results, sorting parameters, and tracking parameters in one line. The trap: if your site serves real content at URLs with query strings — some storefronts and single-page applications do — this pattern blocks that content too. Check a sample of your indexed URLs for question marks before using it.
Block a parameter while keeping the page
Sometimes you want the page indexed but not its parameterized variants. A pattern that matches the base path, then an asterisk, then the parameter name, blocks only URLs carrying that parameter, while the clean URL stays open. This is a precise use of the wildcard: the asterisk bridges the gap between the fixed path and the parameter, and nothing else in the pattern is loose.
Block a directory except one file
Combine a broad disallow for the directory with a longer allow for the single file you want open. The allow pattern must be longer than the disallow pattern to win under the longest-match rule, which it naturally is, because it names the full file path. Test this pair specifically, because it is the pattern most often broken by a second rule elsewhere in the file that the author forgot about.
Five Wildcard Mistakes That Break Real Sites
1. A missing trailing slash blocks a sibling
A disallow for /admin blocks /admin/, /admin/login, and also /administrator/, because all three start with the same characters. If you mean the directory, write /admin/ with the trailing slash. This is not a wildcard mistake in the strict sense, but it is the same class of error: a pattern that matches more than the author pictured. Always read a pattern as a prefix, not as a folder name.
2. An asterisk at the start of a pattern with no anchor
A pattern that begins with an asterisk and names only a short string can match URLs all over the site. For example, a pattern meant to block one tracking parameter can also match a blog post whose slug happens to contain the same characters. Anchor wildcard patterns with as much fixed text as possible: a directory, a file extension, or a parameter name with its equals sign, rather than a bare fragment.
3. Forgetting the dollar sign on exact-page blocks
As shown above, blocking /product without the dollar sign blocks /products and /product/details as well. Whenever you mean one exact page, end the pattern with the dollar sign and test the neighboring URLs. A quick check of the plural, the child path, and the hyphenated variant catches most of these errors.
4. Assuming wildcards work the same everywhere
Most major crawlers support the asterisk and dollar sign, and RFC 9309 standardizes them, but not every bot in the wild implements the standard fully. A small or obscure crawler may treat an asterisk literally, or ignore the dollar sign. If a specific bot matters to you — a training crawler, for instance — do not rely on a clever wildcard pattern alone. Give that bot its own named group with a simple disallow for the whole site or the whole section, and use wildcards for the general case. The free robots.txt generator builds named groups for the major AI crawlers so you do not have to remember each token.
5. Publishing without testing
Robots.txt errors are silent. Nothing on your site breaks visibly when a wildcard blocks the wrong pages; you find out weeks later from a traffic drop or a Search Console report. Test every wildcard rule before publishing, and again after publishing, using a tester that checks specific URLs against your live file. The free robots.txt tester lets you check whether a given crawler may fetch a given URL under your current file, which is the exact question a wildcard mistake gets wrong.
How to Test Wildcard Rules Before You Publish
Testing is a short checklist, not a project. First, list the URLs the rule must block and the URLs it must not block, including the near-misses: the plural, the child path, the same slug in another directory, and the same extension in a different case. Second, run each URL through a tester with the relevant user agent, not just the catch-all. A rule under a named group behaves differently from the same rule under the catch-all group. Third, check the live file after upload, because the file on your server is the one crawlers read, not the draft on your computer. A caching layer or a platform that serves a default robots.txt can silently replace your file; fetching yourdomain.com/robots.txt in a browser confirms what is actually live.
Fourth, re-check after any platform change. Migrations, plugin updates, and CDN setting changes can all replace or override a robots.txt file without warning. One recent example from this blog's news coverage: a platform migration silently removed crawler blocks that site owners believed were still in place. A monthly check of your live file, taking less than a minute, catches that class of failure before it costs you months of unwanted crawling.
Wildcards and AI Crawlers
Wildcard patterns are particularly useful for AI crawler control, because AI crawlers come in families. OpenAI alone operates a training crawler, a search crawler, and a user-triggered fetcher, each with its own token. Anthropic, Perplexity, and others split their crawlers the same way. A site owner who wants to block training but allow citations needs named groups per token, not one wildcard rule for everything, because a wildcard cannot distinguish between tokens — it matches paths, not crawlers. Use wildcards to shape which paths each named crawler may fetch, and use separate groups to decide which crawlers get in at all. The guide to blocking AI crawlers with robots.txt walks through the token-by-token setup.
One more AI-specific caution: some AI systems fetch pages on behalf of a user, and the crawler documentation for those user-triggered fetchers often says robots.txt rules may not apply, because the fetch was initiated by a person rather than by the crawler's own schedule. A wildcard rule, however precise, does not change that. If user-triggered fetching is a concern for a particular page, that page needs a stronger control than robots.txt, such as login protection or a server-level rule, and the wildcard discussion is secondary.
Frequently Asked Questions
Does the asterisk match across slashes?
Yes. In robots.txt, the asterisk matches any sequence of characters, including slashes, so a pattern with an asterisk can match URLs in nested directories. Do not assume it stops at a directory boundary.
Does the dollar sign work in the middle of a pattern?
No. It only anchors the end of the path, and it belongs at the very end of the pattern. A dollar sign elsewhere in a pattern is treated as a literal character to match, which in practice means the rule will almost never match.
What wins when an allow and a disallow both match?
Under RFC 9309, the longest matching pattern wins. If the two patterns are the same length, Google's implementation prefers the allow rule. Avoid ties by making exceptions clearly longer than the rules they override.
Do all crawlers support wildcards?
Major search crawlers do, and RFC 9309 standardizes the behavior, but smaller or older bots may not implement wildcards fully. For a bot you specifically care about, use a named group with a simple rule rather than relying on a wildcard pattern alone.
Can a wildcard rule block my whole site by accident?
Yes, and it happens. A disallow for a bare slash blocks everything, and a loose wildcard pattern can come close. Always test the homepage, a category page, and a deep article URL against any new wildcard rule before publishing.
Is robots.txt a security tool?
No. RFC 9309 states plainly that its rules are not access authorization. Listing a private path in robots.txt also advertises that path to anyone who reads the file. Keep truly private content behind authentication, and use robots.txt only for crawl management.
Wildcards reward a few minutes of careful reading. Write the pattern, list what it must block and what it must spare, test both lists, and only then publish. That habit turns two small characters from the most common source of robots.txt accidents into one of the most useful tools in the file.
