Robots.txt is deceptively simple. Yet misconfigured files cause serious SEO damage every day. Here are the most common mistakes.
Mistake 1: Blocking the Entire Site
# WRONG — blocks everything including Google! User-agent: * Disallow: /
This is the most catastrophic error. Be specific about which user-agents you target.
Mistake 2: Wrong Filename or Location
The file must be named exactly robots.txt (lowercase) at your domain root — not in a subfolder.
Mistake 3: Wrong Wildcards
Disallow: /*.pdf$ blocks all PDFs. Disallow: / blocks everything. Know what you are writing.
Mistake 4: Commenting Out Live Rules
Lines starting with # are comments and are ignored. A mistakenly commented rule will not be enforced.
Mistake 5: Case-Sensitive Paths
Paths in robots.txt are case-sensitive on Linux servers. Disallow: /Admin/ does NOT block /admin/.
