An ecommerce site is the worst place to get robots.txt wrong. Block too little and AI crawlers vacuum up your product descriptions, prices, and reviews — the content your competitors and training datasets want most. Block too much and Googlebot never sees the product pages that bring in sales. The answer is a file that is precise about what it blocks: keep crawlers out of carts, checkouts, and faceted filter URLs, block AI training bots from your catalogue, and leave every product and category page wide open to the search crawlers that send you customers.
This guide gives you a copy-ready robots.txt file for an online store, explains each rule, and shows how to handle the two problems that are unique to ecommerce: faceted navigation that creates thousands of near-identical URLs, and AI crawlers that scrape your prices.
Why Ecommerce Robots.txt Is Different
A blog has a few hundred pages. A store can have tens of thousands, and most of them are not worth crawling. A single category with size, colour, price, and sort filters can generate hundreds of URL combinations that all show nearly the same products. Add internal search results, cart pages, checkout flows, and account pages, and the crawlable surface of a store can be ten times the size of its real catalogue.
That matters for two reasons. First, crawl budget. Googlebot will spend its time on whatever it finds, and if it spends that time on filter combinations and cart pages, it has less time for the product pages that actually sell. Second, AI crawlers love ecommerce content. Product descriptions, specifications, prices, and review text are clean, structured, and easy to ingest — ideal training material and ideal material for an AI shopping answer that sends the customer to someone else.
The goal of the file below is to shrink the crawlable surface to the pages that matter — products, categories, and guides — while keeping the search crawlers that drive revenue fully allowed.
The Copy-Ready Ruleset for an Online Store
This file works for most stores on Shopify, WooCommerce, Magento, and BigCommerce, with small adjustments for each platform's URL patterns. Replace the domain in the sitemap line with your own.
# Block AI training crawlers site-wide
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Allow AI search crawlers so products stay citable in AI answers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Everyone else: keep carts, checkouts, accounts and internal search out
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Disallow: /wishlist/
Sitemap: https://yourstore.com/sitemap.xmlThree things to notice. First, the AI trainer blocks come first and apply to the whole site, including product pages. Your descriptions and prices stay out of training datasets. Second, the search crawlers are explicitly allowed, so your products can still be cited when someone asks ChatGPT or Perplexity for a recommendation. Third, the general group blocks only the transactional and utility paths — nothing that sells.
On Shopify, the cart and checkout paths are /cart and /checkouts/, and internal search lives at /search. On WooCommerce, the equivalents are usually /cart/, /checkout/, and /my-account/. On Magento, watch for /customer/ and /checkout/. Check your own URLs before you copy the file, and test it with the robots.txt tester — a rule that does not match your real paths does nothing.
Taming Faceted Navigation and Filter URLs
Faceted navigation is the biggest crawl problem in ecommerce. A category page with filters for size, colour, brand, and price can produce thousands of combinations, and every combination is a separate URL showing almost the same products. Left alone, crawlers will happily index them all, diluting your category page and wasting crawl budget.
The cleanest fix is at the source: make filter URLs that add no value un-crawlable, and tell Google which version is canonical. In robots.txt, block the query patterns your filters use. The exact pattern depends on your platform, but it usually looks like this:
User-agent: *
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*price=
Disallow: /*?*color=Wildcards in robots.txt are supported by Google and by most major crawlers, following the RFC 9309 standard. A rule like Disallow: /*?*sort= blocks any URL containing a sort parameter, whatever else is in the address. Use them carefully: a wildcard that is too broad can block product pages that happen to carry a parameter. Test every pattern against real product URLs before deploying.
Robots.txt is only half the answer here. Pair it with canonical tags that point every filtered version back to the clean category URL, so that if a filtered URL does get indexed, Google knows which version counts. And if your filters create genuinely useful landing pages — a "red running shoes under $100" page that people actually search for — let that one be crawlable and indexable, and block only the combinations nobody searches for. The distinction is whether the page answers a real search query, not whether it has a filter in the URL.
Blocking AI Crawlers Without Losing Product Visibility
This is the decision that worries store owners most: if I block AI crawlers, will my products disappear from AI shopping answers? The answer depends on which crawler you block, and the distinction is the same one that applies to every site.
Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider — collect content to train models. Blocking them does not affect whether your products appear in ChatGPT search, Perplexity, or Google AI Overviews. Those answers come from search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. Block the trainers, allow the search crawlers, and your products stay visible in AI answers while your catalogue stays out of training data.
The one case where you might block a search crawler too is if an AI shopping feature is sending you traffic that never converts, or if a specific answer engine is reproducing your prices and reviews without sending clicks. That is a business call, and it is reversible: remove the block and the crawler returns. But make it deliberately, crawler by crawler, rather than blocking everything in a panic. A store that disappears from AI answers entirely gives its competitors the space for free.
Our earlier guide to stopping AI crawlers from scraping your prices covers the price-specific tactics in more depth, including how to spot price scraping in your logs.
Product Pages, Reviews, and Structured Data
Robots.txt is not the only file that shapes how AI systems see your store. Product structured data — the schema markup that describes a product's name, price, availability, and reviews — is what lets search engines and AI answers show your products properly. Keep it accurate and keep it crawlable. A product page that is blocked in robots.txt cannot show Google its structured data, because Googlebot never fetches the page.
Reviews deserve the same care. Review text is among the most scraped content on a store, because it reads like a human recommendation. Blocking trainers from the whole site already covers your review pages, so you do not need separate rules for them. What you should not do is block the review pages from Googlebot in an attempt to hide them from AI crawlers — the two are different crawlers, and blocking Googlebot costs you the rich review snippets that help products rank.
One more detail: do not list internal or sensitive URLs in your robots.txt file. The file is public. A rule like Disallow: /admin-supplier-pricing/ tells the world that the path exists. Keep genuinely private paths behind authentication, and keep them out of the file entirely.
Testing and Maintaining the File
Test before every deploy. Paste the file into the robots.txt tester and check four things: GPTBot is blocked from a product page, Googlebot can reach the same page, OAI-SearchBot can reach it, and a cart URL is blocked for the general group. Then generate a clean version with the robots.txt generator if you would rather build the file from your choices than hand-edit text.
After deploying, verify the live file at https://yourstore.com/robots.txt and check Google Search Console's robots.txt report, which shows the file as Google fetched it and flags syntax errors. Then watch two things over the next few weeks: the crawl stats report, where you should see fewer requests to cart, checkout, and filtered URLs; and your server logs, where GPTBot and ClaudeBot requests should drop to zero while search crawlers continue normally.
Review the file whenever you change your store. Adding a new filter, launching a new category structure, migrating platforms, or adding a new app that creates public URLs can all change which paths need blocking. A quarterly review — and a check after any platform change — keeps the file honest. Bot names change too: new crawlers appear regularly, and old tokens stop working, so check the current user-agent names against the operators' documentation when you review.
FAQ
Should an online store block GPTBot?
Yes, in most cases. GPTBot collects content for training, not for search. Blocking it keeps your product descriptions, prices, and reviews out of training datasets, and it does not affect your visibility in ChatGPT search, which uses OAI-SearchBot.
Will blocking AI crawlers remove my products from AI shopping answers?
Only if you block the search crawlers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the indexes that AI answers cite. Block the trainers — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider — and allow the search crawlers, and your products stay citable.
What should an ecommerce robots.txt block?
Cart, checkout, account, wishlist, and internal search pages, plus faceted filter URL patterns that create near-duplicate pages. Never block product pages, category pages, or your sitemap.
How do I stop crawlers indexing filtered category URLs?
Block the filter query patterns in robots.txt with wildcard rules, and add canonical tags pointing filtered versions back to the clean category URL. Let genuinely useful filtered pages — ones people actually search for — stay crawlable.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search, rankings, and AI Overviews depend on Googlebot, which is unaffected.
How long does a robots.txt change take to work?
Expect about 24 hours. Google caches robots.txt for up to a day, and OpenAI and Meta report similar delays. The change stops future collection; it does not remove data already gathered.
For the wider context, see our guides to robots.txt directives explained and llms.txt for ecommerce stores, which pairs with this file to make your catalogue easier for AI systems to understand.
