Your prices are the most valuable data on your store, and AI crawlers are collecting them at scale. Training bots, comparison engines, and rival sellers' scrapers all read product pages, and the result shows up in places you never agreed to: AI answers that quote your prices, comparison sites that undercut you, and competitors who reprice against you in real time. You cannot hide prices from your customers, but you can make them much harder for bots to harvest — and this guide shows you how, layer by layer.
The short version: start with robots.txt rules that block the known AI training crawlers from your product and pricing pages, then add server-level enforcement that rate-limits or blocks aggressive visitors, and finally make your prices harder to extract at scale — render them after the page loads, require interaction for bulk access, and serve your real pricing data through an API that needs a key. No single layer is enough. Together, they turn casual scraping from a free download into an expensive project, and most scrapers move on to an easier target.
Why prices are the first thing bots take
Price data is structured, comparable, and valuable — the three things that make a page worth scraping. A product name and a price are enough to build a comparison table, train a pricing model, or power a shopping answer in an AI chatbot. A thousand product pages give a scraper your entire catalog in an afternoon, and unlike a blog post, prices change constantly, so the scraper comes back every day.
The collectors fall into three groups. The first is AI training crawlers — GPTBot, ClaudeBot, CCBot, Bytespider and the rest — which vacuum up the public web for model training. Your prices become part of a model's general knowledge, and they surface later in AI answers that quote your pricing without a visit, a click, or a sale. The second is shopping and comparison bots: legitimate price-comparison services, affiliate scrapers, and the answer engines that want to show a price next to your product name. The third is your competitors — or, more often, the scraping services they hire — pulling your catalog daily to reprice their own listings.
You cannot solve this by hiding prices from humans. Customers need to see a price before they buy, and search engines need to see it too if you want your products in shopping results. The goal is not invisibility. It is friction: let a human read a price in a second, and make a bot work hard to read ten thousand of them.
Layer 1: robots.txt rules for product pages
The first layer is the cheapest, and it stops the honest bots. Block the training crawlers from your product, category, and pricing paths while leaving Googlebot and the shopping crawlers you want — Googlebot for search, and the crawlers behind the shopping surfaces you use — alone.
Here is a pattern that works for most stores:
# Block AI training crawlers from product and pricing pages
User-agent: GPTBot
Disallow: /products/
Disallow: /product/
Disallow: /shop/
Disallow: /pricing/
User-agent: ClaudeBot
Disallow: /products/
Disallow: /product/
Disallow: /shop/
Disallow: /pricing/
User-agent: CCBot
Disallow: /products/
Disallow: /product/
Disallow: /shop/
Disallow: /pricing/
User-agent: Google-Extended
Disallow: /products/
Disallow: /product/
Disallow: /shop/
Disallow: /pricing/
User-agent: Bytespider
Disallow: /products/
Disallow: /product/
Disallow: /shop/
Disallow: /pricing/
User-agent: Meta-ExternalAgent
Disallow: /products/
Disallow: /product/
Disallow: /shop/
Disallow: /pricing/
# Keep search crawlers working
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/?q=*
Sitemap: https://yourstore.com/sitemap.xmlAdjust the paths to match your platform: Shopify uses /products/, WooCommerce uses /product/, and custom stores vary. The key decision is which bots to block. The trainers — GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Meta-ExternalAgent — are the ones feeding AI models, so they are the natural blocklist. Think before blocking the answer bots: OAI-SearchBot and PerplexityBot are the crawlers behind ChatGPT and Perplexity shopping answers, and blocking them removes your products from those results. Many stores want the citation and not the training, so they block the trainers and allow the answer bots. That split is explained in our guide to allowing Googlebot while blocking AI crawlers.
Generate the file rather than hand-writing it: our free robots.txt generator builds the stanzas for you, and the robots.txt tester checks the syntax before you publish. One misspelled bot name silently breaks a rule, and robots.txt never shows you an error.
One caution: do not block the paths that feed your own shopping visibility. If your products appear in Google Shopping, the crawlers behind those surfaces need access to your product pages. Blocking a path for GPTBot does not affect Googlebot, because the rules are per user-agent — but a careless 'User-agent: * Disallow: /products/' would hurt every crawler at once. Keep the blocks targeted at named AI trainers.
Layer 2: server-level blocking and rate limiting
Robots.txt stops the honest bots. Server rules stop the rest — or at least slow them down enough to matter. If you run your own server, this is the layer with the highest return.
The basic rule is user-agent matching: return a 403 to requests from known AI training crawlers on your product paths. On Nginx that is a small map block plus a check in your server block; the full setup is covered in our guide to blocking AI crawlers with Nginx. If your store runs Apache, the same idea applies with server-level rules for your product directories — build them with our .htaccess generator rather than writing them by hand.
Rate limiting matters more than outright blocking for price scraping, because the most aggressive price scrapers do not use AI bot user agents at all — they use rotating residential IPs and ordinary browser strings. A limit like 'ten product-page requests per minute per IP' is invisible to a human shopper, who reads one page at a time, and devastating to a scraper that needs ten thousand pages an hour. Nginx's limit_req module, Apache's rate-limit modules, and Cloudflare's rate-limiting rules all do this, and Cloudflare's version works even if you do not control the server. Set the limit on product and search-result pages, not on the homepage or checkout, so you never slow down a real buyer.
Also watch for the patterns that mark a scraper: requests arriving in perfect alphabetical order through your catalog, the same IP reading every product in a category within minutes, or traffic spikes at 3 a.m. from a single network. Your access logs show all of this. A weekly ten-minute look at the top IPs hitting /products/ will tell you more about your scraping problem than any dashboard.
Layer 3: make prices hard to extract at scale
This is the layer most stores skip, and it is the one that stops the scrapers robots.txt and firewalls miss. The idea is simple: a human should see the price instantly, but a program that downloads your raw HTML should not find it sitting in a neat, predictable place.
Render prices with JavaScript. If your price is written directly into the HTML, any scraper can grab it with a single request. If it is loaded by JavaScript after the page renders, a basic scraper gets a page with no price at all. This is not a perfect defense — headless browsers render JavaScript too — but it filters out the cheapest, highest-volume scrapers, which are the ones hitting small stores. Check first that your own setup still works: Googlebot renders JavaScript, so your prices stay visible to search, but test it in Search Console's URL inspection tool after any change.
Break the pattern. Scrapers rely on your pages looking the same. If every product page has the price in the same element with the same class name, one selector harvests your whole catalog. Vary the markup, split the price across elements, or serve it as an image for the small number of pages that matter most. None of this stops a determined scraper, but each change forces them to re-engineer their script — and re-engineering costs them more than moving to your competitor's site.
Gate bulk access behind an API key. If you offer a product feed, a partner API, or a bulk pricing endpoint, require a key and log the usage. A scraper that wants ten thousand prices should have to ask for them through a channel you control, where you can set limits, see who is asking, and cut off abuse. This also gives you a legitimate path for the partners you do want — affiliates, comparison sites you approve of — so you are not just saying no to everyone.
Do not publish a clean price feed you do not need to. Some stores ship a public XML or JSON feed of their full catalog with prices for shopping surfaces, then wonder why scrapers have their prices. If a feed exists, ask who actually consumes it. If the answer is one shopping channel, restrict it to that channel's crawler or put it behind authentication. A public feed is a gift to every scraper on the internet.
What about structured data and shopping results?
Here is the tension every store faces: the same price data that feeds scrapers also feeds Google Shopping, rich results, and AI shopping answers. Product schema markup — the structured data that puts a price in search results — is read by the same crawlers you are trying to slow down. You cannot publish schema for Google and hide it from everyone else; it is the same public file.
So decide deliberately. If Google Shopping drives real revenue, keep your product schema complete and accurate, and accept that the price is public — then rely on layers two and three (rate limiting and extraction friction) rather than on hiding the number. If a product line is high-margin, low-volume, or sold mainly through direct relationships, consider dropping schema from those pages and letting buyers see the price on the page without broadcasting it in machine-readable form. The mistake is doing this by accident: publishing full schema everywhere while wondering why your prices are quoted in AI answers.
Also remember that blocking a training crawler does not remove prices it already collected. If GPTBot or CCBot already crawled your catalog, that data is in their corpus. A block stops future collection; it is not a recall. Start now, and the value compounds — but do not promise your team that yesterday's prices will disappear from AI answers.
Platform notes: Shopify, WooCommerce, and the rest
On Shopify, you edit robots.txt through a robots.txt.liquid template in your theme — Shopify has supported that since 2021 — and you can add the AI trainer stanzas there. Shopify also offers bot detection through Shopify Protect and through apps, but there is no server config to edit, so your enforcement layer is Cloudflare or a bot-management app. The full Shopify walkthrough is in our guide to blocking AI crawlers on Shopify.
On WooCommerce and WordPress, you have the most control: edit robots.txt directly, add server rules through your host, and use a security plugin to rate-limit aggressive IPs. Because WordPress is the most-scraped platform on the web, the plugin route is well-trodden — but audit any plugin before installing it, and prefer server or Cloudflare rules over a plugin that runs inside WordPress itself, where it consumes the same resources it is trying to protect.
On hosted builders like Wix, Squarespace, or Webflow, you are in the same position as any no-code user: robots.txt editing where the platform allows it, and a service in front of your domain for anything stricter. Check the platform's own docs for robots.txt support before you plan around it, because the controls vary widely and change often.
When a scraper ignores everything
Some scrapers will ignore your robots.txt, rotate IPs to beat your rate limits, and render your JavaScript. At that point you are dealing with a determined actor, and the honest answer is that no purely technical defense is airtight. Your remaining options are legal and commercial, not technical.
Your terms of service should state plainly that automated collection of pricing data is not permitted — it will not stop a scraper, but it gives you a basis to act. If a specific company is scraping you, a documented complaint to their hosting provider or a formal legal letter often works where technology did not. And if the scraper is a competitor, remember that your best defense is a store worth buying from: scrapers can copy a price, but they cannot copy your fulfillment, your support, or your reputation. Price is one signal; the buying experience is the product.
FAQ
Will blocking GPTBot hide my prices from customers?
No. Robots.txt and server rules only affect crawlers. Human visitors, and the search crawlers you allow, see your prices exactly as before. Only the named AI training bots are turned away.
Does blocking AI crawlers remove my products from Google Shopping?
Not if you block only the AI trainers. Google Shopping uses Google's own crawlers, which are separate user agents. Keep Googlebot allowed and your shopping listings keep working. The danger is a careless blanket rule that blocks every bot from your product paths.
Can a scraper still get my prices if I block GPTBot?
Yes. Blocking the named training crawlers stops those crawlers, not every scraper. A scraper using a fake browser user agent will pass a user-agent rule. That is why rate limiting and extraction friction matter: they raise the cost for every scraper, not just the honest ones.
Should I remove price schema from my product pages?
Only if you are willing to give up rich results and shopping visibility for those products. For most stores the answer is no — keep the schema, and protect prices with rate limiting and rendering friction instead. Consider dropping schema only for high-margin or relationship-sold lines.
How do I know if my prices are being scraped?
Look at your access logs: one IP reading hundreds of product pages in minutes, requests in alphabetical catalog order, or traffic spikes at odd hours are the classic signs. If you are behind Cloudflare, its analytics shows bot traffic by user agent and IP without any log work.
Is it legal to scrape prices from my store?
That depends on your jurisdiction, your terms of service, and how the scraping is done. Public prices are facts, and facts are hard to own — but systematic scraping can still breach terms of service, overload your server, or violate data-protection rules. Talk to a lawyer before threatening or taking legal action, and keep your terms of service explicit that automated price collection is not permitted.
The bottom line
Protecting your prices is a layering problem, not a single setting. Robots.txt stops the honest AI trainers. Server rules and rate limits stop the aggressive ones. Rendering friction and gated APIs make bulk extraction expensive for everyone else. Start with the robots.txt rules today — they take ten minutes — then add rate limiting this week, and review your logs monthly to see what is still getting through. Your prices will never be invisible, but they can be expensive enough to steal that most thieves walk past your store.
