To block AI crawlers on Shopify, create a
robots.txt.liquidtemplate (Online Store > Themes > Edit code > Templates > Add template), then append dedicatedUser-agentblocks withDisallow: /for each training crawler — GPTBot, ClaudeBot, CCBot, Bytespider — and for the training tokens Google-Extended and Applebot-Extended. Shopify gives you no server-level blocking, so robots.txt is your lever: it works because the major AI crawlers honor it, not because anything enforces it.
Shopify store owners face a specific frustration in the AI-crawler debate. On a normal host you'd stack robots.txt (the polite request) with .htaccess or firewall rules (the enforcement). On Shopify, the enforcement layer doesn't exist: no .htaccess, no Nginx config, no IP blocking. Your entire crawler-control toolkit is one Liquid template.
The good news: for the crawlers that matter most, that one template is enough. GPTBot, ClaudeBot, CCBot, PerplexityBot, and Apple's and Google's training tokens all honor robots.txt — including the version Shopify renders. This guide walks through creating the template, adding AI-crawler rules correctly, testing them, and understanding what Shopify can't do (and the workarounds for that).
Why AI crawlers are a special problem on Shopify
Three Shopify realities shape everything below:
- robots.txt is generated, not a file. Shopify renders it from Liquid code on every request. You can't upload a static robots.txt; you customize the template that generates it. Until you create the template, you get Shopify's defaults (which sensibly block
/admin,/cart,/checkout, and internal search URLs, but say nothing about AI crawlers). - There is no hard-blocking layer. No
.htaccess, no server config, no IP deny lists. A scraper that ignores robots.txt cannot be stopped inside Shopify. Enforcement requires something in front of Shopify (a CDN/WAF like Cloudflare) — covered later. - The sitemap is automatic. Shopify generates
/sitemap.xmlitself, split by resource type, and excludes noindexed pages automatically. Don't fight it; just reference it correctly.
If you run WordPress instead, the playbook differs — see how to block AI crawlers on WordPress for the plugin and server-level options Shopify lacks.
Step 1: Create the robots.txt.liquid template
- In Shopify admin, go to Online Store > Themes.
- On your live theme, click the … menu and choose Edit code.
- In the left sidebar, open the Templates folder and click Add a new template.
- Choose robots.txt from the template dropdown and click Create template. Shopify scaffolds the file with the default Liquid code that outputs its standard rules.
- Back up before editing. Copy the entire default template into a text file on your computer. To reset everything later, you simply delete the
robots.txt.liquidtemplate and Shopify reverts to its defaults.
The default template loops over Shopify's rule groups with Liquid objects (robots.default_groups, group.user_agent, group.rules) and prints them. You will keep that loop intact — it carries Shopify's well-optimized defaults — and add your AI-crawler rules after it. Shopify's own documentation for this process is at help.shopify.com — editing robots.txt.
Step 2: Add the AI-crawler rules
Keep the default loop exactly as Shopify generated it, then append plain-text User-agent blocks at the bottom of the template. Plain text works fine here — Liquid passes through anything that isn't a {% %} or {{ }} tag, and the output is a valid robots.txt. This is the simplest reliable pattern:
{% for group in robots.default_groups %}
{{- group.user_agent -}}
{% for rule in group.rules %}
{{- rule -}}
{% endfor %}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
# AI training crawlers - opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
# Training-permission tokens (not crawlers - no search effect when blocked)
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /A few decisions embedded in that snippet, so you can adjust them deliberately:
- Training crawlers blocked, search crawlers allowed. GPTBot, ClaudeBot, CCBot, Bytespider, and Amazonbot exist to collect training data. The search-facing crawlers — Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot — are left alone, because blocking them removes you from search results and AI answers rather than from training sets. For the full reasoning per bot, see should you block AI crawlers: pros and cons.
- ChatGPT-User and Claude-User are a judgment call. These fetch pages live when a user asks the assistant to look at a URL. Blocking them doesn't affect training; it stops assistants from reading your pages on demand. E-commerce stores often leave them allowed — a ChatGPT user asking about your product and getting an accurate answer is a sale, not a threat.
- The
-Extendedtokens are separate blocks. Google-Extended and Applebot-Extended aren't crawlers and have no user-agent to match at a firewall; they're permission signals read from robots.txt. Blocking them has no effect on Google Search or Siri/Spotlight visibility. - Applebot (the real crawler) is intentionally absent. It powers Siri and Spotlight results. Unless you want to disappear from Apple search surfaces, leave it allowed.
If you'd rather generate the bot list than hand-write it, the free robots.txt generator produces the AI-crawler blocks — paste its output into the bottom of your template.
Step 3: Test before you trust it
- View the rendered file. Open
yourstore.com/robots.txtin a browser. Confirm your new blocks appear below Shopify's defaults and the syntax is clean — one directive per line, no Liquid tags leaking into the output. - Validate the syntax. Paste the rendered output into the free robots.txt tester and check each AI-crawler user-agent resolves to
Disallow: /whileGooglebotstill shows allowed. For the full testing discipline, see how to test robots.txt before publishing. - Confirm Googlebot is unaffected. This is the check that matters most. A misplaced
Disallow: /underUser-agent: *de-indexes your store. Verify in the rendered file that the wildcard group still allows crawling, then watch Search Console's Page indexing report for a week after the change. - Re-test after theme updates. The template lives in your theme. Switching themes or major theme updates can drop custom templates — check
/robots.txtafter any theme change.
What Shopify can't do: hard blocking and the workarounds
Be honest about the limit: robots.txt is a voluntary protocol. The major AI labs honor it, but scrapers, aggressive data harvesters, and bots that spoof user-agents will walk straight past your carefully crafted template. On a self-hosted stack you'd answer with .htaccess or firewall rules. On Shopify, the workarounds are:
- Cloudflare in front of Shopify. Pointing your domain's DNS through Cloudflare (orange-cloud proxy) lets you use Cloudflare's bot management, WAF user-agent rules, and the one-click AI-bot block — real enforcement, before requests reach Shopify. Caveats: configure SSL correctly (Full Strict) and expect some setup care around Shopify's domain verification. This is the only true hard-blocking route for Shopify stores.
- Bot-protection apps. Several Shopify apps offer bot filtering and rate limiting at the application layer. They can't match edge enforcement, but they help against unsophisticated scrapers. Evaluate any app on whether it distinguishes AI training crawlers from search crawlers — blunt tools that block "all bots" will hurt your SEO.
- Reduce what's worth scraping. The unglamorous fix: don't publish your full catalog data in easily scraped formats, use Shopify's built-in protections where they exist, and keep an eye on server-log-style analytics (via Cloudflare or an app) for unusual crawl patterns.
For most stores, the honest stack is: robots.txt.liquid rules (handles the compliant majority, including all major AI labs) + Cloudflare in front (handles the non-compliant remainder). Either alone leaves a gap.
Safety checklist: don't accidentally block Google
Shopify's default template already contains groups for specific crawlers alongside the wildcard * group. When adding rules:
- Never add
Disallow: /to theUser-agent: *group — that blocks every crawler including Googlebot. - Give each AI crawler its own
User-agentblock rather than editing shared groups. - Don't "clean up" Shopify's default disallows (
/admin,/cart,/checkout) unless you understand why they're there — removing them can cause indexing of cart and checkout pages. - After any edit, re-check the rendered
/robots.txt— Liquid errors can silently produce malformed output.
Resetting to Shopify's defaults
If your edits go wrong or you want a clean slate: in the theme code editor, find robots.txt.liquid under Templates and delete it. Shopify immediately reverts to its default generated robots.txt. There's no undo for the deletion itself, which is why the backup in Step 1 matters — but the reset path is one click and always available.
How long until AI crawlers honor the change?
There's no guaranteed timeline, and anyone promising one is guessing. What we know: major crawlers re-fetch robots.txt regularly — Googlebot typically refreshes its cached copy within about 24 hours. Anthropic, OpenAI, and Common Crawl crawlers also re-read the file on their own schedules, generally within days, not weeks. Training-data opt-outs additionally depend on each lab's data pipeline: a rule you publish today governs future training runs, not models already trained on previously crawled content. Set the rules, verify the rendered file, then check your bot traffic over the following two to four weeks rather than expecting overnight silence.
FAQ
Can I upload a static robots.txt file to Shopify?
No. Shopify doesn't allow static robots.txt uploads; the file is always generated from the robots.txt.liquid template in your theme. Customization happens through that template's Liquid code, with plain-text User-agent blocks appended at the bottom being the simplest approach.
Will blocking AI crawlers in Shopify's robots.txt hurt my Google rankings?
Not if you block the right tokens. Training crawlers (GPTBot, ClaudeBot, CCBot) and training-permission tokens (Google-Extended, Applebot-Extended) are separate from search crawlers. Blocking them has no effect on Google Search rankings. Only blocking Googlebot itself — or a misplaced wildcard Disallow — would hurt rankings.
Does Shopify's robots.txt.liquid support Crawl-delay?
You can write a Crawl-delay line in the template and it will render, but most major crawlers (including Googlebot) ignore Crawl-delay. It's not a reliable rate-limiting tool on any platform. For crawl-rate problems, use Search Console's crawl settings (where available) or edge rate-limiting via Cloudflare.
Why do my AI-crawler rules disappear after a theme update?
The robots.txt.liquid template belongs to your theme, not your store. Switching themes replaces all templates, and some major theme updates can reset custom ones. After any theme change, open yourstore.com/robots.txt and confirm your blocks are still there; keep your backup snippet handy for quick re-application.
Can I block a specific scraper's IP address on Shopify?
No — Shopify exposes no IP-blocking controls to merchants. IP and user-agent enforcement requires a layer in front of Shopify, such as Cloudflare's WAF rules, or a bot-protection app. robots.txt alone cannot stop a bot that ignores it.
