Yes, you can block Amazonbot with robots.txt. Add these two lines to the robots.txt file in your site root, and Amazonbot will stop crawling your site:
User-agent: Amazonbot
Disallow: /That is the short answer. The longer answer matters, because Amazon runs three separate crawlers, and blocking the wrong one — or only one of the three — can produce results you did not intend. Amazonbot itself powers product improvements and may be used to train Amazon AI models. Two sibling crawlers, Amzn-SearchBot and Amzn-User, handle search results and live Alexa answers, and they follow different rules.
This guide walks through what each Amazon crawler does, the exact robots.txt code to block or allow each one, how to allow Amazonbot on some parts of your site while keeping it out of others, and how to check that your rules actually worked. Everything here is based on Amazon's own published crawler documentation.
What Is Amazonbot?
Amazonbot is Amazon's web crawler. It visits websites across the internet, reads pages, and sends what it finds back to Amazon. According to Amazon's own documentation, Amazonbot is "used to improve our products and services" and "may be used to train Amazon AI models." That single sentence is the reason most site owners care about it: if Amazonbot can crawl your content, that content may end up as training data for Amazon's AI systems.
Amazonbot identifies itself with a user-agent string that looks like this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36The important part for robots.txt is the token Amazonbot. When you write User-agent: Amazonbot in robots.txt, you are targeting this crawler specifically. Robots.txt matching is based on the token, not the full user-agent string, so you do not need to reproduce the whole line.
Amazonbot is not a search engine crawler in the Googlebot sense. Blocking it will not remove your site from Google, Bing, or any search engine. It only affects Amazon's crawlers. That separation is useful: you can make a decision about Amazon without touching your search visibility at all.
Amazon Runs Three Crawlers, Not One
This is the detail most guides get wrong. Amazon documents three distinct user agents, and each one is independent of the others. A robots.txt rule aimed at one does not apply to the others. If you block Amazonbot but leave the other two alone, Amazon can still fetch your pages through them.
1. Amazonbot
The general-purpose crawler. It gathers content that improves Amazon's products and services, and it may be used to train Amazon AI models. This is the crawler most site owners mean when they say "block Amazonbot." It is also the only one of the three tied to AI model training.
2. Amzn-SearchBot
The search crawler. It improves search experiences inside Amazon products and services. Amazon says that by permitting Amzn-SearchBot access to your site, your content is eligible to appear in search experiences such as Alexa. Two things set it apart:
- It does not crawl content for generative AI model training.
- If your robots.txt does not mention Amzn-SearchBot but allows other search bots, it follows the robots.txt directives given to other search bots. In other words, it inherits the rules you wrote for search crawlers generally.
That fallback behavior means a generic User-agent: * block also stops Amzn-SearchBot, but a rule that names only Amazonbot does not.
3. Amzn-User
The live fetcher. Amzn-User supports user actions, such as answering an Alexa question that needs up-to-date information. When someone asks Alexa something and the answer needs fresh web data, Amzn-User may fetch a live page on that person's behalf. Amazon states that because Amzn-User actions can be started by a user, it "may not follow all robots.txt directives." It does not crawl content for generative AI model training.
Here is the practical summary:
- Amazonbot: general crawl, possible AI training. Respects robots.txt.
- Amzn-SearchBot: search and Alexa answers. Respects robots.txt, falls back to other search-bot rules. No AI training.
- Amzn-User: live fetch for a user's Alexa question. May not follow all robots.txt directives. No AI training.
Keep this table in mind as you read the code examples below. Most of the time, the right move is to write rules for all three, with each one getting the treatment that matches what it does.
How to Block Amazonbot With robots.txt
Blocking Amazonbot takes two lines. Open your robots.txt file, which lives at the root of your domain — for example, https://yoursite.com/robots.txt — and add a group for Amazonbot.
Step 1: Open your robots.txt file
If you run WordPress, many SEO plugins include a robots.txt editor. If you run a static site or a custom setup, the file sits in your site's root folder on the server. If you have no robots.txt file at all, create a plain text file named exactly robots.txt and upload it to the root. A robots.txt file in a subdirectory does not count; crawlers only check the root.
Step 2: Add the block rule
To block Amazonbot from your entire site:
User-agent: Amazonbot
Disallow: /Disallow: / means "stay out of every path on this site." That is the whole rule. There is nothing else to add for a full block of Amazonbot itself.
Step 3: Block all three Amazon crawlers
If your goal is to keep Amazon's crawlers off your site entirely, block all three user agents. Remember that Amzn-User may not honor every directive, so this is the strongest robots.txt-level control available, not a guarantee against every live fetch:
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /
User-agent: Amzn-User
Disallow: /Put each group in its own block, separated by a blank line. Do not combine them into one group with multiple User-agent lines unless you are sure your setup handles that correctly — separate groups are clearer and less error-prone.
Step 4: Keep your other rules intact
Your robots.txt file almost certainly has rules for other crawlers. Keep them. A typical file with an Amazon block alongside a general allow looks like this:
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /
User-agent: Amzn-User
Disallow: /
User-agent: *
Allow: /The User-agent: * group at the bottom applies to every crawler that does not have its own group. Amazon's crawlers match their own groups first, so they stay blocked while everyone else keeps crawling normally.
Step 5: Save, upload, and confirm the file is live
Save the file and visit https://yoursite.com/robots.txt in your browser. You should see your new rules. If you see an old version, clear your cache or CDN cache and check again. Amazon fetches robots.txt at the host level, so a rule on www.yoursite.com does not apply to yoursite.com without the www, or to a subdomain like blog.yoursite.com. If you run several hosts, add the rules to each host's robots.txt file. If you would rather build the file with a tool than by hand, the robots.txt generator creates a valid file you can paste your Amazonbot rules into.
How to Allow Amazonbot Selectively
A full block is not always the right call. Many sites want Amazonbot on their public articles but not on member areas, checkout pages, staging directories, or internal tools. Robots.txt supports that with Allow and Disallow pairs.
Block one directory, allow the rest
To keep Amazonbot out of a single folder, such as a members area:
User-agent: Amazonbot
Disallow: /members/
Disallow: /account/Everything outside those two folders stays open to Amazonbot. Use trailing slashes for directories so the rule matches the whole folder, not just a single page.
Block the whole site except one section
To block Amazonbot everywhere except your blog:
User-agent: Amazonbot
Disallow: /
Allow: /blog/The Allow line carves out an exception to the Disallow: / rule. When rules conflict, crawlers that follow the Robots Exclusion Protocol apply the most specific matching rule, so /blog/ wins for blog URLs and everything else stays blocked.
Allow search, block training
This is the split many publishers actually want. It lets your content stay eligible for Alexa search answers while keeping it out of Amazon's AI training crawl:
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Allow: /Because Amazon says Amzn-SearchBot does not crawl for generative AI model training, allowing it while blocking Amazonbot is a coherent middle ground: your pages can still surface in Alexa search experiences, but the training crawler stays out. Leave Amzn-User alone in this setup, since it only fetches pages when a user asks Alexa a question that needs fresh data, and Amazon says it does not train models either.
Keep specific file types away from Amazonbot
If you want to keep PDFs, images, or downloads out of Amazon's crawl while leaving your articles open:
User-agent: Amazonbot
Disallow: /*.pdf$
Disallow: /downloads/The $ marks the end of the URL, so /*.pdf$ matches any URL ending in .pdf. Wildcards like * are supported by most major crawlers, including Amazon's.
Block vs. Allow: How to Decide
There is no single right answer. The choice depends on what your site does and how you feel about your content feeding Amazon's AI models. Here is a simple way to think through it.
Block Amazonbot if:
- Your content is your product. If you sell articles, research, courses, photography, or data, letting a crawler harvest it for model training gives away the thing you charge for. Blocking Amazonbot protects the core asset.
- You license content exclusively. Some licensing deals require you to keep content out of AI training sets. A robots.txt block is the standard way to record that intent.
- You have seen heavy crawl load. If Amazonbot hammers your server and slows the site for real visitors, a block — or a partial block of the heaviest sections — is a fair response.
- You see no value from Amazon surfaces. If your audience does not come through Alexa or Amazon search experiences, there is little upside to letting the crawler in.
Allow Amazonbot if:
- Your audience uses Alexa. If your readers or customers ask Alexa questions your content answers, allowing Amzn-SearchBot keeps you eligible for those answers.
- Discovery matters more than control. Recipe sites, local businesses, and how-to publishers often gain more from being found in one more place than they lose from a crawl.
- You are comfortable with AI training use. Some publishers see model training as a form of distribution — their ideas reaching people through AI answers. If that is your view, there is no reason to block.
The middle path most sites should consider
For a typical blog or business site, the split rule — block Amazonbot, allow Amzn-SearchBot — is the best default. It keeps your content out of the training crawl while preserving your eligibility for Alexa search answers. You can always tighten or loosen it later. The decision is reversible: change the file, and Amazon picks up the new rules when it next fetches robots.txt.
What Blocking in robots.txt Does Not Do
Robots.txt is a request, not a lock. Understanding its limits keeps you from being surprised later.
Robots.txt is voluntary
The Robots Exclusion Protocol works because well-behaved crawlers honor it. Amazon says its automated crawling respects the protocol, honoring the user-agent and the allow and disallow directives. But a crawler that chose to ignore your file could still fetch your pages. Robots.txt stops polite crawlers; it does not stop a determined scraper.
Amzn-User may still fetch pages
Amazon is explicit that Amzn-User, the live fetcher, may not follow all robots.txt directives because its fetches are triggered by a user's request. If someone asks Alexa a question that needs your page, Amzn-User may fetch it even if you blocked it. If you need hard enforcement against live fetches, you need server-level controls — firewall rules, bot management, or an edge rule that returns a block response to the Amzn-User user-agent — not robots.txt alone.
Blocking does not remove content Amazon already has
A robots.txt block stops future crawling. It does not erase pages Amazon already fetched. If you want content gone from an index, you need removal tooling or a noindex directive, not just a crawl block. Note the difference: Disallow in robots.txt tells a crawler not to visit. noindex tells an index not to keep the page.
Use meta tags for page-level control
Amazon says its crawlers respect page-level robots meta tags: noarchive (do not use the page for model training), noindex (do not index the page), and none (do not index the page). A meta tag on a single page is more precise than a robots.txt rule when you want to keep one article out of training while leaving the rest of the site open:
<meta name="robots" content="noarchive">This tells crawlers not to use the page for model training. It is a good companion to robots.txt when you want fine-grained control.
Amazon does not support crawl-delay
Amazon states that its crawlers do not support the crawl-delay directive. If Amazonbot is crawling too fast for your server, do not add Crawl-delay: 10 and expect it to slow down. Use a partial block, server-level rate limiting, or contact Amazon at the address listed at the end of this article.
How to Verify Your Amazonbot Block Worked
Adding rules to robots.txt is only half the job. Amazon caches robots.txt for up to 30 days, and it says settings may take about 24 hours to show up in its systems. So verification needs two steps: confirm the file is correct, then confirm the crawler's behavior changed.
1. Confirm the file is live and correct
Visit https://yoursite.com/robots.txt in a browser and read it. Check for typos: User-agent must be spelled correctly, the token must be Amazonbot (not "AmazonBot" with a capital B in the wrong place — matching is case-insensitive for the field name, but keeping the documented token casing avoids confusion), and every Disallow line must start with a slash. A quick way to validate syntax is the robots.txt tester, which checks your file against a user agent and URL before you rely on it.
2. Check your server logs for Amazonbot visits
Your web server logs show every request, including the user-agent string. Search the logs for Amazonbot and look at the timestamps:
grep -i "amazonbot" /var/log/nginx/access.log | tail -20If your block is working, you should see Amazonbot requests drop off. A few visits right after you publish the change are normal — Amazon may still hold a cached copy of your old robots.txt for up to 30 days, and the new rules take effect when it fetches the fresh file. What you want to see is the trend: requests falling to zero over the following days and staying there.
3. Watch for the cached robots.txt delay
Amazon says it fetches host-level robots.txt files or uses a cached copy from the last 30 days. That means a block you publish today might not change Amazonbot's behavior until its next fetch of your robots.txt. Do not panic if you still see Amazonbot hits the next morning. Give it a few days, then check the logs again. If Amazonbot is still crawling heavily after two weeks, re-check the file for errors, confirm it is served from the correct host, and confirm your server returns a 200 status for it — Amazon says that when a robots.txt file cannot be fetched, it behaves as if the file does not exist.
4. Confirm the block at the host level
Amazon reads robots.txt per host. If your site answers on both yoursite.com and www.yoursite.com, check both URLs. A rule on one host does not cover the other. The same goes for subdomains. This is the most common reason a "block" appears to fail: the rule exists, but on the wrong host.
5. Check with a user-agent test
You can simulate a crawler request with curl, sending Amazonbot's user-agent string to your own site, to confirm your server and any edge rules behave as expected:
curl -s -A "Mozilla/5.0 (compatible; Amazonbot/0.1)" https://yoursite.com/robots.txtThis shows you what Amazonbot sees when it requests your robots.txt. It does not prove Amazonbot obeyed the rule, but it proves the rule is visible to the crawler, which is the part you control.
Common Mistakes When Blocking Amazonbot
- Blocking only Amazonbot and forgetting the other two. If you also want to stop Alexa search crawling and live fetches, write rules for Amzn-SearchBot and Amzn-User too.
- Using the wrong token. The robots.txt token is
Amazonbot. Tokens like "Amazon" or "AmazonBotCrawler" do not match, and the rule will be ignored. - Putting robots.txt in a subdirectory. Only
yoursite.com/robots.txtcounts. A file atyoursite.com/site/robots.txtis never read. - Adding Crawl-delay and expecting it to work. Amazon does not support it. Use server-level rate limiting instead.
- Assuming the block is instant. Amazon caches robots.txt for up to 30 days and says changes may take about 24 hours to reflect. Wait, then verify in logs.
- Forgetting subdomains and www vs. non-www. Each host needs its own robots.txt rules.
- Confusing blocking with removal. A
Disallowrule stops future visits. It does not delete content Amazon already collected. Usenoindexor removal tooling for that. - Blocking Googlebot by accident. A sloppy
User-agent: *block stops every crawler, including Googlebot. Always put Amazon-specific rules in their own group and leave your general rules alone.
FAQ
Does blocking Amazonbot hurt my Google rankings?
No. Amazonbot is Amazon's crawler, not Google's. Blocking it has no effect on Google Search, Googlebot, or your position in search results. Google's own robots.txt guide makes the same general point: robots.txt controls which crawlers may access your URLs, and a rule aimed at one crawler does not affect the others.
Will blocking Amazonbot remove my content from Alexa answers?
Blocking Amazonbot alone does not necessarily remove you from Alexa answers, because Alexa search experiences are served by Amzn-SearchBot, a separate crawler. To stop Alexa search crawling, block Amzn-SearchBot as well. Note that Amzn-User, the live fetcher, may still fetch a page when a user asks Alexa a question that needs fresh information.
How long does it take for an Amazonbot block to work?
Amazon says each user-agent setting is independent and may take about 24 hours for its systems to reflect changes. It also caches robots.txt for up to 30 days, so the new rules apply once it fetches your updated file. In practice, expect a few days before crawler behavior fully changes, and verify with server logs rather than assuming the change was instant.
Can I block Amazonbot from AI training but allow it for search?
Yes — that is the split-rule approach. Block Amazonbot, which is the crawler Amazon ties to AI model training, and allow Amzn-SearchBot, which Amazon says does not crawl for generative AI model training and powers search experiences such as Alexa. You can also add a noarchive meta tag to individual pages to signal that they should not be used for model training.
Does Amazonbot respect robots.txt?
Yes, according to Amazon's current documentation. Amazon says its automated crawling respects the Robots Exclusion Protocol, honoring the user-agent and the allow and disallow directives. Older articles sometimes claim Amazonbot ignored robots.txt; that no longer matches Amazon's published position, so check the date on anything you read about this crawler. One exception remains: Amzn-User may not follow all robots.txt directives because its fetches are triggered by live user requests.
What should I do if Amazonbot keeps crawling after I blocked it?
First, wait — cached robots.txt can delay the change. Then check that your file is served correctly at the root of the right host with a 200 status, and that the token is spelled Amazonbot. Check your logs to confirm the requests really are Amazonbot and not another crawler with a similar name. If everything checks out and the crawling continues, contact Amazon at amazonbot@amazon.com and include your domain name, as Amazon's documentation instructs.
The Bottom Line
Blocking Amazonbot with robots.txt is a two-line job, but doing it well means handling all three Amazon crawlers. Block Amazonbot to keep your content out of Amazon's AI training crawl. Decide separately whether Amzn-SearchBot may keep your pages eligible for Alexa search answers, and remember that Amzn-User may still fetch pages when a real person asks Alexa a question. Write the rules, publish them at the root of every host you run, wait for Amazon's cache to refresh, and confirm the result in your server logs. If you want to build the file without hand-editing it, the robots.txt generator will produce a clean file, and the robots.txt tester will check it before you publish. For a broader look at how crawler control fits into AI search visibility, see the guides on the CrawlGuards blog.
