Cohere operates two crawler tokens you may see in your server logs: cohere-ai and cohere-training-data-crawler. If you want Cohere's products to stop using your content, block both. If you only want to opt out of AI training while keeping live retrieval working, block only the training token. Here is the full breakdown — including one honest caveat: Cohere has not published detailed documentation about what cohere-ai does, so part of this picture comes from community observation rather than an official spec.
# Block both Cohere crawlers User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: /
That block is the complete opt-out. The rest of this guide explains what each token does, why the distinction between them matters, and how to enforce the block if robots.txt alone is not enough.
What Is Cohere, and Why Does It Crawl Your Site?
Cohere is an AI company whose products focus on enterprise search, retrieval-augmented generation (RAG), and embeddings — the technology that lets software find meaning in text rather than just matching keywords. Unlike consumer chatbots, Cohere's products mostly run behind other companies' applications: a support search box, a document assistant, a research tool.
That matters because Cohere's crawlers serve two different purposes, and your decision about them should be different for each:
- Training: collecting web content to train or improve models. Once your content is in a training set, you cannot take it back. Opting out only prevents future collection.
- Live retrieval: fetching a page at the moment a user or application asks about it, so the answer can be grounded in a real source. This is closer to what a search engine does.
Cohere's two tokens map roughly onto those two purposes — but, as we will see, the mapping for one of them is not officially confirmed.
The Two Cohere Crawler Tokens
cohere-training-data-crawler — the training crawler
This is the straightforward one. The name says what it does: it collects training data. If you do not want your content in Cohere's training sets, block this token. This is the token to prioritize, because training is the irreversible use of your content.
User-agent: cohere-training-data-crawler Disallow: /
cohere-ai — the live retrieval fetcher
The cohere-ai token is the one to be careful about. Cohere has not published detailed public documentation explaining exactly what this crawler does, which pages it visits, or what the fetched content is used for. Community sources and bot-tracking services describe it as a live retrieval fetcher — a crawler that visits pages on demand when a Cohere-powered product needs to read them — rather than a bulk training crawler.
That description is plausible and widely repeated, but it is not an official Cohere statement, so treat it as the best available understanding rather than a confirmed fact. What is confirmed is the practical part: cohere-ai is a real user-agent token seen in server logs, it is expected to follow robots.txt rules, and a Disallow rule is the correct first step if you want it to stop visiting.
User-agent: cohere-ai Disallow: /
When a crawler's purpose is not officially documented, the safe move is to decide based on your policy, not on guesses about the bot. If your policy is "no AI company uses our content," block both tokens. If your policy is "no training, but retrieval is fine," block only the training token.
Which One Should You Block?
There are three sensible positions:
Option 1: Block both (maximum protection)
Use this if you publish content you sell — paid articles, licensed data, course material, proprietary research — or if your policy is simply that no AI company may use your content in any way. Blocking both tokens is the cleanest expression of that policy, and it costs you nothing in search rankings, because neither token is a search crawler.
Option 2: Block only the training crawler (the balanced choice)
Use this if your concern is specifically about training data. You keep live retrieval working, which means Cohere-powered products can still fetch and cite your pages — the same trade-off as allowing ChatGPT-User or Perplexity-User while blocking GPTBot. This is the right default for most free-content publishers.
Option 3: Allow both (maximum visibility)
Use this if you want every AI product to be able to read and cite your content. Some publishers take this position deliberately: they would rather be cited everywhere than protected anywhere. That is a legitimate strategy, but it should be a decision, not an accident caused by never writing a robots.txt rule.
Whatever you choose, write it down. A one-line note in your content policy — "we block AI training crawlers but allow retrieval fetchers" — turns a scattered set of robots.txt rules into a policy you can apply consistently as new crawlers appear.
How to Add the Rules
- Open your robots.txt file. It lives at
yourdomain.com/robots.txt. - Add one group per token. Each
User-agentline starts a group, and theDisallowline under it applies to that group only. Keep the two Cohere tokens in separate groups so the file stays easy to edit later. - Keep your
Sitemap:line at the bottom of the file. - Test before you publish. Paste your file into the robots.txt tester and check both tokens. Or generate the whole file with the robots.txt generator and add the Cohere tokens to your block list.
One formatting warning: do not stack both tokens into a single group with a shared Disallow unless you are sure you want both blocked — because that is what a shared group means. Separate groups keep each decision independent and make future edits trivial.
Robots.txt Is a Request — Add Enforcement If You Need It
Robots.txt only works on crawlers that read it and choose to obey. The major AI providers do. Scrapers that ignore robots.txt, or that spoof a browser user agent, will not be stopped by any robots.txt rule, including these.
If you see Cohere-token traffic continuing after a correct robots.txt block, or if you see suspicious traffic with no recognizable bot token at all, add a server-level layer:
- Apache / .htaccess: match the user-agent string and return a 403. Our guide to blocking AI crawlers with .htaccess has copy-paste rules, and the htaccess generator builds them for you.
- Cloudflare: the "AI Scrapers and Crawlers" toggle under Security > Bots blocks known AI bots at the edge, and WAF rules can target specific user agents.
- Server logs: check whether the visits are actually coming from Cohere's tokens or from something impersonating them. A real bot's requests will match its documented behavior; a scraper wearing a borrowed user agent will not.
For most sites, robots.txt plus one enforcement layer is enough. You do not need all three layers — you need the policy layer plus one layer that can say no.
How to Verify the Block
- Read the file. Open
yourdomain.com/robots.txtand confirm both groups are present, spelled correctly, and each followed byDisallow: /. - Test both tokens. Use the robots.txt tester to confirm each token is blocked on your key URLs.
- Watch your logs. Search access logs for
cohere-aiandcohere-training-data-crawler. Because robots.txt is a polite request, allow a reasonable window — crawlers re-fetch robots.txt on their own schedule, so a visit in the first day or two after the change does not prove the block failed. - Check for conflicts. If visits continue, look for a wildcard
Allowgroup that could be overriding your rules, a second robots.txt served by a CDN, or a plugin that generates its own robots.txt and ignores your file.
Common Mistakes
- Blocking only
cohere-aiand thinking training is handled. If training is your concern, the training token is the one that matters. Block it explicitly. - Assuming
cohere-aiis a training crawler. Community evidence points to live retrieval, not bulk training. Do not build your policy on a label Cohere never confirmed. - Typos in the tokens.
cohere-training-data-crawleris long and easy to mistype. Copy it from this page rather than typing it from memory. - Expecting robots.txt to delete past training data. It cannot. It only prevents future crawling. If your content was already collected, a robots.txt rule does not remove it from any dataset.
- Blocking Cohere and forgetting the other twenty crawlers. A single-provider block is not an AI policy. For the full list, see the complete list of AI crawlers, and read should you block AI crawlers before you commit to a site-wide policy.
FAQ
What is the cohere-ai user agent?
cohere-ai is a user-agent token associated with Cohere's products. Community sources describe it as a live retrieval fetcher that visits pages on demand. Cohere has not published detailed official documentation of its purpose, so that description should be treated as the best available understanding, not a confirmed spec.
What is cohere-training-data-crawler?
It is Cohere's training-data crawler token. Blocking it opts your site out of having its content collected for Cohere's model training.
Will blocking Cohere hurt my Google or Bing rankings?
No. Neither Cohere token is a search crawler. Blocking them has no effect on your search rankings or on whether your pages appear in search results.
Does blocking cohere-ai stop Cohere products from citing my site?
Blocking the retrieval token should stop live fetches of your pages by Cohere-powered products. Blocking only the training token leaves retrieval working. If you want full separation from Cohere, block both.
How do I know if Cohere is visiting my site?
Search your server access logs for cohere-ai and cohere-training-data-crawler. Those tokens appear in the user-agent field of each request.
What if the crawlers ignore my robots.txt?
First verify the file is correct and reachable. If visits continue, add enforcement: a .htaccess user-agent rule or a Cloudflare WAF rule that returns a 403 for those tokens.
Bottom line: Cohere gives you two tokens and two separate decisions. Block cohere-training-data-crawler to opt out of training. Block cohere-ai as well if you want no Cohere retrieval either. And remember the caveat: the retrieval token's exact purpose is not officially documented, so base your policy on what you want — not on assumptions about the bot.
