Yes, you can block Mistral AI crawlers with two lines in your robots.txt file. Add a group that names MistralAI-User and disallows everything, and Mistral's user-triggered fetcher will stop visiting your pages. But before you paste that block, it helps to know what you are actually blocking, because Mistral runs three separate crawlers, and only one of them is the fetcher most people mean when they say "Mistral AI crawler." Block the wrong one and you could give up visibility in Mistral's answers for no real gain.
This guide walks through each Mistral crawler, the exact robots.txt code for every scenario, and how to test that your rules work. You will also get a simple decision framework so you can choose between blocking everything, blocking only training, or leaving Mistral alone.
What MistralAI-User Actually Is
MistralAI-User is not a traditional web crawler. According to Mistral's own crawler documentation, MistralAI-User handles user actions in Vibe, Mistral's assistant product. When a user asks Vibe a question, the fetcher may visit a web page to help answer it, and it includes a link to the source in its response.
Two details in that description matter a lot:
- It only runs when a user asks. Mistral states that MistralAI-User is not used for crawling the web in any automatic fashion. It does not wander the web building an index on its own schedule.
- It is not used for training. Mistral states that content fetched by MistralAI-User is not used for generative AI training of any kind. Blocking it does not protect your content from ending up in a training dataset, because it was never going there in the first place.
So when you block MistralAI-User, what you are really doing is telling Mistral: do not fetch my pages to answer a user's live question. The practical effect is that Vibe will not be able to cite your pages. Your content will not show up as a linked source in Mistral's answers. If your business depends on being cited in AI answers, that is a real cost. If you would rather keep your pages out of AI answers entirely, that is exactly the outcome you want.
You can recognize MistralAI-User in your server logs by its full user-agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)Note the token itself: MistralAI-User. That is the string you put in robots.txt. You do not need the whole user-agent string, just the product token. Robots.txt matching is case-insensitive, but it is good practice to copy the token exactly as Mistral spells it.
Meet the Whole Mistral Crawler Family
Mistral documents three crawler tokens. They do different jobs, and robots.txt treats each one as a separate crawler. A rule aimed at one token does nothing to the others.
MistralAI-User: the user-triggered fetcher
As described above, this one fetches pages on demand when a user asks a question. It is the token this article focuses on, because it is the one most site owners ask about. Blocking it removes your pages from Mistral's live answers and citations.
MistralAI-Index: the search indexer
MistralAI-Index is the automated crawler. It crawls the web to build an index for Mistral search, which helps answer user questions in Vibe. Mistral states that content crawled by MistralAI-Index is not used for generative AI training. If you block MistralAI-User but leave MistralAI-Index alone, Mistral can still index your pages for search, but it cannot fetch them live to answer a specific user question. If you block both, your site effectively disappears from Mistral's search results and answers.
MistralAI-Training: the training crawler
MistralAI-Training is the crawler tied to model training. If your main concern is keeping your content out of training datasets, this is the token that matters most. Many publishers block the training crawler while allowing the search and fetcher crawlers, because that combination protects their content from training use while keeping their visibility in AI answers.
The key takeaway: blocking "Mistral" is not one decision. It is three decisions, one per token. Most sites that block Mistral want to block the training crawler. Fewer want to block the indexer and the fetcher, because those two are what make a site visible inside Mistral's products.
Should You Block Mistral? A Simple Decision Framework
There is no universally right answer. The right choice depends on what your site does and where your traffic comes from. Work through these three questions.
1. Do you want to appear in AI answers?
If your site earns money from people discovering it, reading it, and clicking through, then visibility in AI answers is usually worth keeping. When Vibe cites your page, that citation is a referral path, similar to a search result. Blocking MistralAI-User and MistralAI-Index closes that path. For blogs, news sites, documentation sites, and businesses that rely on discovery, the default should be to allow the fetcher and the indexer.
2. Do you object to your content being used for training?
This is a separate question, and for many publishers the answer is yes. Training use does not send you traffic or credit. If you object to it, block MistralAI-Training. You can do that while still allowing MistralAI-User and MistralAI-Index, which keeps your citations and search visibility intact. This split approach, block training, allow retrieval, is the most common posture among publishers who have thought it through.
3. Is Mistral traffic actually costing you something?
Check your server logs before blocking anything for performance reasons. MistralAI-User only fires when a user asks about a page, so its volume is typically low. MistralAI-Index is the automated one, so it is the one that could show up as sustained crawl traffic. If your logs show Mistral crawlers hammering your server, blocking the indexer is a reasonable response. If your logs show a handful of hits a week, blocking for performance reasons solves a problem you do not have.
Rule of thumb: Block MistralAI-Training if you do not want your content used for training. Allow MistralAI-User and MistralAI-Index if you want to stay visible in Mistral's answers. Block all three only if you want no presence in Mistral's products at all.
The Exact robots.txt Code
All of these examples go in the robots.txt file at the root of your site, for example https://example.com/robots.txt. Each example is complete and ready to paste. If you already have a robots.txt file, add these groups alongside your existing rules rather than replacing the whole file.
Option 1: Block MistralAI-User only
Use this if you want to stop live fetches for Vibe answers but keep Mistral's search indexing working.
User-agent: MistralAI-User
Disallow: /Option 2: Block all three Mistral crawlers
Use this if you want no Mistral presence at all: no training, no indexing, no live fetches.
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
Disallow: /Notice that all three tokens sit in a single group, followed by one Disallow: / rule. In robots.txt, a group starts with one or more User-agent lines, and the rules that follow apply to every token listed in that group. This is the standard way to apply the same rule to several crawlers.
Option 3: Block training only, allow search and answers (recommended for most sites)
This is the split approach described earlier. It blocks the training crawler and explicitly allows the other two.
User-agent: MistralAI-Training
Disallow: /
User-agent: MistralAI-User
Allow: /
User-agent: MistralAI-Index
Allow: /The Allow: / lines are technically optional, since anything not disallowed is allowed by default. Including them anyway makes your intent explicit to anyone reading the file, and it guards against a future User-agent: * rule accidentally sweeping Mistral up. Explicit beats clever in a file that other people and tools will read.
Option 4: Block Mistral from part of your site only
You may want Mistral to index your public articles but stay out of your members-only area, your staging section, or your internal docs. Path rules handle that.
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
Disallow: /members/
Disallow: /internal/
Disallow: /staging/Each Disallow line is a path prefix. Disallow: /members/ blocks everything under /members/, including /members/dashboard and /members/settings. It does not block /membership, because that URL does not start with /members/. The trailing slash matters.
Option 5: Block Mistral plus other AI crawlers in one file
If you are doing a broader AI-crawler review, you can put Mistral in the same file as other operators. Keep each operator in its own group so the file stays readable and easy to update later.
# Block AI training crawlers
User-agent: MistralAI-Training
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Allow AI search and retrieval crawlers
User-agent: MistralAI-User
User-agent: MistralAI-Index
Allow: /Comments start with # and are ignored by crawlers. Use them. A robots.txt file that nobody can understand is a file that will eventually be broken by a well-meaning edit.
Step by Step: Add the Rule, Test It, Deploy It
Step 1: Find your current robots.txt
Open https://yourdomain.com/robots.txt in your browser. If you get a 404, you do not have one yet, and you are starting from a blank file. If the file exists, read it before editing. Look for existing groups that mention Mistral tokens or a User-agent: * group, because a wildcard group already sets the baseline for every crawler not named elsewhere.
Step 2: Write the new rules
Pick the option from the previous section that matches your decision. If you are unsure, start with Option 3, block training only. It is the least destructive choice and the easiest to reverse. If you do not want to hand-write the file, CrawlGuards' free robots.txt generator builds a valid file from your choices, including AI crawler rules, without you having to remember the syntax.
Step 3: Check the syntax before you publish
The most common robots.txt errors are small: a missing colon, a typo in the token, a group where the rules come before the User-agent lines, or a Disallow path that does not start with a slash. Run your updated file through a validator. CrawlGuards' robots.txt tester lets you check whether a given URL is allowed or blocked for a specific crawler, which is exactly the check you need here. Test at least three URLs: your homepage, one article, and one URL inside any directory you disallowed.
Step 4: Publish the file at the root
Upload the file to /robots.txt at the root of your domain. It must be served as plain text over the same host it controls. A robots.txt file at https://example.com/robots.txt does not control https://blog.example.com or https://example.com:8443. Each subdomain needs its own file. If you use a CDN or caching layer, confirm the new file is actually being served and not a cached copy of the old one.
Step 5: Confirm it is live and watch your logs
Fetch the URL in your browser and confirm the new rules are there. Then check your server logs over the next few days for the MistralAI-User token. Compliant crawlers fetch robots.txt before crawling, so you should see robots.txt requests from Mistral followed by no further page fetches for the blocked paths. If you still see page fetches after the block, give it a few days: crawlers cache robots.txt, and the cached copy is typically refreshed within about 24 hours. If fetches continue for weeks, something else is going on, and the verification section below will help you diagnose it.
Common Mistakes That Make the Block Fail
Typing the token wrong
Robots.txt matches the product token inside the user-agent string, not the whole string. MistralAI-User is the token. Variations like MistralAI, Mistral, MistralAI-User/1.0, or MistralAI-Bot will not match. Mistral has never documented a token called MistralAI-Bot. If you see it in a blog post or a template, do not use it. Stick to the three tokens Mistral documents: MistralAI-User, MistralAI-Index, and MistralAI-Training.
Putting rules before the User-agent lines
Every group must start with its User-agent lines. Rules that appear before any User-agent line are ignored. This is the easiest mistake to make when pasting a snippet at the top of an existing file.
Assuming the most specific rule wins automatically
Under the robots.txt standard, RFC 9309, as interpreted by Google, when two rules could apply to the same URL, the longest, most specific matching path wins. An Allow: /public/ rule beats a Disallow: / rule for URLs under /public/, because the allow path is longer. This is useful, but it also means a broad block can be quietly overridden by a narrower allow rule elsewhere in the same group. Read the whole group, not just the line you added.
Expecting robots.txt to remove pages from search results
Robots.txt controls crawling, not indexing. Blocking a page does not remove it from an index if it is already there, and it does not stop an index from listing the URL without a description. If you need a page removed from search results, use a noindex directive, and keep the page crawlable so crawlers can see that directive. Robots.txt and noindex solve different problems. Do not mix them up.
Expecting instant, retroactive effect
A new robots.txt rule stops future crawls. It does not delete content that was already collected, and it does not undo a crawl that happened yesterday. If your goal is to remove content that is already in a dataset, robots.txt cannot do that. Your options are the operator's own removal or opt-out process, if one exists, and legal routes for content that was collected without permission.
Trying to block by IP address instead
Mistral publishes IP address lists for its crawlers, which are useful for verifying that a request really came from Mistral. Blocking those IPs at the firewall is a different tool, and it has a sharp edge: if a crawler cannot fetch your robots.txt, it cannot learn that it is blocked, and some crawlers treat an unreachable robots.txt conservatively. Use robots.txt for the opt-out and the IP lists for verification, not the other way around.
How to Verify Mistral Is Respecting Your Rules
Trust, but verify. Robots.txt is a voluntary protocol, so checking that it works is part of the job.
- Check your server logs. Search for
MistralAI-User,MistralAI-Index, andMistralAI-Trainingin your access logs. After a block, you should see requests for/robots.txtfrom those tokens and then no page fetches on blocked paths. The user-agent string is self-reported, so a log hit alone proves nothing about who really sent the request. - Verify the IP address. Mistral publishes IP lists for MistralAI-User and MistralAI-Index. If a request claims to be MistralAI-User, check whether its source IP appears on Mistral's published list. A mismatch means someone is spoofing the token, and your robots.txt rule will not stop them, because they were never following it in the first place.
- Test with the tester. Re-run your file through a robots.txt tester after every edit, not just the first time. Files get edited by many hands over the years, and a tester catches regressions in seconds.
- Watch for the cache window. Crawlers cache robots.txt for roughly a day. Do not conclude the block failed because you saw one fetch the morning after you deployed it. Look at the trend over a week.
What Blocking Mistral Will Not Do
It is worth being honest about the limits. Robots.txt is a request, not a lock. The standard itself says its rules are not a form of access authorization.
- It will not stop scrapers that ignore robots.txt. Bad actors, aggressive scrapers, and datasets built from bulk downloads do not check your file. If your content appears in a dataset that was assembled by scraping, a robots.txt rule written today will not remove it.
- It will not stop impersonators. Anyone can send a request with any user-agent string. A scraper claiming to be MistralAI-User is not Mistral, and your robots.txt rule will not slow it down. IP verification is how you tell the difference.
- It will not control other companies' crawlers. Blocking Mistral does nothing about OpenAI, Anthropic, Google, or Common Crawl. Each operator has its own tokens and its own documentation. If you want a full AI-crawler policy, you need a rule for each operator you care about.
- It will not hide your site from the public web. Anything publicly accessible can be fetched by a browser, copied by a reader, or archived by a third party. Robots.txt tells well-behaved crawlers what you prefer. It does not make content private. If content must stay private, put it behind authentication.
Keeping Your AI Crawler Policy Up to Date
AI crawler tokens change. Operators add new crawlers, rename products, and occasionally retire tokens. Mistral's own documentation is the authoritative source for its tokens, and it is worth re-checking once or twice a year. When you review, check three things: whether the token list has changed, whether the stated purpose of each crawler has changed, and whether your robots.txt still matches your current policy. A policy written in 2026 may not be the policy you want in 2027.
It also helps to keep your AI crawler rules in one place. If your robots.txt scatters AI rules across several groups mixed in with search-engine rules, it becomes hard to audit. Group the AI crawler rules together with a clear comment, and keep a short note in your own documentation about why each rule exists. Future you will be grateful.
If you also want to help AI systems understand your site's structure, consider adding an llms.txt file alongside robots.txt. The two files do different jobs: robots.txt says what crawlers may fetch, while llms.txt gives AI systems a curated map of your most important pages. Used together, they give you both control and visibility. You can read more about how AI crawlers work in general on the CrawlGuards blog.
Frequently Asked Questions
What is the robots.txt token for Mistral AI?
Mistral documents three tokens: MistralAI-User for user-triggered fetches in Vibe, MistralAI-Index for automated search indexing, and MistralAI-Training for model training. The token most people ask about is MistralAI-User, and the block is two lines: User-agent: MistralAI-User followed by Disallow: /.
Does blocking MistralAI-User stop Mistral from training on my content?
No. Mistral states that MistralAI-User is not used for training. If your concern is training, block MistralAI-Training. Blocking MistralAI-User only stops live fetches for answering user questions.
Will blocking Mistral hurt my SEO?
Blocking Mistral has no effect on Google Search or Bing. Mistral's crawlers are separate from search-engine crawlers, and Googlebot does not read Mistral's rules. The only visibility you lose is inside Mistral's own products, such as citations in Vibe answers.
How long does it take for a robots.txt block to work?
Compliant crawlers fetch robots.txt before crawling and typically refresh their cached copy within about 24 hours. Expect the block to take effect within a day or two. If you still see page fetches after a week, check your syntax with a tester and verify the requests are really coming from Mistral's published IP addresses.
Can I block Mistral from just one section of my site?
Yes. Use path rules instead of blocking the whole site. For example, Disallow: /members/ under the Mistral group blocks everything in that directory while leaving the rest of your site open to Mistral's crawlers.
Does robots.txt stop Mistral from showing my content if someone else links to it?
Not entirely. Robots.txt stops crawling, but it does not erase a URL from an index or prevent a user from pasting your link into an assistant. If a page is already indexed, blocking the crawler will not remove the listing. For removal from search results, use a noindex directive rather than a robots.txt block.
The Bottom Line
Blocking Mistral AI crawlers with robots.txt is simple: name the token, disallow the paths, test the file, and deploy it. The harder part is deciding which tokens to block. For most sites, the smart answer is to block MistralAI-Training and allow MistralAI-User and MistralAI-Index. That keeps your content out of training data while keeping your pages citable in Mistral's answers. Block all three only if you have decided that no presence in Mistral's products is the right call for your site.
Whatever you choose, write it down, test it, and check your logs a week later. A robots.txt rule you never verify is just a hope with a filename.
