DeepSeekBot is the web crawler DeepSeek uses to collect public web pages that may feed its AI models. You can add a robots.txt stanza for it today, but you should know the honest truth first: DeepSeek has published no crawler documentation, no official user-agent string, no IP range, and no reverse-DNS pattern. The token DeepSeekBot appears in public bot registries, and adding it to robots.txt is worth doing, but it is not a reliable block on its own. If you want DeepSeek's crawlers kept out with confidence, you need a firewall-level rule in front of robots.txt. This guide shows you both layers, what each one can and cannot do, and how to check whether any of it worked.
What DeepSeekBot is — and what DeepSeek has not said about it
DeepSeek is the Chinese AI lab behind the DeepSeek-V2, DeepSeek-Coder and DeepSeek-R1 model families. Its models are open-weight, which means anyone can download and run them, and that makes its training-data appetite easy to underestimate. The company needs web-scale text to train future models, and like every other frontier lab it collects that text with a web crawler.
That crawler is known in public bot registries as DeepSeekBot. Here is where the certainty ends. Unlike OpenAI, which publishes a full page for GPTBot with its user-agent string, IP ranges and reverse-DNS verification, DeepSeek has published nothing comparable. There is no crawler page on DeepSeek's site, no documented user-agent string, and no published list of IP addresses you can verify against.
That gap matters more than it sounds. Robots.txt works by matching a user-agent token. If the operator never tells you the token, a robots.txt rule has nothing reliable to match. Public registries such as robotstxt.com's AI crawler list list DeepSeekBot as the token, and several independent bot directories report the same value, so it is the best available target — but it is reported, not documented. Treat it as a reasonable request, not a verified control.
There is a second wrinkle. Independent testing in late 2026 found DeepSeek reading pages through a full browser carrying an ordinary Firefox user-agent string, with nothing in the request that named DeepSeek at all. Other reports describe DeepSeekBot requests that do carry the DeepSeekBot token. Both can be true: a lab can run more than one collection system, and the one that identifies itself is not necessarily the one that collects the most. The practical consequence is simple: a robots.txt stanza may catch one kind of DeepSeek crawl and miss another entirely.
Why site owners want to block DeepSeekBot
The reasons are the same ones that drive GPTBot and ClaudeBot blocks, with one extra twist.
Training without a return visit. DeepSeekBot collects pages that may be used to train models. Unlike a search crawler, it does not send you traffic in exchange. Your content goes in; nothing measurable comes back. For a small publisher, that is a one-sided deal.
No citation path. DeepSeek does not document a separate search crawler that powers citations in its answers, the way OpenAI documents OAI-SearchBot or Anthropic documents Claude-Web. Blocking the training crawler therefore costs you no documented visibility, because there is no documented visibility to lose. That is a meaningful difference from GPTBot, where blocking training and blocking ChatGPT citations are separate decisions.
Open-weight distribution. DeepSeek's models are downloadable. Content absorbed into an open-weight model can be redistributed by anyone running that model, anywhere, with no further relationship to DeepSeek. Some site owners are comfortable with a hosted model learning from their work and uncomfortable with that same knowledge being packaged into a file anyone can copy. That is a legitimate distinction, and it is yours to make.
Bandwidth and log noise. Training crawlers are heavy. They fetch aggressively, revisit often, and rarely respect crawl-delay. On a small site or a metered host, that traffic costs real money and pollutes analytics.
None of this means blocking is right for everyone. If your business depends on being known inside AI tools, and DeepSeek's assistant becomes a meaningful discovery channel, a block trades future visibility for present control. For most independent publishers, blogs and documentation sites, the trade favours blocking — but it is a trade, and you should make it deliberately.
The robots.txt rule for DeepSeekBot
Add this stanza to the robots.txt file at the root of your domain. It is the same two-line pattern used for every named crawler:
# Block DeepSeekBot (DeepSeek training crawler)
User-agent: DeepSeekBot
Disallow: /Tokens in robots.txt are matched case-insensitively, so deepseekbot and DeepSeekBot are the same rule. Place the stanza above your catch-all User-agent: * group if you have one, and keep it separate from the groups for crawlers you want to allow. A common mistake is to put a training block inside a group that also allows Googlebot — grouping matters, because a crawler follows the most specific group that names it.
If you want to block DeepSeekBot from most of the site but keep a press kit or a public landing page open, use a path exception:
User-agent: DeepSeekBot
Disallow: /
Allow: /press-kit/Test the result before you call it done. Google's robots.txt specification documents how token matching and grouping work if you want to check the rules yourself. CrawlGuards' free robots.txt tester checks whether a given user-agent and URL combination is allowed or blocked by your live file, and it catches the grouping mistakes that silently void a rule. If you are building the file from scratch, the robots.txt generator produces a clean file with the AI-crawler presets already filled in.
One warning about expectations: because DeepSeek does not document its crawler, you cannot verify from their side that the rule was read. With GPTBot you can watch your logs for the token and confirm the crawl rate drops. With DeepSeekBot, a drop in requests carrying that token tells you the identified crawler stopped — it tells you nothing about the unidentified one.
Why robots.txt alone is not enough for DeepSeek
Three facts from the research above combine into a hard limit.
First, the token is undocumented. If DeepSeek changes it, retires it, or never used it in the first place, your rule matches nothing.
Second, public registries report that DeepSeekBot does not respect robots.txt. That claim is second-hand — it comes from bot directories, not from DeepSeek — but it is consistent with the absence of any documentation. A company that wanted its opt-out respected would document the opt-out.
Third, at least some DeepSeek collection appears to happen through ordinary browser user-agents that no robots.txt rule can name. A rule cannot match a token that is never sent.
None of this makes the robots.txt stanza pointless. It costs two lines, it is the standard mechanism every other operator honours, and it creates a clear record of your preference — which matters for the legal side of the question, covered below. It just cannot be your only layer. The reliable control sits one level up, at the firewall.
The reliable control: block at the firewall or CDN
If your site sits behind Cloudflare, a WAF rule that matches the DeepSeekBot user-agent string will stop the identified crawler before it reaches your server. The setup is a few clicks: create a custom rule, set the condition to match requests where the user-agent contains "DeepSeekBot", and set the action to Block. This works even if the crawler ignores robots.txt, because the request never reaches your origin.
Cloudflare also offers a managed "Block AI Bots" toggle that covers a list of known AI crawlers. It is a good baseline, but a named rule for DeepSeekBot is worth adding explicitly, because managed lists change and a named rule is visible in your own configuration. If you use a different CDN or a server firewall, the same logic applies: match the user-agent substring at the edge and refuse the request there.
For the unidentified crawler — the one that arrives looking like a Firefox browser — there is no clean user-agent rule. Your options are narrower and each has a cost. Rate-limiting aggressive anonymous crawlers by request rate catches heavy fetchers without naming them. A JavaScript challenge or bot-management score from your CDN separates browsers driven by humans from browsers driven by scripts, but it can also challenge real users on unusual networks. IP-range blocking is the bluntest option and the riskiest: without a published range from DeepSeek, you would be guessing, and guessing wrong blocks real visitors. For most sites, the honest answer is that the identified crawler can be stopped cold, and the unidentified one can only be made expensive and slow — which, in practice, is usually enough to push a collector elsewhere.
Whatever you choose, keep the two layers consistent. A robots.txt stanza that says "stay out" and a firewall that waves the same bot through sends a mixed signal. A firewall that blocks a bot your robots.txt explicitly allows is also a contradiction, though a harmless one. Write the preference once, in robots.txt, and enforce it at the edge.
How to check whether the block is working
Verification is harder here than with a documented crawler, but it is not guesswork.
Watch your logs for the token. Search your server logs for requests carrying "DeepSeekBot" in the user-agent. Before the block, note the volume and the paths requested. After the block, that volume should fall to zero for the identified crawler. Our earlier guide to checking whether GPTBot crawled your site uses the same log-search technique, and it applies here with the token swapped.
Watch your firewall's blocked-request count. If you added a WAF rule, your CDN dashboard will show how many requests it stopped. A rule that never fires either means the crawler never visited or the rule is not matching — check both by comparing against the log volume before the change.
Expect a delay. Robots.txt changes are picked up on each crawler's next fetch, which can take hours to days. Firewall rules apply immediately to new requests, which is another reason the edge layer is the reliable one.
Do not expect a confirmation from DeepSeek. There is no Search Console equivalent, no crawler verification tool, and no documented support channel for crawler questions. Silence is the normal state. Your logs are the only confirmation you will get.
Does blocking DeepSeekBot hurt your SEO?
No. DeepSeekBot is a training crawler, not a search crawler. It has no documented relationship with Google, Bing or any search index, and blocking it does not change how your pages rank. The same logic applies to every training-only crawler: search engines do not punish you for declining to feed a model.
The only SEO-adjacent risk is a self-inflicted one. If you copy a block list that also names a search crawler — Googlebot, Bingbot, OAI-SearchBot or PerplexityBot — you can remove yourself from the answers those services show. That is why the stanza above names DeepSeekBot and nothing else, and why you should check any list you paste for names you did not intend to block. Our allowlist strategy guide walks through the split: block the trainers, keep the answer engines open.
There is one more consideration worth naming. DeepSeek's models are widely used as the base for other tools, including search products that cite sources. If a future DeepSeek-powered search product emerges with its own crawler, that crawler will have its own name and its own robots.txt token, and the decision will need revisiting. A block today is not a block forever; review your AI-crawler rules twice a year, the same way you review everything else that touches your traffic.
The legal side: a robots.txt block is a record of your preference
Robots.txt is a request, not a lock. That sentence appears in nearly every guide on this site, and it is true in the technical sense: a crawler can ignore the file. But the file is also a public, timestamped statement of what you permit. That statement is gaining legal weight.
The EU's text-and-data-mining rules let rights holders reserve their works from mining by machine-readable means, and robots.txt is the machine-readable means courts and regulators recognise. The shared AI-training opt-out token agreed by Google, Microsoft, OpenAI and Anthropic in October 2026 — __usage_ai-training — is built on the same idea: one robots.txt entry that states a training preference across companies. Our guide to that shared opt-out explains how to add it alongside per-crawler rules. The DeepSeek stanza and the shared token are complementary: one names the crawler, the other states the preference in the new shared language.
In the United States, the copyright picture is still being argued in court, and no robots.txt rule decides a lawsuit on its own. But a clear, public opt-out is stronger than silence in any forum where consent is the question. Adding the stanza costs nothing and creates the record. That alone justifies it, even if the crawler never reads it.
A complete DeepSeekBot block, in order
- Add the robots.txt stanza. Two lines:
User-agent: DeepSeekBot,Disallow: /. Keep it separate from your allow groups. - Add the shared training opt-out. The
__usage_ai-traininggroup covers the four companies that support it; it does not cover DeepSeek, which is why step one still matters. - Add a firewall rule at the edge. Match the DeepSeekBot user-agent substring and block it before it reaches your server. This is the layer that actually enforces the preference.
- Test the file. Run your live robots.txt through a tester and confirm DeepSeekBot is blocked while Googlebot and OAI-SearchBot remain allowed.
- Check your logs a week later. Requests carrying the DeepSeekBot token should be gone, and your firewall should show the blocked attempts.
- Review twice a year. DeepSeek may document its crawler tomorrow, launch a search product, or change its token. A two-minute review keeps your rules honest.
Frequently asked questions
What is DeepSeekBot?
DeepSeekBot is the web crawler associated with DeepSeek, the AI lab behind the DeepSeek family of open-weight models. Public bot registries describe it as a training crawler that collects public web pages which may be used to train future models. DeepSeek itself has published no documentation for it.
Does DeepSeekBot respect robots.txt?
There is no official statement either way, because DeepSeek has published no crawler documentation. Public registries report that it does not respect robots.txt, and independent testing has observed DeepSeek collection through ordinary browser user-agents that no robots.txt rule can name. Add the stanza — it is the standard request — but enforce the block at your firewall if it matters to you.
Will blocking DeepSeekBot remove my site from DeepSeek's answers?
There is no documented DeepSeek search crawler or citation system to be removed from. Blocking the training crawler affects training data collection, not a search index. If DeepSeek launches a search product with its own crawler, that will be a separate name and a separate decision.
What user-agent string should I block?
Match the substring DeepSeekBot in the user-agent header, both in robots.txt and in any firewall rule. Tokens are matched case-insensitively in robots.txt. Do not block the bare substring "deepseek" on its own — it is too broad and risks catching unrelated traffic.
Can I verify that DeepSeekBot is really DeepSeek?
No. DeepSeek publishes no IP range and no reverse-DNS pattern for its crawler, so there is no way to confirm that a request carrying the DeepSeekBot token genuinely comes from DeepSeek. Anyone can send that user-agent string. The reverse is also true: a genuine DeepSeek request may carry no identifying token at all.
Is the shared __usage_ai-training opt-out enough on its own?
No, not for DeepSeek. The shared token is supported by Google, Microsoft, OpenAI and Anthropic. DeepSeek is not among the companies that announced support for it. Use both: the shared token for the companies that honour it, and the named DeepSeekBot stanza for DeepSeek.
The honest summary: add the two-line robots.txt stanza, because it is free and it records your preference — then enforce it at the firewall, because that is the layer DeepSeek cannot ignore. Check your logs a week later. That is the whole job.
