Quick answer: Bytespider is ByteDance's AI training crawler — the bot that collects web content for the models behind TikTok and ByteDance's other AI products. Block it with User-agent: Bytespider and Disallow: / in robots.txt. Because Bytespider has a documented history of ignoring robots.txt rules, serious sites should pair that with server-level blocks (.htaccess, nginx, or a firewall). Here's the complete playbook.
What is Bytespider, exactly?
Bytespider is the web crawler operated by ByteDance, the company behind TikTok. Its job is collecting web content to train AI models across ByteDance's portfolio. It declares itself with a user-agent string in the Bytespider family and references ByteDance's crawler documentation.
Technically, it's an ordinary HTTP crawler: it makes GET requests, reads page content, and doesn't execute JavaScript like a real browser. What sets it apart from crawlers like GPTBot or ClaudeBot isn't the technology — it's the compliance history. In 2023, security researchers and site owners documented instances of Bytespider crawling pages despite robots.txt disallow rules telling it not to. ByteDance has since updated its crawler practices, but the incident established a different baseline of trust than the US-based labs enjoy.
That history is why Bytespider deserves its own guide rather than a line in a generic blocklist: for most AI crawlers, robots.txt is a complete solution. For Bytespider, it's step one of two.
Step 1: Block Bytespider in robots.txt
Always start here. The declaration matters even for a crawler with a spotty compliance record — it establishes your stated preference, and many Bytespider requests do comply:
# Block ByteDance's AI training crawler
User-agent: Bytespider
Disallow: /If you want to be thorough, also catch variant spellings some site owners report seeing:
User-agent: Bytespider
Disallow: /
User-agent: Bytedance
Disallow: /After adding the rule, verify it's live at yourdomain.com/robots.txt and run it through our free robots.txt tester. Don't have a robots.txt yet? Our robots.txt generator builds one with the Bytespider preset included.
Step 2: Enforce it at the server level
This is the step most guides skip and the one that matters most for Bytespider. robots.txt is a request; server-level rules are enforcement. Pick the method that matches your stack:
Apache (.htaccess)
### Block ByteDance bots ###
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytedance|Bytespider [NC]
RewriteRule .* - [F,L]This returns 403 Forbidden to any request whose user agent contains "Bytedance" or "Bytespider" (case-insensitive). Place it near the top of your .htaccess, before WordPress's own rewrite rules.
Nginx
# Inside your server block
if ($http_user_agent ~* "Bytespider|Bytedance") {
return 403;
}Cloudflare (or any WAF)
Create a firewall rule that blocks requests where the user agent contains Bytespider. WAF rules act before traffic reaches your server, so they're the cleanest option if you're already behind Cloudflare — and they let you add IP ranges later without touching server config.
A note on IP blocking: it's the strongest option, but ByteDance's crawler IPs change over time, so a pure IP blocklist needs maintenance. Start with user-agent rules; add IP ranges only if user-agent blocks prove insufficient.
Step 3: Verify Bytespider is actually gone
With server access, verification is direct — grep your access logs for the token and check the status codes:
grep -i 'bytespider' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -cYou want to see 403s (blocked at the server) or the requests disappearing entirely. If you still see 200s after adding robots.txt alone, that's your signal that the server-level rule was necessary — which, given Bytespider's history, it often is.
Also double-check you haven't broken anything else: the .htaccess pattern above is specific to the ByteDance tokens, so Googlebot, Bingbot, and legitimate visitors are unaffected. Test a few normal pages after deploying.
Should you block Bytespider? The honest tradeoffs
Unlike some AI crawlers, Bytespider feeds no public search product you benefit from. Blocking it costs you nothing in Google Search, Bing, ChatGPT answers, or Perplexity citations — it's a pure training crawler. The case for blocking is straightforward:
- Bandwidth. Bytespider is known as an aggressive crawler. On smaller servers, its crawl volume alone is a reason to block.
- Training opt-out. If you don't want your content in ByteDance's training datasets, robots.txt plus enforcement is the way to say so.
- Precedent. The 2023 compliance incidents mean "polite request" alone is a weaker guarantee here than with OpenAI or Anthropic crawlers.
The case against blocking is thin: if you actively want maximum distribution of your content into every AI system, including ByteDance's, leave it allowed. For most publishers, bloggers, and businesses, blocking is the right default.
How Bytespider fits into your full AI-crawler policy
Bytespider is one piece of a bigger decision. The consistent strategy that works across all the labs: block the training crawlers, allow the answer-engine crawlers. In one robots.txt that looks like this:
# Training crawlers: blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Answer engines: allowed (no rules needed, but explicit is fine)
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /This keeps you visible everywhere people actually search — Google, ChatGPT, Perplexity — while opting out of model training across the board. For the complete token list, see our complete list of AI crawlers and the copy-paste robots.txt template.
How to tell if Bytespider is hitting your site right now
Before blocking anything, it's worth checking whether Bytespider is actually a problem for you. The signs are distinctive:
- Volume. Bytespider is a heavy crawler. In access logs it often shows up as sustained bursts of requests — dozens or hundreds of pages in a short window — rather than the steady trickle of a well-behaved indexer.
- Timing. Look for crawl waves at odd hours, hitting deep archive pages and tag/category pages that human visitors rarely touch.
- The user-agent string. Grep your logs and look at what you're dealing with:
# How much Bytespider traffic are you getting?
grep -ci 'bytespider' /var/log/nginx/access.log
# When did it visit, and what did it fetch?
grep -i 'bytespider' /var/log/nginx/access.log | awk '{print $4, $7}' | head -20If the count is zero, Bytespider isn't currently interested in your site — the robots.txt rule is still worth adding as insurance, but there's no fire. If you're seeing hundreds of hits a day on a small site, the server-level block in Step 2 will pay for itself in bandwidth alone.
No server access (shared hosting, site builders)? Check whatever analytics you have for traffic spikes with near-zero engagement — high page counts, no conversions, odd hours. That's the crawler signature, and it's often the only visibility you'll get.
The middle path: rate-limit instead of blocking
Full blocking isn't the only option. Some site owners are fine with ByteDance's crawler existing but not with it hammering their server at 50 requests a second. If that's you, rate-limiting gives you the middle path — allow the crawl, cap the speed:
# nginx: slow Bytespider down instead of blocking it
limit_req_zone $binary_remote_addr zone=spider:10m rate=5r/m;
location / {
if ($http_user_agent ~* "Bytespider|Bytedance") {
limit_req zone=spider burst=10 nodelay;
}
}This caps matching requests to roughly 5 per minute — enough for any legitimate indexing purpose, nowhere near enough to strain a server. Cloudflare users can do the same with a rate-limiting rule scoped to the Bytespider user agent. The tradeoff is honest: you stay in ByteDance's datasets, but on your terms. For most publishers who've decided they don't want to be in those datasets at all, blocking remains the cleaner choice.
Bytespider vs the other aggressive crawlers
Bytespider isn't the only crawler with a reputation for heavy or impolite behavior. If you're building a general "bad bot" policy, these are the usual companions on the blocklist:
- CCBot (Common Crawl) — crawls the whole web to build a public dataset that many AI labs train on. Polite about robots.txt, but enormous in volume. Blocking it cuts off a major indirect path into training datasets.
- MJ12bot (Majestic) — an SEO backlink crawler, not an AI trainer, but famously aggressive. Included here because the server-level techniques are identical.
- Undeclared scrapers — the real problem child. These don't announce themselves at all, so no user-agent rule catches them. Rate limiting, WAF bot management, and behavioral detection are the only defenses — which is exactly why the enforcement layer in Step 2 matters beyond just Bytespider.
The pattern to remember: robots.txt handles the honest bots, server rules handle the heavy ones, and behavioral defenses handle the dishonest ones. Bytespider sits squarely in the middle category — which is why it gets two layers while GPTBot only needs one.
What ByteDance's own crawler documentation says
ByteDance publishes crawler documentation describing Bytespider's purpose and its declared user-agent strings — the company does identify its bot rather than hiding it. The docs position Bytespider as a standard web crawler and describe the usual robots.txt mechanism for opting out. That's the official story, and for many sites it holds.
The gap between the documentation and the 2023 incident reports is exactly why this guide has two steps instead of one. Official docs describe intent; your server logs describe behavior. When the two disagree, behavior wins — which is why the enforcement layer isn't paranoia, it's just engineering. Re-check your logs a week after deploying both layers: if the 403s stop appearing and no 200s return, the crawler has taken the hint. If 200s persist from IPs that resolve to ByteDance infrastructure, escalate to IP-range blocks at your firewall and consider reporting the pattern — persistent non-compliance after an explicit opt-out is worth documenting, if only for your own records.
One practical tip: keep your Bytespider rules in a clearly commented section of your robots.txt and .htaccess, separate from your general AI-crawler rules. Crawler behaviors change — tokens get renamed, new variants appear — and a labeled section makes the file maintainable a year from now when you've forgotten why that block exists.
Frequently asked questions
What is Bytespider?
Bytespider is ByteDance's AI training web crawler. It collects web content used to train AI models behind TikTok and ByteDance's other AI products. It identifies itself with a user-agent string in the Bytespider family.
Does Bytespider respect robots.txt?
It declares itself and generally responds to robots.txt, but it has a documented history — reported by researchers and site owners in 2023 — of ignoring disallow rules on some sites. Treat robots.txt as the starting point and add server-level enforcement if compliance matters to you.
Will blocking Bytespider hurt my SEO?
No. Bytespider is a pure training crawler with no connection to Google Search, Bing, or any answer engine's citations. Blocking it has zero effect on your rankings or your visibility in ChatGPT, Perplexity, or AI Overviews.
How do I block Bytespider in WordPress?
Two layers: add User-agent: Bytespider / Disallow: / to your robots.txt (via an SEO plugin like Rank Math or Yoast, or manually), and add the .htaccess user-agent block shown above for enforcement. Our WordPress AI-crawler guide covers the full setup.
Can I block Bytespider with Cloudflare?
Yes. Create a WAF firewall rule matching user agents containing "Bytespider" (and "Bytedance" for variants) and set the action to Block. This stops the traffic before it reaches your origin server and is the cleanest enforcement option.
What's the difference between Bytespider and TikTok's crawler?
They're the same operation — Bytespider is ByteDance's crawler, and ByteDance owns TikTok. The training data it collects feeds models across ByteDance's portfolio, including TikTok's AI features. One block covers both.
