Short answer: PerplexityBot is Perplexity's automatic search crawler — it builds the index that Perplexity answers cite and link to, it honors robots.txt, and blocking it removes your pages from Perplexity results. Perplexity-User is a different agent: it fetches a single page because a person asked Perplexity a question, and because a user requested the fetch, Perplexity says it generally ignores robots.txt. Most site owners who “block Perplexity” only block the first bot, which costs them citations, while the second bot keeps arriving. The two need two different controls.
Perplexity's two-agent design confuses site owners more than almost any other AI vendor's setup, because both agents show up in logs with similar names and both can carry your content into an answer. The difference is not technical trivia. It decides whether your site appears in Perplexity answers at all, whether blocking actually stops anything, and what you need beyond robots.txt if you want a real exit. This guide separates the two agents, explains what each one does with your content, and gives you a setup for each goal.
PerplexityBot: the crawler that decides whether you get cited
PerplexityBot is the automatic crawler. It visits sites on its own schedule, reads pages, and builds the index that Perplexity's answers retrieve from. When a Perplexity answer shows your page as a source with a link, that citation almost always traces back to PerplexityBot having crawled and indexed the page.
Two facts from Perplexity's own documentation matter most. First, PerplexityBot honors robots.txt: its help center states that it will not index the full or partial text content of any site that disallows it. Second, Perplexity says PerplexityBot is not used to crawl content for AI foundation models — Perplexity does not build foundation models, so your content is not used for pre-training. That makes blocking PerplexityBot a very different decision from blocking a training crawler such as GPTBot or ClaudeBot. With a training crawler, blocking costs you nothing in search visibility. With PerplexityBot, blocking costs you the citations themselves.
There is one more nuance worth knowing. Perplexity says that even for a blocked site, it may still index the domain, the headline, and a brief factual summary. Blocking removes your page text from the index, but it does not erase every trace of your site from Perplexity's awareness. If your goal is total invisibility, robots.txt alone will not deliver it.
Perplexity-User: the fetcher that acts for a person
Perplexity-User is not a crawler at all in the usual sense. It does not roam the web building an index. It visits a specific page because someone using Perplexity asked a question that the page can answer, or because a user action inside Perplexity triggered a fetch. Think of it less like a librarian shelving books and more like a reader sent to fetch one specific book.
That distinction is why Perplexity says Perplexity-User generally ignores robots.txt. The reasoning is the same one OpenAI gives for ChatGPT-User and Anthropic's counterpart: because a person requested the fetch, the agent is treated more like a browser acting on a user's behalf than like an autonomous crawler. A robots.txt rule naming Perplexity-User is therefore a statement of intent, not a reliable control. It may be respected in some cases, but you cannot count on it.
Perplexity-User is also not a training agent. Like PerplexityBot, it is not used to crawl content for foundation models. Its job is retrieval for an answer, and its output is a cited link in that answer. Blocking it does not protect you from training use, because Perplexity does not train foundation models on your content in the first place. Blocking it mainly stops individual pages from being fetched on demand — and even then, unreliably.
The two bots side by side
Here is the comparison that matters when you write rules or read logs:
- PerplexityBot: automatic, scheduled crawling; builds the search index; honors robots.txt; blocking it removes your pages from Perplexity results; not used for foundation-model training; verified through Perplexity's published IP ranges.
- Perplexity-User: triggered by a user's question; fetches one page at a time; generally ignores robots.txt; blocking it is unreliable through robots.txt; not used for foundation-model training; verified through its own published IP ranges.
The same pattern repeats across vendors. OpenAI splits GPTBot (training) from OAI-SearchBot (search) and ChatGPT-User (user fetch). Anthropic splits ClaudeBot (training) from Claude-SearchBot (search) and Claude-User (user fetch). In every case, the search crawler is the one that controls citations, and the user fetcher is the one that robots.txt cannot reliably stop. Learning this pattern once saves you from re-learning it for every vendor.
What most site owners get wrong
Blocking everything with one rule. A site owner reads that Perplexity is “an AI company,” adds a block for PerplexityBot, and assumes they have opted out of AI use of their content. What they have actually done is remove their pages from Perplexity answers — a traffic source — while gaining no training protection, because Perplexity does not train foundation models on crawled content. If training opt-out is your goal, your effort belongs on GPTBot, ClaudeBot, CCBot, and the other training crawlers, not on PerplexityBot.
Assuming a block stops all Perplexity traffic. After blocking PerplexityBot, the owner still sees Perplexity-User in their logs and concludes the block failed. It did not fail; it was never aimed at that agent. Perplexity-User arrives because users keep asking questions your pages can answer. If you want those fetches to stop, you need a firewall rule, not a robots.txt edit.
Blocking PerplexityBot and expecting instant disappearance. Perplexity says robots.txt changes can take up to 24 hours to be reflected in its systems. A page that was already indexed may keep appearing in answers for a day after the block. Test after 24 hours, not after 24 minutes.
Trusting the user-agent string alone. Anyone can send a request claiming to be PerplexityBot. Scrapers borrow well-known bot names precisely because sites welcome them. Perplexity publishes its bot IP ranges as JSON files, one for each agent, and those ranges are the reliable check. A request with the right name but the wrong IP address is not Perplexity.
The setup for each goal
If you want Perplexity citations: allow PerplexityBot explicitly, and make sure no broader rule blocks it by accident. A wildcard block or a security plugin rule that refuses “AI bots” as a group can catch PerplexityBot along with the training crawlers. Check your file for any rule that names PerplexityBot with Disallow, and check your firewall for a rule that blocks its IP ranges. Perplexity also notes that where a firewall sits in front of your site, you may need to explicitly allowlist its bots. Then write pages that answer questions directly: a clear answer in the first sentence under each heading, specific facts with named sources, and clean structure that an answer engine can quote.
User-agent: PerplexityBot Allow: /
If you do not want Perplexity citations: disallow PerplexityBot in robots.txt, and accept the limits. Your page text leaves the index, though the domain, headline, and a brief factual summary may remain. Wait at least 24 hours before judging the result. And understand that this choice also removes a referral channel — Perplexity answers link their sources, and those links send readers.
User-agent: PerplexityBot Disallow: /
If you want to stop Perplexity-User fetches: robots.txt will not do it reliably. The control that works is a firewall rule that matches the Perplexity-User agent and its published IP ranges. That requires a firewall you control — for example, Cloudflare in front of your site — and it needs care, because an over-broad rule can block legitimate visitors or other Perplexity traffic you want. Test the rule, watch your firewall logs for a few days, and keep the rule scoped to the paths that actually need it rather than the whole site.
If a page must never reach an AI answer: do not rely on either bot control. Put the page behind a login or password protection. A page that requires authentication cannot be fetched by Perplexity-User for a stranger's question, and it cannot be indexed by PerplexityBot. Access control is the only control that does not depend on a bot's good behavior.
How to verify what is actually visiting your site
Start with your server or CDN logs and search for the two tokens separately. You will often find that PerplexityBot visits in regular crawls while Perplexity-User appears in bursts tied to user questions. Counting them together hides the pattern and leads to wrong conclusions about whether a block worked.
Next, verify the IPs. Fetch Perplexity's published JSON files for each agent and compare the source addresses in your logs against them. This step separates real Perplexity traffic from scrapers wearing its name, and it is the step most site owners skip. A scraper calling itself PerplexityBot is still a scraper, and it should be handled by your firewall and security rules, not by your Perplexity policy.
Finally, check the outcome, not just the logs. Ask Perplexity a question your page answers and see whether your page appears as a source. If you blocked PerplexityBot and waited 24 hours, it should not. If you allowed it, it should be eligible — though eligibility is not a guarantee of citation, since Perplexity ranks sources by relevance, freshness, and diversity, and it generally avoids citing the same domain twice in one answer.
FAQ
Does blocking PerplexityBot stop Perplexity from using my content for AI training?
That is the wrong question for Perplexity. Perplexity says it does not build foundation models, so its bots are not used to crawl content for pre-training. Blocking PerplexityBot mainly removes your pages from Perplexity answers. For training opt-out, block GPTBot, ClaudeBot, CCBot, and the other training crawlers.
Why do I still see Perplexity-User after blocking PerplexityBot?
Because they are different agents. PerplexityBot is the automatic indexer; Perplexity-User fetches a page when a person asks a question. A robots.txt block on PerplexityBot does not cover Perplexity-User, which generally ignores robots.txt because a user requested the fetch.
Can robots.txt block Perplexity-User?
Not reliably. Perplexity says Perplexity-User generally ignores robots.txt rules. To stop those fetches, use a firewall rule matched to its user agent and published IP ranges, or put the page behind authentication.
How long does a Perplexity robots.txt change take?
Perplexity says changes can take up to 24 hours to be reflected in its systems. Wait at least a day before judging whether a block or an allow worked.
Should most sites allow or block PerplexityBot?
Most content sites should allow it. Perplexity answers link their sources, so an allowed PerplexityBot is a referral channel, and blocking it buys no training protection since Perplexity does not train foundation models on crawled content. Block it only if you specifically do not want Perplexity citations.
How do I know a visit really came from Perplexity?
Compare the source IP against Perplexity's published IP ranges for that agent. The user-agent string alone can be faked by anyone, so the IP check is the reliable verification.
Build your file with the free robots.txt generator and verify it with the robots.txt tester. For comparison, see PerplexityBot: block or allow and how to get cited by ChatGPT. OpenAI documents its own search and fetch agents at developers.openai.com/api/docs/bots.
