PerplexityBot: Should You Block or Allow It? The Complete 2026 Guide

If you are deciding whether to block or allow PerplexityBot, here is the short, honest answer: for most websites, allow it. PerplexityBot is Perplexity's search crawler, and Perplexity states in its official crawler documentation that it is not used to train AI foundation models. Blocking it mainly removes your chance of appearing — and being cited with a link — in Perplexity answers. That said, "allow" is not right for everyone. If your content is gated, licensed, or expensive to produce, a blanket allow deserves a second look. The right policy depends on what you value more: referral traffic from an answer engine, or keeping content out of AI search entirely.

This guide explains exactly what PerplexityBot does, how it differs from Perplexity-User, what Perplexity's documentation actually promises, and gives you copy-paste robots.txt rules for both blocking and allowing it — so you can make the call with facts instead of fear.

What PerplexityBot actually is

PerplexityBot is Perplexity's automated web crawler. It visits websites, collects page content, and feeds it into Perplexity's search index — the system that lets Perplexity answer questions and show source links. According to Perplexity's own documentation, the bot exists to "surface and link websites in search results on Perplexity." The same documentation says plainly: "It is not used to crawl content for AI foundation models."

That sentence matters more than almost anything else in this article. Many publishers assume every AI company's crawler is collecting training data. In Perplexity's case, the company documents the opposite for this bot. There is no separate, declared Perplexity training crawler — unlike OpenAI (GPTBot) or Anthropic (ClaudeBot), which run dedicated training bots you can block independently. Perplexity's crawler setup is simpler: one bot for search indexing, one agent for on-demand user fetches. Neither is documented as a training crawler.

PerplexityBot vs Perplexity-User: the difference that trips everyone up

This is the part most site owners get wrong, and it is the single most important section of this guide. Perplexity operates two agents, and they behave very differently:

PerplexityBot — the search crawler

PerplexityBot crawls the web automatically to build Perplexity's search index. It honors robots.txt. If your robots.txt disallows it, Perplexity says the bot will not index the full or partial text of the disallowed pages. Perplexity also publishes its crawler IP ranges so you can verify that traffic claiming to be PerplexityBot really is — impostor bots sometimes borrow well-known user-agent strings to inherit a welcome they don't deserve.

Perplexity-User — the on-demand fetcher

Perplexity-User is not a crawler at all. It fetches a page when a person using Perplexity asks a question and Perplexity decides it needs that page to give an accurate answer and link to it. Because a human triggered the fetch, Perplexity's documentation says this fetcher "generally ignores robots.txt rules." It behaves more like a browser acting on someone's behalf than like a bot filling an index.

Why does this matter? Because blocking PerplexityBot in robots.txt does not fully remove your site from Perplexity's answers. A disallow rule stops the indexing crawl, but if a user asks about your site directly, Perplexity-User may still fetch the page. If your goal is a total exit from Perplexity, robots.txt alone will not get you there — you would need enforcement at the server or firewall level.

The exact user-agent strings (verified)

Getting these strings exactly right matters. A typo in robots.txt, or in a firewall rule, silently does nothing. Here are the two tokens exactly as Perplexity documents them:

In your robots.txt, you match on the token — the short name between ; and / in the full string. So User-agent: PerplexityBot is the correct directive. Two things to note:

• Perplexity documents only these two agents. There is no officially documented "Perplexity-User-verified" or any other variant at the time of writing. If you see advice referencing additional tokens, treat it as unverified until Perplexity's own docs confirm it.
• Perplexity publishes its current IP ranges as JSON files — one for PerplexityBot and one for Perplexity-User. Use these in your firewall rather than trusting the user-agent string alone, since any script on the internet can copy a user-agent string.

The case for allowing PerplexityBot

Allowing PerplexityBot is the default recommendation — Perplexity's own, and ours for most sites. The reasons are practical rather than philosophical:

• Discovery in a growing answer engine. Perplexity describes its crawler as the way sites get surfaced and linked in Perplexity search results. If you want people to find your pages through Perplexity's answers, the crawler has to be able to read them.
• Citations with links. Perplexity answers routinely include source links to the pages they draw on. Unlike some answer engines where attribution is buried or missing, Perplexity's format shows your site name and links to the page. That is direct referral traffic you would not otherwise get.
• No training-data concern, per Perplexity's documentation. Because Perplexity states this bot is not used to crawl content for AI foundation models, the most common reason people block AI crawlers — "I don't want my content training their models" — does not apply here the way it does to GPTBot or ClaudeBot. You are opting into search visibility, not model training.
• Cheap to reverse. Perplexity says crawler-setting changes may take up to 24 hours to propagate through its systems. That is a short experiment cycle. Allow it, watch your logs and referral traffic for a few weeks, and reverse the decision if the results disappoint.

The case for blocking PerplexityBot

Blocking is legitimate too, in the right situations. Honest reasons to disallow it:

• You want no presence in Perplexity at all. Some publishers have a policy of staying out of every AI answer engine, for branding or licensing reasons. That is a valid call — just remember that robots.txt only stops the indexing crawl, not Perplexity-User fetches.
• Server load or scrape abuse. In August 2025, Cloudflare published a report alleging that Perplexity accessed freshly created test domains whose robots.txt blocked its crawlers, using undisclosed user agents and rotating IP ranges. Perplexity disputed the report, framing the activity as user-initiated fetches rather than crawling. If you are skeptical of the stated policy, or you see unexplained fetch volume in your logs, a block plus firewall enforcement is the conservative route.
• Licensed or gated content. If you sell access to your content or license it to partners, letting it be summarized freely in answer engines can undercut your business model. Note that a robots.txt block is a public file — it even advertises which paths you care about. Truly sensitive content belongs behind authentication, not behind a disallow rule.

One warning that applies to everyone: robots.txt is a voluntary protocol. A Disallow line is a request, not a lock. If you need something off the internet's radar, use access control — passwords, tokens, IP allowlists — not robots.txt.

robots.txt rules: block, allow, or partial

Below are ready-to-use configurations for the three common policies. Test any of them with a robots.txt tester before deploying, and you can generate a PerplexityBot-ready robots.txt with your chosen rules in minutes.

Policy 1: Allow PerplexityBot (recommended for most sites)

If you have no Perplexity rules at all, PerplexityBot is allowed by default — robots.txt defaults to allow. But an explicit allow documents your intent and keeps it clear when you add other restrictions:

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Policy 2: Block PerplexityBot completely

To opt out of Perplexity search indexing, disallow the crawler. Including Perplexity-User here is honest about your intent, even though Perplexity's docs say that fetcher generally ignores robots.txt anyway:

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

If you actually want this block to mean something, back it with firewall rules that filter on Perplexity's published IP ranges. Robots.txt alone cannot stop the user-triggered fetches.

Policy 3: Allow search indexing, keep premium sections private

Many publishers want Perplexity visibility for their free content while keeping paid or members-only sections out of AI search. A targeted disallow does exactly that:

User-agent: PerplexityBot
Allow: /
Disallow: /premium/
Disallow: /members/
Disallow: /api/
Disallow: /checkout/

Adjust the paths to your real paywalled directories. Remember that Allow before Disallow is fine in robots.txt — the most specific matching rule wins per RFC 9309.

How to verify your block (or allow) is actually working

Don't trust the configuration — check the logs. Here is a practical verification routine:

1. Wait 24 hours after changing robots.txt. Perplexity says its systems may take up to a day to reflect crawler-setting changes.
2. Search your server logs for requests with the PerplexityBot and Perplexity-User tokens in the user-agent string. Look at which URLs they hit and what response codes they received.
3. Verify the source IP against Perplexity's published JSON IP lists. A request claiming to be PerplexityBot from an IP outside those ranges is an impostor — block it, and don't let it influence your policy decision.
4. Distinguish the two agents. After a full block of PerplexityBot, you should stop seeing the crawler in logs. If Perplexity-User requests continue, that is expected behavior per Perplexity's documentation — it means users are asking about your pages, not that your block failed.
5. Check for stealth fetching. If you blocked everything and still see suspicious fetch patterns — generic browser user agents hitting many pages from cloud IPs — that matches the pattern Cloudflare flagged in 2025. Investigate with rate limiting or WAF bot-management rules rather than another robots.txt line.

How Perplexity's setup compares to other AI crawlers

Quick context, because PerplexityBot often gets confused with training crawlers:

• OpenAI runs three agents: GPTBot (training — blockable), OAI-SearchBot (search — keep allowed if you want ChatGPT citations), and ChatGPT-User (on-demand fetch, may bypass robots.txt like Perplexity-User).
• Anthropic runs ClaudeBot (training), Claude-SearchBot (search), and Claude-User (on-demand fetch). Notably, Anthropic documents that even Claude-User honors robots.txt — a stricter posture than Perplexity-User.
• Perplexity runs just PerplexityBot (search indexing, honors robots.txt) and Perplexity-User (on-demand fetch, generally ignores robots.txt). There is no declared Perplexity training crawler, which simplifies the decision: with Perplexity, the choice is really about search visibility, not about training data.

The decision, distilled

Ask yourself one question: would I turn down free, cited referral traffic from an answer engine whose operator says it won't train models on my content?

• If the answer is no — you want every qualified visitor you can get — allow PerplexityBot. Make the allow explicit, keep an eye on your logs, and use an llms.txt file to guide answer engines toward your best content.
• If you run a licensing business, gated content, or a hard policy against AI answer engines — block both tokens in robots.txt and enforce it at the firewall, knowing that on-demand Perplexity-User fetches may still arrive when real users ask about your pages.

FAQ

Does PerplexityBot respect robots.txt?

Yes, according to Perplexity's own documentation. Perplexity states that PerplexityBot follows robots.txt and will not index the full or partial text of pages that disallow it. Perplexity's Help Center adds that a blocked page may still leave the domain, headline, and a brief factual summary discoverable — so blocking stops indexing, not every mention.

Is PerplexityBot used to train AI models?

Per Perplexity's documentation, no. The company states PerplexityBot "is not used to crawl content for AI foundation models," and says the same for Perplexity-User. There is no declared separate Perplexity training crawler. Independent verification of internal training pipelines isn't publicly available, so publishers who are skeptical can still block — but the stated policy is clear.

What is the difference between PerplexityBot and Perplexity-User?

PerplexityBot is an automated crawler that builds Perplexity's search index and honors robots.txt. Perplexity-User is an on-demand fetcher triggered when a person asks Perplexity a question; it may visit a specific page to answer and link it, and Perplexity says it generally ignores robots.txt because a user requested the fetch. You control the first with robots.txt; you can only control the second with firewall or server-level rules.

Will blocking PerplexityBot remove my site from Perplexity answers?

Not entirely. Blocking PerplexityBot stops the indexing crawl, which removes your pages from Perplexity's search index over time. But Perplexity-User may still fetch individual pages when users ask about them, since it generally ignores robots.txt. If you need a complete opt-out, enforce it at the WAF or server level using Perplexity's published IP ranges — and test afterward.

How long does a robots.txt change take to affect PerplexityBot?

Perplexity says it may take up to 24 hours for its systems to reflect crawler-setting changes. If you change your robots.txt today, don't judge the results until tomorrow's logs. Also verify the bot actually re-fetched your robots.txt — some operators cache it aggressively.

Should I block Perplexity-User too?

It depends on what you're trying to achieve. Adding User-agent: Perplexity-User / Disallow: / expresses your intent, but Perplexity's documentation says the fetcher generally ignores robots.txt, so treat that line as a signal rather than enforcement. If on-demand fetches are a real problem for you, use WAF rules based on Perplexity's published IP ranges instead.