Perplexity runs two different kinds of traffic. PerplexityBot is the search indexing crawler. Perplexity-User fetches a page in real time when a person asks Perplexity about it. A robots.txt rule for PerplexityBot does not necessarily cover the second one, because an independent analysis reports that Perplexity documents Perplexity-User as not adhering to robots.txt [2].

The two agents

Perplexity published updated instructions for site owners covering both agents, including exact user-agent strings, IP ranges, recommended robots.txt settings and AWS WAF rules. They point to the official guide at docs.perplexity.ai/guides/bots [1]. Treat that guide as the source of truth. It can change, and this article does not reproduce its strings or IP lists.

PerplexityBotPerplexity-User
RoleSearch indexing traffic [2]User-initiated traffic [2]
Triggered byPerplexity's own crawl scheduleA person asking Perplexity to fetch a page
robots.txtUsed as the control; the IPTC guidance gives PerplexityBot as its example token to block [4]Reported as documented not to adhere to robots.txt [2]

The practical consequence: PerplexityBot decides whether your pages can enter Perplexity's search index. Perplexity-User is closer to a browser acting for one person, one request at a time.

Why a page can be fetched when PerplexityBot is blocked

There are three separate reasons. Check which applies before changing anything.

  1. The fetch was user-initiated. If Perplexity-User made the request, a PerplexityBot rule was never the relevant control. Per the independent paper, Perplexity documents that this agent does not follow robots.txt [2].
  2. robots.txt is voluntary. It works only for crawlers that choose to read and honor it. The IPTC guidance frames it as an opt-out mechanism for AI crawlers, which means it is only as effective as the crawler's compliance [4].
  3. Your layers disagree. A page can be blocked at one layer and open at another: robots.txt, CDN or WAF rules, origin server config, and cached or mirrored copies. A Perplexity forum thread reports a site where allowlisting by http.user_agent containing PerplexityBot or Perplexity-User in AWS WAF did not solve an access problem [3]. User-agent matching alone is not a reliable control in either direction.

A fourth possibility is that the request is not Perplexity at all. User-agent strings are trivially spoofed, so a log line is not proof of origin. That is why the published IP ranges matter.

What robots.txt can and cannot do here

To opt out of Perplexity's indexing crawler, name its token:

User-agent: PerplexityBot
Disallow: /

To allow it while blocking other things, use a separate group:

User-agent: PerplexityBot
Allow: /
Disallow: /private/

Two limits apply to any robots.txt rule:

  • It governs crawling by compliant agents. It does not authenticate anything and does not stop a determined or non-compliant client.
  • It does not remove content that is already in an index or that has been copied elsewhere. Removal is a separate process.

Whether to add a rule for Perplexity-User is a judgment call. If the agent ignores robots.txt, as reported [2], the line does nothing. Adding it is harmless, but do not count on it.

Controlling user-initiated fetches

If you truly need to stop Perplexity-User requests, you need enforcement on your side instead of a request in robots.txt.

  • Use the published IP ranges. Perplexity's guide lists them [1]. Matching on IP is harder to fake than matching on user-agent. Pull the current list from the guide rather than copying one from a blog post, including this one.
  • Combine signals. Because user-agent allowlisting alone failed for at least one site [3], decide your rule from user-agent and source IP together, and test it.
  • Mind the trade-off. Blocking user-initiated fetches means a person who pastes your URL into Perplexity gets no content from it. That may be what you want for paywalled or private material. For public marketing pages it may work against you.
  • Use authentication for anything that must stay private. A login wall or signed URLs work regardless of any bot's manners.

Verify what is actually happening

  1. Fetch https://yourdomain.com/robots.txt and confirm the PerplexityBot group is the one you intended. Check for a wildcard User-agent: * group that may override your intent, and for CDN-injected robots.txt content.
  2. Search your server or CDN logs for PerplexityBot and Perplexity-User. Compare source IPs with the ranges in the official guide [1]. Requests that claim the name but come from other addresses are not Perplexity.
  3. Test your WAF rule by sending a request with the user-agent token, for example curl -I -A "PerplexityBot" https://yourdomain.com/, then compare with a plain request. Note that this tests your rule's user-agent logic only, not the IP condition.
  4. Re-test after any CDN or WAF change. Rules at different layers drift apart.

For the robots.txt half of this, our walkthrough on checking whether AI crawlers can reach your site covers what a block and an allow look like. The free scan at aaivisibility.com reports a robots.txt verdict for 22 AI crawlers from your live file. It reads robots.txt only, so it cannot see WAF rules or log traffic.

What is uncertain

  • This article relies on Perplexity's forum post [1] and an independent paper's reading of Perplexity's documentation [2]. Read the official guide yourself for current wording.
  • The notes here do not include Perplexity's actual IP ranges or full user-agent strings, so check the guide for both.
  • We have no evidence about how blocking any Perplexity agent changes whether your pages are cited. The mechanics above describe access, not outcomes.

What to do next

  • Decide separately for each agent: do you want to be indexed, and do you want user-requested fetches to work?
  • Put a PerplexityBot rule in robots.txt that matches that decision.
  • If you need to block Perplexity-User, enforce it at the WAF or CDN with the published IP ranges and test the rule.
  • Put anything truly private behind authentication.
  • Re-check logs after a week to confirm the traffic you see matches your rules.

For the wider picture of which AI crawlers exist and how their tokens differ, see the 18 checks behind the AI readiness score.