PerplexityBot

What PerplexityBot is

PerplexityBot is an automated crawler from Perplexity. A crawler is software that requests webpages. PerplexityBot helps Perplexity find public pages that may appear as linked sources in Perplexity search results.

Think of PerplexityBot as a library scout. The scout notes open books so readers can find them later. The scout does not decide which book every reader sees.

Perplexity gives PerplexityBot a narrow purpose. The crawler supports search-result discovery. Perplexity states that PerplexityBot does not collect webpage text for AI foundation model training.

What PerplexityBot does

PerplexityBot requests public webpages, stores page records, and helps Perplexity choose possible source links. Crawl access only makes retrieval possible; it does not promise that Perplexity will cite a page.

Use the three-part split:

• Fetch: PerplexityBot requests a webpage.
• Index: Perplexity stores page records for search use.
• Link: Perplexity may show a website as a source.

How PerplexityBot differs from Perplexity-User

PerplexityBot and Perplexity-User do different jobs. PerplexityBot handles automatic crawling for search results. Perplexity-User supports webpage fetches that come from user actions inside Perplexity.

Name	Job	robots.txt behavior
PerplexityBot	Automatic crawling	Perplexity states that it follows robots.txt
Perplexity-User	User-requested fetches	Perplexity states that it generally ignores robots.txt

The distinction changes site controls. A publisher may block PerplexityBot while still seeing Perplexity-User requests tied to user actions. Perplexity states that Perplexity-User generally ignores robots.txt because a user requested the fetch.

Which user-agent strings identify Perplexity crawlers

A user agent is the label a crawler sends with a webpage request. Perplexity publishes separate user-agent strings for PerplexityBot and Perplexity-User.

PerplexityBot:

Perplexity-User:

A copied user-agent string can fake the name. Server logs need a second check: compare the source IP address with the official Perplexity IP endpoints.

How robots.txt affects PerplexityBot

robots.txt tells crawlers which URL paths they may request. Perplexity states that PerplexityBot will not index full or partial text from a site that disallows it through robots.txt.

robots.txt works like a door sign for crawlers. A compliant crawler reads the sign before entering. The sign does not lock the room

Google Search Central warns that robots.txt does not hide pages from Google. Google recommends noindex or password protection when a publisher wants stronger removal or privacy control.

RFC 9309 makes the same boundary plain. Robots Exclusion Protocol rules are not access authorization.

How to block PerplexityBot with robots.txt

A publisher can disallow PerplexityBot by naming the crawler in robots.txt. The broad rule below asks PerplexityBot to avoid every URL path.

A narrower rule can block one folder.

Example: a publisher wants PerplexityBot to avoid /private-reports/. The publisher adds the second rule to the root robots.txt file. RFC 9309 defines robots.txt as a file made available by the service for crawler rules.

How to verify PerplexityBot requests

Verification needs two matches: the PerplexityBot user agent and an official Perplexity IP range. Perplexity tells WAF users to combine user-agent matching with IP address verification.

A usable log record should include:

• Date checked
• Requested URL
• User-agent string
• Source IP address
• Matching Perplexity IP endpoint
• Rule action, such as allow or block

Perplexity says its JSON endpoints are the source of truth for WAF configurations. Perplexity also recommends periodic updates because IP ranges can change.

What happens when a page blocks PerplexityBot

Perplexity states that a robots.txt block stops PerplexityBot from indexing full or partial page text. Perplexity also states that it may still index the domain, headline, and a brief factual summary.

This limit matters for removal work. A robots.txt block can reduce page-text crawling. It may not erase every mention of a URL or domain from Perplexity.

Google Search Central gives the same type of warning for Google Search. A URL blocked by robots.txt can still appear if other pages link to it.

What the Cloudflare dispute adds

Cloudflare reported on 4 August 2025 that Perplexity used undeclared crawling behavior after sites blocked declared Perplexity crawlers. Cloudflare said it observed changed user agents, changed ASNs, and IP addresses outside official Perplexity ranges.

That dispute should change monitoring, not the definition. Official Perplexity pages state declared crawler behavior. The Cloudflare report supports extra checks for user agent, IP range, ASN, and WAF events.

What publishers should check after a robots.txt change

A publisher should check robots.txt, server logs, and WAF events after changing PerplexityBot access. Perplexity says crawler-control changes may take up to 24 hours across its systems.

Use this checklist:

1. Confirm /robots.txt loads from the domain root.
2. Confirm the rule names PerplexityBot.
3. Check server logs after the rule change.
4. Match user-agent strings with official IP endpoints.
5. Review Perplexity-User separately from PerplexityBot.
6. Check WAF allow, challenge, or block events.
7. Record mismatches for later review.

The publisher choice is crawler access. robots.txt can ask PerplexityBot to avoid page text. Server rules and authentication protect material that must remain private.