AI crawler · Perplexity · AI search index
PerplexityBot: what it is, its user agent, and how to block it
PerplexityBot is Perplexity's AI search crawler. Builds Perplexity's search index so pages can be shown and linked in its answers. Not used to train foundation models. In our 13 October 2026 study, 242 of 2,332 live top sites with readable robots.txt rules (10.4%) disallow PerplexityBot. Part of the AI crawler list.
What PerplexityBot does
Operator: Perplexity. Job: AI search index. Obeys robots.txt.
“To ensure your site appears in search results, we recommend allowing `PerplexityBot` in your site's `robots.txt` file and permitting requests from our published IP ranges listed below.”
Perplexity's PerplexityBot documentationPublished IP ranges (JSON)
PerplexityBot user agent
robots.txt token: PerplexityBot (matched case-insensitively).
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)robots.txt: allow or block PerplexityBot
Allow
User-agent: PerplexityBot
Allow: /Block
User-agent: PerplexityBot
Disallow: /Affects AI answers and citations. Blocking it keeps your pages out of the index Perplexity's AI search answers cite. It is not a training crawler.
Once a group names PerplexityBot, it ignores your User-agent: * rules, so copy any paths you still want blocked into it.
How many top sites block PerplexityBot
Disallow PerplexityBot in robots.txt
10.4%
242 of 2,332 sites
Allowed by robots.txt, refused at the firewall
8.5%
221 of 2,586 sites
In our 13 October 2026 study, 242 of 2,332 live top sites with readable robots.txt rules (10.4%) disallow PerplexityBot. Of the 2,586 listed sites whose robots.txt allowed PerplexityBot and that answered a browser cleanly, 221 (8.5%) refused our PerplexityBot request at the firewall with an error or challenge page.
Sample: Tranco top 5,000, tested once on 13 October 2026 from one Indian fixed-line connection. The robots.txt share counts live sites (home page 2xx) whose rules we could read, as on the AI visibility hub. Firewall tests came from our server, not the crawler's published IP ranges, so a refusal can also be an edge that verifies crawlers by IP. Method and CSV in the study.
Check your own site
The AI crawler checker reads your robots.txt the way PerplexityBot does and tests what your firewall returns. To write the rules, use the AI robots.txt generator.
Check your page: free, about 20 seconds, no sign-in
What it checks
- robots.txt for PerplexityBot
- Finds the group PerplexityBot obeys under RFC 9309 and the rule that decides your page.
- What your CDN lets through
- Requests the page as OAI-SearchBot, PerplexityBot and Claude-SearchBot and compares the answers with a browser request.
- Content without JavaScript
- AI crawlers don't run JavaScript. The audit checks how much of the page is in the raw HTML.
Free to run. Pay only for the fixes you want.
Free
Free account
Your score without signing in; with a free account, every check's result and the full fix for the top 1 issues.
★ Starter
$9 / ₹499 once
Every issue with its exact fix for one site, PDF/Markdown/JSON exports, re-audit for 30 days. Nothing renews.
Pro
$29 / ₹1,499 / month
Full reports on every site, weekly alerts when AI access breaks, competitor compare.
Agencies: white-label widget, branded reports and lead capture on Agency.
Questions
What is PerplexityBot?+
PerplexityBot is Perplexity's AI search crawler. Builds Perplexity's search index so pages can be shown and linked in its answers. Not used to train foundation models.
What is PerplexityBot's user agent?+
Perplexity documents it as: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). Match on the token PerplexityBot, since version numbers change.
How do I block PerplexityBot?+
Add "User-agent: PerplexityBot" followed by "Disallow: /" to your robots.txt. Obeys robots.txt, per Perplexity.
Does blocking PerplexityBot affect AI search citations?+
Blocking it keeps your pages out of the index Perplexity's AI search answers cite. It is not a training crawler.
How many websites block PerplexityBot?+
In our 13 October 2026 study, 242 of 2,332 live top sites with readable robots.txt rules (10.4%) disallow PerplexityBot. Of the 2,586 listed sites whose robots.txt allowed PerplexityBot and that answered a browser cleanly, 221 (8.5%) refused our PerplexityBot request at the firewall with an error or challenge page.