AI crawlers › PerplexityBot
PerplexityBot: what it is, and how to allow or block it
PerplexityBot is documented as a search robot, not a training one: its operator says outright that it is not used to crawl content for AI foundation models.
Search crawler · operator: Perplexity · every fact below is quoted from Perplexity, “Bots” and links to it. Nothing on this page is inferred.
what the operator documents
The facts, and where each one comes from
| Fact | What the operator says | Source |
|---|---|---|
| Operator | Perplexity. | Perplexity docs |
| Name to match in robots.txt | PerplexityBot | Perplexity docs |
| Documented request string | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Perplexity docs |
| Documented purpose | “Designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” | Perplexity docs |
| Does it honour robots.txt | The operator documents that it respects robots.txt rules, and recommends allowing it if you want to be included in results. | Perplexity docs |
| The user-triggered sibling | Perplexity-User visits pages when a person asks a question. The operator documents that it generally ignores robots.txt rules, because the visit is triggered by a user rather than by a crawl, and that it is not used for training. | Perplexity docs |
| Published address list | https://www.perplexity.com/perplexitybot.json | Perplexity docs |
What the operator does not publish
- A way to keep the user-triggered robot out through robots.txt. The operator documents that it generally ignores those rules, so if a page must not be fetched on a user's behalf, robots.txt is the wrong instrument — authentication is the right one.
allow or block
The exact robots.txt lines
Paste these into the robots.txt file at the root of your site. Groups are matched by name, so a group for one robot has no effect on another.
Allow it explicitly
User-agent: PerplexityBot
Allow: /Block it
User-agent: PerplexityBot
Disallow: /Allow the search robot, keep one directory out
User-agent: PerplexityBot
Disallow: /checkout/
Allow: /Two names, two behaviours: the crawler follows your file and the user-triggered fetcher is documented as generally not following it. Anything that must stay private needs a login, not a directive.
what it means for you
What allowing or blocking it actually changes
Blocking it costs you
Inclusion in that engine’s results, which is what its own documentation says the robot is for.
The pattern we keep measuring
This robot type reads robots.txt first, every time, in our own server logs. The models answering a buyer’s question with a fetch tool never read it at all (Note 04). Optimising your robots.txt for the second group is advice aimed at the wrong population.
What actually changes the answer
Facts in the bytes of the page the fetcher lands on. That is the measured part: 78% recovery against a 30% control, with agents stopping after an average of 1.1 pages (Note 41).
And the lever that is not in this file at all
Permission is one job; being legible when the robot arrives is another. Google’s page-quality tooling now audits websites for agent-readiness, and the package passes Google’s agent-readiness audit — which is a claim about the surfaces on your site, not about who you let in. We make no claim that any of it improves your search position.
The distinction that decides everything on this page
| Crawlers like this one | A model with a fetch tool | |
|---|---|---|
| Reads robots.txt | Always, and first | Never, in our runs |
| What it is doing | Indexing, then leaving | Answering somebody’s question now |
| What reaches it | Your directives | The bytes of the page it fetched |
Our own server logs plus six models per site across two client sites: the crawlers requested robots.txt before anything else, every time, and not one fetch-tool model requested it once. Source: MARKETING_NOTES_AGENT_VISIBILITY.md, Note 04.
the rest of them
The other crawlers people ask us about
- ClaudeBot — Anthropic, training crawler
- GPTBot — OpenAI, training crawler
- OAI-SearchBot — OpenAI, search crawler
- Google-Extended — Google, usage control, not a crawler
Definitions for the vocabulary on this page: AI crawler, answer engine, llms.txt, fact block.
The separate question of whether your site is legible when one of these arrives is graded by Google’s own tooling now: our explainer on the Agentic Browsing audit covers every check it runs.
free audit
See what AI sees on your site
Blocking and allowing is housekeeping. The question underneath it is whether the page a robot reaches actually answers what your buyers ask.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.