AI crawlers
The AI crawlers, one page each
Five robots, three different jobs, and one distinction that decides whether your robots.txt is doing anything useful at all. Every fact on these pages is quoted from the operator’s own documentation and links to it.
| Robot | Operator | What it is for | Blocking it costs you |
|---|---|---|---|
| ClaudeBot | Anthropic | Training crawler | Nothing measurable in the answers buyers get |
| GPTBot | OpenAI | Training crawler | Nothing measurable in the answers buyers get |
| OAI-SearchBot | OpenAI | Search crawler | Eligibility to appear in that assistant’s search |
| Google-Extended | Usage control, not a crawler | Grounding eligibility — but not Google Search | |
| PerplexityBot | Perplexity | Search crawler | Inclusion in that engine’s results |
the expensive mistake
Blocking training and blocking search are one line apart
The operators separate their robots by job. Training crawlers collect content for model building. Search crawlers build the index an assistant searches, and blocking those removes you from what users see. User-triggered fetchers arrive because somebody is asking about you right now, and several operators document that those do not follow robots.txt at all.
Which means a single blanket rule does the one thing almost nobody intends:
The blanket rule that removes you from assistants
User-agent: *
Disallow: /The shape most owners actually want: no training, keep search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Copy it if it matches your intent, and read the per-robot page before you keep it. We are not going to tell you which side of the training question to be on — that is a judgement about your content. The search side is different: blocking it has a documented cost.
measured
Why robots.txt is the wrong lever for the thing you care about
Our own server logs show these crawlers requesting robots.txt before anything else, every time. In controlled runs across two client sites with six models each, not one model with a fetch tool requested it once (Note 04). The population that reads your directives is indexing you. The population answering a buyer’s question has never read them.
That second population is the one with the money in it, and the only thing that reaches it is what is in the bytes of the page it fetched. In sixty-four cold visits across twelve real company sites, 89% could not state a price and 57 of 64 read one page and stopped (Note 27). With a fact block in place on a controlled test, 78% of fact checks were recovered against 30% in the control (Note 41).
Permission and readiness are separate levers, and only one of them lives in this file. Google’s page-quality tooling now audits websites for agent-readiness; the package passes Google’s agent-readiness audit, which is a statement about what your site publishes rather than about who you allow. We make no claim that it improves your search position.
Sources: MARKETING_NOTES_AGENT_VISIBILITY.md, Notes 04, 27, 41.
Longer form, with every check Google’s tooling runs: the Agentic Browsing audit, explained.
free audit
See what AI sees on your site
Your robots.txt decides who may read you. Your homepage decides what they find. We measure the second one.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.