Glossary › AI crawler
AI crawler — an automated client that fetches pages for an AI operator
An AI crawler is a robot that requests pages on behalf of an AI operator — for training, for a search index, or on behalf of a person who just asked a question.
The category is usually discussed as one thing and behaves as at least three. Training crawlers collect content for model building. Search crawlers build the index an assistant searches. User-triggered fetchers arrive because somebody is asking about you right now, and several operators document that these do not follow robots.txt, precisely because a person asked.
The distinction is not academic; it decides which lever works. Our own server logs show the crawlers requesting robots.txt before anything else, every time. In controlled runs, models with a fetch tool never requested it once (Note 04). Every piece of advice about signalling to AI through robots.txt is aimed at the population that is only indexing you.
One more distinction worth keeping straight: permission and readiness are different levers. Your directives decide who may read you; Google’s page-quality tooling separately audits whether a site is legible to agents at all, and the package passes the llms.txt and page-structure checks of Google’s agent-readiness audit; the WebMCP audits cover interactive tools a fact package deliberately does not install. Allowing a crawler in and giving it something worth reading are two jobs.
What we measured
Two populations, measured separately: crawlers read robots.txt always and first; fetch-tool models read it never (Note 04). Of sixty-four cold fetch-tool visits, 57 read one page and stopped (Note 27).
Source: MARKETING_NOTES_AGENT_VISIBILITY.md. Every note number is a dated measurement with its method and its cost.
disambiguation
What AI crawler is not
- It is not a scraper in the adversarial sense. The major operators publish their names, their address lists and their opt-out directives.
- It is not the thing answering your buyer’s question. That is usually a model with a fetch tool, reading one page.
one page each
The crawlers, individually
- ClaudeBot — Anthropic, training crawler
- GPTBot — OpenAI, training crawler
- OAI-SearchBot — OpenAI, search crawler
- Google-Extended — Google, usage control, not a crawler
- PerplexityBot — Perplexity, search crawler
free audit
See what AI sees on your site
Vocabulary is cheap. The measurement is the part that tells you whether any of it applies to your site.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.