AI crawlers › OAI-SearchBot
OAI-SearchBot: the one you probably should not block
OAI-SearchBot is the robot OpenAI documents as surfacing websites in ChatGPT’s search features. Of all the robots on these pages it is the one where blocking has a direct, documented cost to your visibility.
Search crawler · operator: OpenAI · every fact below is quoted from OpenAI, “Bots” (crawler reference) and links to it. Nothing on this page is inferred.
what the operator documents
The facts, and where each one comes from
| Fact | What the operator says | Source |
|---|---|---|
| Operator | OpenAI. | OpenAI docs |
| Name to match in robots.txt | OAI-SearchBot | OpenAI docs |
| Documented request string | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | OpenAI docs |
| Documented purpose | “For search. Used to surface websites in search results in ChatGPT’s search features.” | OpenAI docs |
| What blocking it means | The operator documents that disallowing it prevents a site from appearing in ChatGPT search. | OpenAI docs |
| Not a training crawler | The operator documents training crawling under a separate name, GPTBot, with its own robots.txt group. | OpenAI docs |
| Published address list | https://openai.com/searchbot.json | OpenAI docs |
What the operator does not publish
- A ranking mechanism. Nothing on the operator's page says how a site is ordered or chosen once it is eligible, and anyone selling you a position in an assistant's answer is selling something they cannot show you.
allow or block
The exact robots.txt lines
Paste these into the robots.txt file at the root of your site. Groups are matched by name, so a group for one robot has no effect on another.
Allow it explicitly — the usual answer
User-agent: OAI-SearchBot
Allow: /Block it — know what this costs
User-agent: OAI-SearchBot
Disallow: /Keep it out of one area only
User-agent: OAI-SearchBot
Disallow: /internal/
Allow: /A blanket User-agent: * with Disallow: / blocks this one too. That is how most sites accidentally remove themselves from an assistant’s search index while trying to opt out of training.
what it means for you
What allowing or blocking it actually changes
Blocking it costs you
Eligibility to be surfaced in that assistant’s search features, by the operator’s own description. This is the concrete one.
Allowing it does not fix your answers
Being fetchable is not the same as being answerable. Across sixty-four cold agent visits to twelve real company sites, 89% could not state a price for what the business sells, and 57 of 64 read one page and stopped (Note 27). The robot arrived. The answer was not on the page.
What to do after you allow it
Put the answers where a one-page reader will find them. That is the whole of our product and it is measurable: 78% fact recovery with the block in place against 30% without it (Note 41).
And the lever that is not in this file at all
Permission is one job; being legible when the robot arrives is another. Google’s page-quality tooling now audits websites for agent-readiness, and the package passes the llms.txt and page-structure checks of Google’s agent-readiness audit; the WebMCP audits cover interactive tools a fact package deliberately does not install — which is a claim about the surfaces on your site, not about who you let in. We make no claim that any of it improves your search position.
The distinction that decides everything on this page
| Crawlers like this one | A model with a fetch tool | |
|---|---|---|
| Reads robots.txt | Always, and first | Never, in our runs |
| What it is doing | Indexing, then leaving | Answering somebody’s question now |
| What reaches it | Your directives | The bytes of the page it fetched |
Our own server logs plus six models per site across two client sites: the crawlers requested robots.txt before anything else, every time, and not one fetch-tool model requested it once. Source: MARKETING_NOTES_AGENT_VISIBILITY.md, Note 04.
the rest of them
The other crawlers people ask us about
- ClaudeBot — Anthropic, training crawler
- GPTBot — OpenAI, training crawler
- Google-Extended — Google, usage control, not a crawler
- PerplexityBot — Perplexity, search crawler
Definitions for the vocabulary on this page: AI crawler, answer engine, llms.txt, fact block.
The separate question of whether your site is legible when one of these arrives is graded by Google’s own tooling now: our explainer on the Agentic Browsing audit covers every check it runs.
free audit
See what AI sees on your site
Blocking and allowing is housekeeping. The question underneath it is whether the page a robot reaches actually answers what your buyers ask.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.