AI crawlers › GPTBot
GPTBot: what it is, and how to allow or block it
GPTBot is the crawler OpenAI documents as gathering content to make its foundation models more useful and safe. It is not the robot that surfaces you in ChatGPT search, and confusing the two is the most expensive robots.txt mistake we see.
Training crawler · operator: OpenAI · every fact below is quoted from OpenAI, “Bots” (crawler reference) and links to it. Nothing on this page is inferred.
what the operator documents
The facts, and where each one comes from
| Fact | What the operator says | Source |
|---|---|---|
| Operator | OpenAI. | OpenAI docs |
| Name to match in robots.txt | GPTBot | OpenAI docs |
| Documented request string | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | OpenAI docs |
| Documented purpose | Crawling to “make our generative AI foundation models more useful and safe” — the training population. | OpenAI docs |
| What blocking it means, in the operator's words | “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” | OpenAI docs |
| Other robots the same operator names | OAI-SearchBot for search, ChatGPT-User for pages fetched because of a user action, and OAI-AdsBot for checking pages submitted as ads. | OpenAI docs |
| Published address list | https://openai.com/gptbot.json | OpenAI docs |
What the operator does not publish
- Any statement that allowing it improves how you appear in ChatGPT. The operator's page separates the training robot from the search robot; only the search one is documented as affecting what users see.
allow or block
The exact robots.txt lines
Paste these into the robots.txt file at the root of your site. Groups are matched by name, so a group for one robot has no effect on another.
Block it from everything
User-agent: GPTBot
Disallow: /Allow it explicitly
User-agent: GPTBot
Allow: /Block training, keep search — the line most owners actually want
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Block one section, allow the rest
User-agent: GPTBot
Disallow: /members/
Allow: /Rules are matched per group, so a group for GPTBot does not affect OAI-SearchBot and vice versa. A single User-agent: * block with Disallow: / hits every robot on this list at once.
what it means for you
What allowing or blocking it actually changes
Blocking it costs you
Nothing measurable in the answers a buyer gets today, by the operator's own description of what it is for.
Blocking it buys you
A documented signal that your content should not be used for foundation model training.
What it does not touch
The assistant that fetches your homepage while somebody is asking about you. Our own server logs show the crawlers reading robots.txt first, every time; in controlled runs, models with a fetch tool never read it at all (Note 04). Those are two different populations and robots.txt only reaches one of them.
And the lever that is not in this file at all
Permission is one job; being legible when the robot arrives is another. Google’s page-quality tooling now audits websites for agent-readiness, and the package passes Google’s agent-readiness audit — which is a claim about the surfaces on your site, not about who you let in. We make no claim that any of it improves your search position.
The distinction that decides everything on this page
| Crawlers like this one | A model with a fetch tool | |
|---|---|---|
| Reads robots.txt | Always, and first | Never, in our runs |
| What it is doing | Indexing, then leaving | Answering somebody’s question now |
| What reaches it | Your directives | The bytes of the page it fetched |
Our own server logs plus six models per site across two client sites: the crawlers requested robots.txt before anything else, every time, and not one fetch-tool model requested it once. Source: MARKETING_NOTES_AGENT_VISIBILITY.md, Note 04.
the rest of them
The other crawlers people ask us about
- ClaudeBot — Anthropic, training crawler
- OAI-SearchBot — OpenAI, search crawler
- Google-Extended — Google, usage control, not a crawler
- PerplexityBot — Perplexity, search crawler
Definitions for the vocabulary on this page: AI crawler, answer engine, llms.txt, fact block.
The separate question of whether your site is legible when one of these arrives is graded by Google’s own tooling now: our explainer on the Agentic Browsing audit covers every check it runs.
free audit
See what AI sees on your site
Blocking and allowing is housekeeping. The question underneath it is whether the page a robot reaches actually answers what your buyers ask.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.