AI crawlers › ClaudeBot
ClaudeBot: what it is, and how to allow or block it
ClaudeBot is the crawler Anthropic documents as collecting web content for model training. It is one of three robots Anthropic names, and the other two are the ones that show up when somebody is actually asking a question.
Training crawler · operator: Anthropic · every fact below is quoted from Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” and links to it. Nothing on this page is inferred.
what the operator documents
The facts, and where each one comes from
| Fact | What the operator says | Source |
|---|---|---|
| Operator | Anthropic. | Anthropic docs |
| Name to match in robots.txt | ClaudeBot | Anthropic docs |
| Documented purpose | It “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” | Anthropic docs |
| Other robots the same operator names | Claude-User, which the operator describes as supporting Claude users — when individuals ask questions, it may access websites on their behalf — and Claude-SearchBot, which “navigates the web to improve search result quality for users.” | Anthropic docs |
| Does it honour robots.txt | The operator states its bots respect “do not crawl” signals by honouring industry standard directives in robots.txt. | Anthropic docs |
| Crawl rate control | The operator states it supports the non-standard Crawl-delay extension to robots.txt. | Anthropic docs |
| Published address list | A JSON file of source addresses for the operator's bots is published at https://claude.com/crawling/bots.json. | Anthropic docs |
What the operator does not publish
- A full request string. The operator's page names the robots to match in robots.txt but does not print the complete request header those robots send, so this page does not print one either. Match on the name.
allow or block
The exact robots.txt lines
Paste these into the robots.txt file at the root of your site. Groups are matched by name, so a group for one robot has no effect on another.
Block it from everything
User-agent: ClaudeBot
Disallow: /Allow it explicitly
User-agent: ClaudeBot
Allow: /Slow it down instead of blocking it
User-agent: ClaudeBot
Crawl-delay: 1Block training but keep the assistant and the search side
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /The absence of a rule is permission: a crawler that finds no matching group in your robots.txt treats the site as open. You only need the allow lines if you want the intent to be legible to the next person who edits the file.
what it means for you
What allowing or blocking it actually changes
Blocking it costs you
Nothing you can measure today in the answers buyers get. This is the training-time robot: the operator documents it as collecting content that could contribute to training, not as the thing that fetches your page when a person asks a question.
Blocking it buys you
Your content is not collected for that purpose. Whether that is worth anything to you is a judgement about your content, not a measurement, and we will not pretend otherwise.
Blocking the wrong one costs you real answers
If you block the whole family with a wildcard, you also block the robots the operator documents as serving people who are asking questions right now. That is the expensive mistake, and it is one line in a file.
And the lever that is not in this file at all
Permission is one job; being legible when the robot arrives is another. Google’s page-quality tooling now audits websites for agent-readiness, and the package passes Google’s agent-readiness audit — which is a claim about the surfaces on your site, not about who you let in. We make no claim that any of it improves your search position.
The distinction that decides everything on this page
| Crawlers like this one | A model with a fetch tool | |
|---|---|---|
| Reads robots.txt | Always, and first | Never, in our runs |
| What it is doing | Indexing, then leaving | Answering somebody’s question now |
| What reaches it | Your directives | The bytes of the page it fetched |
Our own server logs plus six models per site across two client sites: the crawlers requested robots.txt before anything else, every time, and not one fetch-tool model requested it once. Source: MARKETING_NOTES_AGENT_VISIBILITY.md, Note 04.
the rest of them
The other crawlers people ask us about
- GPTBot — OpenAI, training crawler
- OAI-SearchBot — OpenAI, search crawler
- Google-Extended — Google, usage control, not a crawler
- PerplexityBot — Perplexity, search crawler
Definitions for the vocabulary on this page: AI crawler, answer engine, llms.txt, fact block.
The separate question of whether your site is legible when one of these arrives is graded by Google’s own tooling now: our explainer on the Agentic Browsing audit covers every check it runs.
free audit
See what AI sees on your site
Blocking and allowing is housekeeping. The question underneath it is whether the page a robot reaches actually answers what your buyers ask.
One model fetches your homepage and answers buyer questions from it. You get the recovery score and the specific answers it could not find. That is the whole free tier: it shows you the problem, and the fix package is not included.