Claude-SearchBot and Claude-User

Anthropic says Claude-SearchBot indexes content to improve the quality of search results for Claude’s users. Keeping it out stops that indexing, and Anthropic says it may reduce your site’s visibility and accuracy in the results Claude gives its users.

The token is Claude-SearchBot.

When a person asks Claude something that needs a page, Claude may open it as Claude-User. Keeping it out stops Claude from fetching your content for those questions, which Anthropic says may reduce your visibility in user-directed web search.

Here Anthropic differs from OpenAI, which says robots.txt rules may not apply to the fetches a person starts, and Perplexity, which says its fetcher generally ignores them. Anthropic’s page says its bots honour robots.txt and makes no exception for this one. The token is Claude-User.

ClaudeBot collects web content that could contribute to training Anthropic’s models. Anthropic says keeping it out signals that your site’s future material should be left out of its training data. It has nothing to do with whether Claude can find or cite you: that is the two above.

With nothing about them in your file, all three are allowed. To keep training out and say plainly that search is welcome:

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

• One file per subdomain. Anthropic says the rule has to be added for every subdomain you want to opt out: blog.example.com reads its own robots.txt, not the one on example.com.
• Crawl-delay works here. Anthropic supports it, as a way to slow its bots without keeping them out: Crawl-delay: 1 under the bot’s line. Google ignores the same line, so it slows only the bots that read it.
• The full user agents are not on Anthropic’s page; it names the tokens. Match on the token in your logs and your rules.

• A wildcard that closes everything. User-agent: * with Disallow: /, then groups that let in Googlebot and Bingbot. Claude-SearchBot has no group of its own, so it reads the * group and stays out.
• One group for every AI bot. A list of AI user agents above a single Disallow: /, copied to stop training, often has Claude-SearchBot and Claude-User in it beside ClaudeBot.
• Blocking by IP. Anthropic warns that refusing its addresses may not opt you out reliably, because it keeps the bots from reading your robots.txt. It publishes the addresses its bots come from in bots.json; a request from one of them is Anthropic’s. Use the list to let them through a firewall, and robots.txt to say what they may read.

The Trakis audit reports the first two as AI-005, with how many of your pages each bot is kept out of; the robots.txt validator shows the line that does it.

Paste your file into the robots.txt validator, opened with Claude-SearchBot filled in. It runs in your browser and shows, for Claude-SearchBot and every other search and AI bot, whether it may come in and the exact line that decides. The list of AI crawlers has the others.

• GPTBot
• OAI-SearchBot and ChatGPT-User
• Google-Extended and Applebot-Extended
• CCBot
• PerplexityBot and Perplexity-User
• Meta-ExternalAgent

Read on 28 October 2026; operators change these pages, so we read them again every month.

• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Anthropic, crawler IP addresses (JSON)