AI crawlers, and what your robots.txt tells each
A robots.txt speaks to each crawler by name. The AI ones come in three kinds, and they are not one decision: the ones that answer people decide whether an assistant can cite you, the ones that train models decide nothing about search.
Search engines
Google and Bing — and through them, AI Overviews and Copilot.
| Googlebot Google | Google Search and AI Overviews. |
| Bingbot Microsoft | Bing and Copilot. |
AI answers
The bots that let an assistant find, read and cite your pages when someone asks.
| OAI-SearchBot OpenAI | ChatGPT search. OpenAI: a site that keeps it out does not appear in ChatGPT search answers. Separate from GPTBot. |
| ChatGPT-User OpenAI | Pages ChatGPT opens for a user. OpenAI: because a person starts these fetches, "robots.txt rules may not apply". |
| Claude-SearchBot Anthropic | Claude search. Anthropic: its bots honour robots.txt. |
| Claude-User Anthropic | Pages Claude opens for a user. Anthropic: its bots honour robots.txt, this one included. |
| PerplexityBot Perplexity | Perplexity search. Perplexity: used to surface and link sites in its answers, not for training. |
| Perplexity-User Perplexity | Pages Perplexity opens for a user. Perplexity: because a person asked, this fetcher "generally ignores robots.txt rules". |
AI training
Keeping these out does not stop an assistant from citing you; it is a choice many sites make on purpose.
| GPTBot OpenAI | Training OpenAI models. OpenAI: keeping it out signals the content should not be used for training. Search is OAI-SearchBot, set separately. |
| ClaudeBot Anthropic | Training Claude. Anthropic: collects content that may be used for training. Claude search is Claude-SearchBot. |
| Google-Extended Google | Training Gemini. Not a crawler: a token for whether Googlebot's pages train Gemini and ground Gemini Apps. Google: it "does not impact a site's inclusion in Google Search". |
| Applebot-Extended Apple | Training Apple Intelligence. Not a crawler: Apple says it "does not crawl webpages"; it controls whether Applebot's pages train Apple's models. |
| CCBot Common Crawl | The open dataset many models train on. Common Crawl: CCBot reads robots.txt before anything else and obeys Crawl-delay (its FAQ). |
| Meta-ExternalAgent Meta | Training Meta AI. Meta: crawls for uses such as training foundation models. |
What a robots.txt can and cannot do
Each crawler reads the group that names it, or the User-agent: * group when none does. That is how a file meant to stop training ends up stopping search too: a * group that closes everything, opened again only for Googlebot and Bingbot, keeps every AI answer bot out without naming one. The Trakis audit reports that as AI-005.
A Disallow is a request. The fetchers that act for a person, ChatGPT-User and Perplexity-User, may not follow it, by their operators’ own account. And a firewall or Cloudflare’s AI bot settings can block a crawler before it reads robots.txt at all: what Cloudflare’s settings do.
Check your own file
The robots.txt validator shows, for every crawler above, whether your file lets it in and the line that decides, in your browser. From a crawler’s page it opens with that crawler filled in, like this one for OAI-SearchBot.
Last changed 14 October 2026.