Trakis

AI crawlers

CCBot

CCBot is the crawler of Common Crawl, a non-profit that publishes a free, open copy of a large part of the web. Language models have been trained on that copy, which is why lists of AI crawlers to block include CCBot, though it is not an AI company’s bot.

What it does

Common Crawl describes its archive as an open repository of web crawl data that anyone can access and analyse. Its own pages talk about research, not AI. But the archive is a common source of training text: the paper that introduced GPT-3 lists a filtered Common Crawl as its largest dataset, 410 billion tokens and 60% of the training mix (Brown et al., 2020, table 2.2).

Common Crawl’s FAQ says CCBot reads robots.txt first and fetches a page only if it is allowed, obeys Crawl-delay, and honours nofollow on links.

How it identifies itself

The token in robots.txt is CCBot. The user agent:

CCBot/2.0 (https://commoncrawl.org/faq/)

Common Crawl warns that other crawlers call themselves CCBot. The real one comes from its published ranges (ccbot.json), and its IPv4 addresses have reverse DNS: the address resolves to a name under crawl.commoncrawl.org, and that name resolves back to the same address.

Block it

The lines Common Crawl gives, as a group of their own:

User-agent: CCBot
Disallow: /

It checks robots.txt again from time to time, so a change is picked up without writing to anyone. To slow it down instead of blocking it, a Crawl-delay in its group is obeyed. Common Crawl also keeps an opt-out registry, linked from its FAQ.

What blocking it means, and what it does not

  • It stops CCBot fetching your pages for future crawls. Neither of Common Crawl’s pages says what happens to crawls already published, so ask them if that matters to you.
  • It is not a search engine’s crawler. Each bot reads its own group, so keeping CCBot out does not touch Googlebot, Bingbot or OAI-SearchBot.
  • It is one of several training crawlers. The others have their own pages: GPTBot, Meta-ExternalAgent, and the rest in the list of AI crawlers.

Check what your robots.txt tells CCBot

Paste your file into the robots.txt validator, opened with CCBot filled in. It runs in your browser and shows, for CCBot and every other search and AI bot, whether it may come in and the exact line that decides. The list of AI crawlers has the others.

Sources

Read on 14 October 2026; operators change these pages, so we read them again every month.

Last changed 14 October 2026.