Trakis

AI crawlers

GPTBot

GPTBot is the crawler OpenAI uses to collect pages that may be used to train its models. Blocking it in robots.txt keeps your pages out of that, and nothing else: ChatGPT search is a different bot, set separately.

What it does

In OpenAI’s words, GPTBot “is used to crawl content that may be used in training our generative AI foundation models.” A site that disallows it, they say, signals that its content should not be used for that.

It is not OpenAI’s only agent, and the others are set separately: OAI-SearchBot, which decides whether your pages can appear in ChatGPT search, and ChatGPT-User, which opens a page when someone asks ChatGPT to.

How it identifies itself

The token in robots.txt is GPTBot. The full user agent, as OpenAI gives it in its example (the version number may change, OpenAI notes):

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Anyone can send that header. OpenAI publishes the addresses GPTBot really comes from at openai.com/gptbot.json: a request that says GPTBot from anywhere else is not OpenAI.

Block it

Two lines, as a group of their own, not directly under another User-agent line:

User-agent: GPTBot
Disallow: /

Or only part of the site, here everything under /articles/:

User-agent: GPTBot
Disallow: /articles/

Keep training out, and stay in ChatGPT search

OpenAI says each of its settings is independent of the others, and gives this case as the example: allow OAI-SearchBot to appear in search results while disallowing GPTBot.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The validator can write this for your own file: its “Rewrite my file” button keeps the training bots out and the answer bots in, and shows every changed line before you copy it.

What blocking it does not do

  • It does not take you out of ChatGPT search. That is OAI-SearchBot.
  • It does not stop a page from being opened when a person asks ChatGPT for it. That is ChatGPT-User, and OpenAI says robots.txt rules “may not apply” to it.
  • It is a request in a file, not a lock. A firewall or Cloudflare’s AI bot settings can block GPTBot at the edge, which robots.txt does not show: what Cloudflare’s settings do.

The mistake to look for

A file that blocks GPTBot on purpose sometimes blocks OAI-SearchBot by accident, most often through a User-agent: * group with Disallow: / that lets in only Googlebot and Bingbot. Every AI bot without a group of its own is out with it. The Trakis audit reports that as AI-005.

Check what your robots.txt tells GPTBot

Paste your file into the robots.txt validator, opened with GPTBot filled in. It runs in your browser and shows, for GPTBot and every other search and AI bot, whether it may come in and the exact line that decides. The list of AI crawlers has the others.

Sources

Read on 14 October 2026; operators change these pages, so we read them again every month.

Last changed 14 October 2026.