How sites treat search engines and AI crawlers

What we measure ourselves, across thousands of sites rather than one: each study with its question, its method, what it cannot tell you, and its data and code on Zenodo for anyone to check.

4 November 2026

Block training, keep answers? What 13,938 sites ask of AI bots

Of the sites that keep GPTBot out by name, how many also keep out ChatGPT’s search? Across 24,419 domains, read from Common Crawl’s archive: 17.1%, and 31.3% of the sites that wrote the rule themselves.

Data and code: doi.org/10.5281/zenodo.23416952

How we run them

The question comes first and stays narrow, so a number can answer it.
Nothing comes from our customers’ audits. A study reads public data, and says where it came from.
Before a number goes out, we read a sample by hand without looking at what the code said, and report how often the two agreed.
The data and the code go on Zenodo with a DOI, so the numbers can be checked and cited.
Our crawler and what it does for a study are on the bot page, with the address to write to if you want a site left out.

Last changed 4 November 2026.