Robots.txt validator

The same way Google does, and the same way the Trakis audit does — it uses the same parser:

• A crawler reads one group: the one that names it, or User-agent: * if none does. Groups that name the same crawler are merged.
• Inside that group, the longest matching path wins. When an Allow and a Disallow are equally long, the Allow wins.
• * matches any characters and $ marks the end of the address. Paths are case-sensitive.
• No matching rule means allowed. So does a missing robots.txt.
• Google reads only the first 500 KiB, ignores Crawl-delay, and stopped reading Noindex in robots.txt in 2019.

Every verdict shows the line that decides it, so you can see why — not only what.

Blocking AI training does not have to cost you AI search. OpenAI, Anthropic and Perplexity each run separate bots for their search, for the pages an assistant opens when a person asks, and for collecting training data — and each reads its own line in robots.txt. Keeping out GPTBot and ClaudeBot does not keep ChatGPT or Claude from finding and citing you.

Three details most robots.txt checkers get wrong:

• Google-Extended and Applebot-Extended are not crawlers. Nothing fetches your pages under those names; they control how Google and Apple may use pages their search crawlers already fetched. Blocking Google-Extended does not affect Google Search.
• ChatGPT-User and Perplexity-User fetch a page because a person asked. OpenAI says robots.txt rules “may not apply” to them; Perplexity says its fetcher “generally ignores” them. The check shows what your file asks; what each company does with it is on its own page, linked under every bot.
• Search engines stay search engines: Google’s AI Overviews and Microsoft Copilot read through Googlebot and Bingbot. Blocking those is blocking search.

A file that lets a bot in does not mean the bot gets in. Firewalls and CDNs can refuse it before it ever reads robots.txt. On Cloudflare, since 1 July 2026 every plan can block AI bots by what they do — search, agent or training — and, from 15 September 2026, domains newly added to Cloudflare start with training and agent bots blocked on pages that show ads (Cloudflare’s changelog, documentation). That block happens at Cloudflare’s edge, and this check cannot see it. Cloudflare can also put lines of its own at the top of the robots.txt it serves, so paste the file from your live site, not the one in your code. What each setting does and what to check: Cloudflare’s AI bot settings.

The Trakis audit reads robots.txt for every page it crawls, together with the 65 other checks — indexing, canonicals, structured data, links, speed — and counts each problem once, with the pages it covers. Free: create an account and add your site. What this tool checks here is AI-005 in the list of checks.