Cloudflare’s AI bot settings, and five things to check
Since July, every Cloudflare plan can block AI bots by what they do, and since mid-September new domains start with some of them blocked. If your site is on Cloudflare, part of who gets in, and part of your robots.txt, may now be decided by a setting you never opened. Here is what each one does, from Cloudflare’s own documentation, and how to check yours.
What changed
- 1 July 2026. Cloudflare split AI traffic into three behaviours, Search, Agent and Training, and let every customer, the Free plan included, set each one on its own (changelog).
- 15 September 2026. Domains added to Cloudflare from this date start with Training and Agent blocked on pages that display ads; Search stays allowed. Every setting that blocks training also stops mixed-purpose crawlers, the ones used for both search and training. That includes the old “Block AI bots” switch, which Cloudflare marked as deprecating the same day (documentation).
The three settings
| Setting | What Cloudflare says it covers | What blocking it costs you |
|---|---|---|
| Search | Crawlers that collect or index your content so they can answer questions about it later. | Being found in those answers, and the visits they send. |
| Agent | Automated activity acting in real time for a person, such as chat fetch bots and browser-use agents. | When someone asks an assistant to open your page, it is refused. |
| Training | Crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers used for training and search. | Your pages stay out of those models. Through mixed-purpose crawlers, possibly out of that company’s search too. |
Each one has three choices: Block on all pages, Block on pages with ads, or Allow. Which pages count as showing ads is decided by Cloudflare’s automated detection, not by you. A block covers the verified bots Cloudflare classifies under that behaviour, plus unverified bots that behave the same way.
Five things that cause SEO trouble
1. A default you didn’t choose. If your domain was added to Cloudflare on or after 15 September and your pages show ads, Training and Agent are blocked on them unless someone changed it. If you want people’s assistants to be able to open your pages, look at Agent first.
2. Blocking training can reach into search. Cloudflare’s policy page says every configuration that blocks training, the old switch included, now also stops mixed-purpose crawlers. Its pages don’t list which crawlers those are. If you block training and a search or answer engine stops sending visitors, start here. AI Crawl Control → Crawlers lists every AI crawler that visits, its category, and how many of its requests failed.
3. “Search” means all search. Cloudflare’s bot taxonomy says that since 1 July there is “no longer a meaningful distinction” between AI search and traditional search; both are Search. Its documentation doesn’t say whether the Search policy touches Googlebot or Bingbot, and we haven’t tested it. Read Search → Block as “leave search”, and choose it only if that is what you mean. What was blocked shows in Security Analytics.
4. The robots.txt crawlers read may not be your file. With Bot Preference Sync on, Cloudflare writes lines for every policy you set to block and puts them above your own robots.txt, or serves a file of its own if your server has none (robots.txt setting). On the Free plan, a site with no robots.txt and no managed file gets Cloudflare’s Content Signals Policy: comments that define the categories but set no preference. So the robots.txt in your code is not necessarily the one Googlebot reads. For Training the same page also lists a Disallow choice: it writes the robots.txt lines and blocks training crawlers at the edge, but leaves mixed-use crawlers out of that block.
5. “Syntax not understood” in Search Console. The file Cloudflare manages carries Content-Signal lines, like this:
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /Google doesn’t read them, and Search Console may flag them. Cloudflare says it has seen no impact on crawling rates or SEO from those reports. Robots.txt itself is only a request; Cloudflare’s own page says compliance is voluntary. The edge block is what actually turns a bot away.
Google's training opt-out isn't a crawler
If what you want is to keep Gemini from training on your pages, a Cloudflare block can’t do it. Google’s control is Google-Extended, and it is a robots.txt token, not a crawler: nothing fetches your pages under that name, so there is nothing at the edge to stop. Google says it doesn’t affect inclusion in Google Search. It works only as a line in robots.txt:
User-agent: Google-Extended
Disallow: /Apple’s Applebot-Extended works the same way.
A five-minute check
- In the Cloudflare dashboard, open your domain → Security → Settings, filter by Bot traffic and open Configure AI bot policies. Note the three values, and whether Bot Preference Sync is on.
- Open AI Crawl Control → Crawlers: who visits, in which category, how many requests failed. Its Robots.txt tab lists crawlers that asked for paths your file disallows. On the Free plan it recognises crawlers by their user agent.
- Open https://your-domain/robots.txt in a browser. That is the file crawlers get. Paste all of it into the robots.txt validator: for Googlebot, Bingbot, the AI answer bots and the training bots it shows who may fetch a page, and the exact line that decides.
- Decide what you want and make the two agree. A common choice is Training blocked, Search and Agent allowed, and a robots.txt that keeps training bots out and lets answer bots in. The validator’s Rewrite my file button writes that file and shows every changed line first.
Common questions
Will blocking AI training hurt my Google rankings? Google says Google-Extended doesn’t affect Google Search. The Cloudflare risks are different ones: mixed-purpose crawlers, and the Search setting. Check both.
Is robots.txt enough to keep AI bots out? No. It says what you want; well-behaved crawlers follow it and others don’t. A block at the edge is what enforces it.
My site was on Cloudflare before 15 September. Does any of this apply? The new defaults are for new domains. But if you had the old “Block AI bots” switch on, it now also covers mixed-purpose crawlers, and Cloudflare is replacing it with the three policies.
Why does Search Console show errors in a robots.txt I never edited? If the site is on Cloudflare, open the live file and look for its managed lines (points 4 and 5): in Cloudflare’s example they end with # END Cloudflare Managed Content.
Check the rest of the site
A robots.txt that lets a bot in is one part of being found. The Trakis audit reads robots.txt for every page it crawls (the AI crawler check is AI-005) along with indexing, canonicals, structured data, links and speed, and counts each problem once, with the pages it covers. Create a free account and add your site, or read every check it runs.
Sources
All read on 13 October 2026.
- Cloudflare changelog, New options to manage AI traffic, 1 July 2026.
- Cloudflare docs, Block AI bots: the three policies, their choices and the 15 September defaults.
- Cloudflare docs, robots.txt setting, last updated 8 October 2026: Bot Preference Sync, Content Signals, Search Console.
- Cloudflare docs, Verified bots, last updated 8 October 2026: the behaviour taxonomy.
- Cloudflare docs, Manage AI crawlers, last updated 28 July 2026: AI Crawl Control.
- Google, Google’s common crawlers: Google-Extended.
Last changed 13 October 2026.