Cloudflare changes its AI-crawler defaults on 15 September, and its own dashboard makes the problem look worse than it is

From 15 September 2026, Cloudflare sets new defaults for the three categories of AI traffic it now distinguishes: Search, Training and Agent. On pages that display ads, Training and Agent crawlers will be blocked by default, while Search bots remain allowed. The new defaults apply to all new domains onboarding to Cloudflare, and Cloudflare has said they will reach existing free-tier sites in time. Customers with existing sites can opt out in their Security settings at any point before the date.

Announced the same day, Cloudflare is also reworking how publishers might be paid. In a companion post, it says it is “starting to shape Pay Per Crawl into Pay Per Use”, running experiments with two partners. Ceramic.ai has built what it calls a pay-per-query model, where publishers who opt in are paid when their content appears in its search results. You.com takes a different route, letting agents pay on demand for a specific piece of premium content. Both are explicitly experiments: Cloudflare has given no broad availability date and says only that it will share what it learns. The principle is worth noting even so, because payment would follow whether content was used rather than how often it was fetched.

Why this matters

For any European remote-work publisher, community operator or newsletter behind Cloudflare, this is a configuration decision that deserves a deliberate answer rather than a default one. Allowing Training crawlers may preserve long-term presence in AI-generated answers, where citation is becoming a real discovery route. Blocking Agent crawlers may stop browsing agents reaching the site at all. Neither answer is universally right: it depends on whether readers arrive through AI referrals, direct return visits, or long-tail search. Publishers who depend on narrow country-specific search traffic should work out the trade-off explicitly.

What we found when we checked our own configuration

Remote Work Europe sits behind Cloudflare, so we audited our own settings before the date rather than after. Three things are worth passing on, because they are not obvious from the dashboard.

The “unsuccessful requests” figure is misleading. Cloudflare’s AI Crawl Control reports a count of unsuccessful AI-crawler requests, and ours read around 2,000 in twenty-four hours, rising by 74%. That looks like crawlers being turned away in volume. It was not. The figure bundles 429 (rate limited), 403 (blocked) and 404 (page does not exist) into a single number, and in our case it was overwhelmingly 404s: old external links pointing at URLs that no longer existed. A publisher reading that number on Monday could reasonably conclude they were blocking AI crawlers when they were doing nothing of the kind.

Test the crawlers directly instead of trusting the dashboard. Requesting a real page with each user agent in turn, and reading the status code, answers the question in seconds. A 200 means the crawler is getting through, whatever the aggregate figure suggests. Anything else is worth investigating individually.

“AI crawlers” is not one decision, it is at least three. OpenAI alone runs separate crawlers for separate jobs: GPTBot gathers training data, OAI-SearchBot indexes for ChatGPT’s search results, and ChatGPT-User fetches a page live when somebody asks a question about it. Only the last of those produces a referral you can see in analytics. A publisher who blocks “AI bots” as a single category may keep the crawler that sends readers and lose the ones that build long-term presence, or the reverse, without ever intending either.

Context

This follows Cloudflare’s 2025 launch of default AI-crawler blocking on new sign-ups, covered in our reporting on the zero-referral data, and Playwire’s June critique that default blocking was reducing publisher visibility in AI answers. The July announcements answer both threads: Cloudflare is holding its publisher-first position while testing a way for publishers who keep Training crawlers on to be compensated for it.

What to watch

• 15 September 2026: the new defaults take effect. Anyone wanting to keep current settings needs to record that choice in Security settings before the date, not after.
• Existing free-tier sites: Cloudflare has said the defaults will reach these in time, without saying when. Worth checking your own configuration explicitly rather than assuming your plan exempts you.
• The Pay Per Use experiments: whether the Ceramic.ai and You.com pilots produce figures publishers can actually assess, and whether any of the larger AI companies follow with comparable per-answer compensation.