Cloudflare sets 15 September AI-crawler default-flip date and launches Pay-Per-Use publisher compensation

Cloudflare announced two paired changes on 1 July 2026 that reshape how publishers behind its network can control and monetise access by AI crawlers. From 15 September 2026, ad-monetised pages will block AI Training and Agent crawlers by default unless the publisher actively opts in. Traditional Search bots are unaffected. The default-flip applies to all new Cloudflare customers, all new sites added by existing customers, and every free-tier customer’s existing sites. Paid customers with existing sites retain their current settings unless they choose to change them. Cloudflare frames the change as the operational second act of its 2025 “Content Independence Day” – moving from opt-in blocking to opt-in access for the categories of AI traffic that don’t return referrals.

On the same day, Cloudflare’s earlier “Pay Per Crawl” marketplace evolved into “Pay Per Use.” The billing model shifts from charging AI crawlers per-fetch to compensating publishers per-answer-inclusion. In practice, publishers get paid only when their content is genuinely surfaced inside an AI-generated response, not for every crawl attempt against their pages. Initial ecosystem partners announced include Ceramic.ai and You.com. It is the most publisher-friendly compensation model any major infrastructure provider has proposed to date because it aligns payment with value received rather than with server load.

Why this matters

For any European remote-work publisher, community operator, or newsletter behind Cloudflare, this is a load-bearing configuration decision that needs a deliberate answer before 15 September 2026. Allowing Training bots may preserve long-term visibility in AI-generated answers where citations become the primary discovery path. Blocking Agent bots may stop Perplexity Comet-style browsing agents from finding the site at all. Neither choice is universally correct – it depends on whether audience growth comes from AI referral traffic, direct returning readers, or long-tail SEO. Small European publishers who rely on niche country-specific search traffic (remote-work Spain, digital nomad Croatia, freelance Netherlands) should audit the trade-off explicitly rather than accept the default.

The Pay-Per-Use launch also opens a genuine revenue path that didn’t exist a month ago. Publishers who negotiated no compensation from AI companies to date now have an infrastructure-level option that does not require bespoke deal-making. Watching how Ceramic.ai and You.com’s initial partnerships perform in the first quarter will indicate whether other AI providers follow, or whether the model stays niche.

Context

This follows Cloudflare’s 2025 launch of default AI-crawler blocking on new sign-ups (already covered in RWE’s May coverage of Cloudflare’s zero-referral data) and Playwire’s June critique that the default block was silently killing publisher GEO visibility. The 1 July announcements answer both threads: Cloudflare is doubling down on the publisher-first posture, but adding a compensation mechanism so that publishers who choose to keep Training bots on can get paid for it.

What to watch

• 15 September 2026: the default-flip goes live. Any publisher’s decision to keep current settings needs to happen before that date, not after.
• Q3 2026: first Pay-Per-Use compensation reports from Ceramic.ai and You.com integrations. Real revenue data will indicate whether the model is worth opting into for small European publishers.
• Cross-industry watch: whether OpenAI, Anthropic, or Google follow with any comparable per-answer compensation.