Cloudflare now writes your robots.txt for you, and AI training is a switch
Cloudflare announced Bot Preference Sync on 21 August 2026. The feature takes the AI-bot preferences a site owner sets in the Cloudflare dashboard and writes them directly into that site’s robots.txt, so the file reflects the dashboard settings without anyone maintaining it by hand. Generated rules are prepended to the existing file, and Cloudflare states that “any existing Disallow directives are maintained”.
Preferences are set across three categories. Search and Agent each offer allow, block on ad-serving pages, or block everywhere. Training is a straight allow or disallow. The feature is on by default for new customers, while existing customers – including those on the legacy managed robots.txt – are prompted to review their settings before the transition. Cloudflare said it would reach all customers on every plan, including Free, within the following week.
Why this matters
If your site sits behind Cloudflare, a dashboard setting you may never have opened is about to determine what your robots.txt says to AI crawlers. That matters most if you have deliberately allowed AI training crawlers in order to be cited by AI search tools, because “block Training” and “keep Search and AI citations” are now separate switches, and the defaults will not know which position you chose on purpose. The concrete action is to open the Cloudflare dashboard, look at what Search, Agent and Training are set to, and compare that against what your current robots.txt actually says. Do it before the prompt arrives rather than after, because the review prompt is easy to click through.
Context
This sits directly ahead of Cloudflare’s separate change on 15 September 2026, when it flips its default position on AI crawlers for sites it serves. Bot Preference Sync is the control surface for that shift: it is where the choice gets expressed and where it gets propagated to the file crawlers actually read. The wider backdrop is a year of publishers testing whether AI companies should pay for the content that trains and answers, from the APIG complaint against Google in France to the Penske Media case in Washington, where the judge said in late August that the arrangement “all seems really unfair”.
What to watch
The 15 September default flip is the date to have your settings finalised by. Check the dashboard, then check the live robots.txt afterwards to confirm the two agree.