Google, Microsoft and OpenAI back one robots.txt line to opt out of AI training, starting in 2027

Google, Microsoft and OpenAI have said they will honour a shared robots.txt instruction, User-Agent: __usage_ai-training, that tells all participating providers not to use a site’s content to train AI models. None of them honours it yet. Microsoft starts on 1 February 2027, Google is aiming for early 2027, and OpenAI says “in the next few months”. The token is described in an Internet Draft rather than an official standard, and Anthropic has started work on support without a public timeline.

The details differ by company. At Google, using the shared line without a separate Google-Extended rule will be treated as disallowing Google-Extended, which covers both training future Gemini models and grounding answers in Gemini Apps and Vertex AI. It does not affect inclusion in Google Search, and AI Overviews and AI Mode have their own control in Search Console. Microsoft keeps training and grounding as separate settings and says grounding restrictions have no effect on being listed in Bing results.

Why this matters

For publishers and anyone who runs a website, this is the first single switch for AI training across the major providers. But training and citation are different things. A site that wants to be quoted in AI answers needs to check what each provider bundles with the opt-out before adding it, because at Google the same line also removes the site from Gemini’s grounding.

What to watch

Bing Webmaster Tools opens a new setting on 15 December 2026 for sites still using NoArchive and NoCache tags, which Microsoft stops recognising for training and grounding after 1 March 2027.