Website owners have long had to write a separate robots.txt rule for each AI company that crawls their pages. Four of the biggest developers now say they will honor one shared line instead.
According to The Next Web, Google, Microsoft, OpenAI and Anthropic will each respect a single signal in robots.txt that keeps a site’s content out of generative AI training. The announcements came on 30 October 2026, and the rollout will be staggered.
Where the idea comes from
The signal is defined in an individual draft submitted to the Internet Engineering Task Force (IETF).
Reports say it was written by Kevin Kelley of Anthropic
, and it is titled Machine-Readable Training Preference Signal. SiliconANGLE, as relayed by DEV Community, says the draft is dated 30 October 2026 and expires on 3 May 2027.
It belongs to the work of the IETF’s AI Preferences working group. Affilae reports that the policy token is called __usage_ai-training. A site adds a robots.txt group with that token and lists the paths it does not want used for training.
The draft has no formal standing. The Next Web notes that the IETF datatracker says the organization has not endorsed it, and it is not part of the standards process. It may change.
Who starts when
Timing differs by company:
- Anthropic has already begun honoring the token.
- OpenAI confirmed on its blog that support will arrive in the coming months.
- Microsoft will honor the preferences from 1 February 2027.
- Google says it plans support during 2027.
Google’s plan, posted on its Search Central blog by John Mueller, builds on Google-Extended, the training control it has offered since 2023, according to Unite.ai. If a site uses the new token to opt out and has not set a rule for Google-Extended, Google intends to treat that as a Google-Extended block. Other providers can handle it in a similar way. Site owners could opt out across all participating companies or choose individually.
What it does not do
The limits are significant. OpenAI said the new signal does not replace its existing controls, so current robots.txt rules stay in force. It also said the token does not guarantee exclusion from training.
Microsoft said the new preferences do not set up licensing or payment arrangements, according to The Next Web. So the token lets a publisher say no. It does not create a way to be paid.
Training and live answers are treated as separate matters. Daily.dev’s summary of Microsoft’s approach says the company is adding a distinct page-level control for grounding, meaning the lookup of content to build AI answers. Blocking grounding keeps a page out of those lookups but, the summary says, cuts visibility in AI-generated results considerably. Microsoft’s new preferences will replace the NoArchive and NoCache tags it has used since 2023.
Affilae adds that the token does not affect search rankings in Google or Bing. Whether a page appears in Google’s AI Overviews or AI Mode depends on Google Search settings, not on this signal.
The regulatory backdrop
Google said the proposal is consistent with the EU’s Code of Practice for general-purpose AI under the AI Act, according to The Next Web. The code recognizes machine-readable ways for rights holders to reserve their content. Whether regulators will treat a voluntary, unofficial draft as sufficient is not addressed in the sources reviewed.
Robots.txt itself has always been a voluntary convention between site operators and crawler operators. Nothing in the draft compels a crawler to comply, and companies outside the four named are not bound. OpenAI has encouraged other AI providers to honor the signal.
For publishers, the practical gain is fewer lines of configuration. The practical question is enforcement, and the sources do not answer it.
