Site owners who want to keep their content out of AI training currently maintain a list: GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, and whatever ships next. A new policy token, __usage_ai-training, is meant to replace that list with one robots.txt group. It is defined in an Internet-Draft submitted to the IETF on October 30, 2026 by an Anthropic engineer, and on the same day Google, Microsoft and OpenAI announced that they will honor it.

That group covers training only, and no provider says it reads the token in production yet: the announced rollouts run into 2027. The draft's definition of training also leaves out answer-time grounding, and at Google, opting out with the token takes a site out of Gemini grounding as well.

How the token works

The draft, draft-kelley-aipref-training-preference-signal-00 by Kevin Kelley of Anthropic, adds no new robots.txt syntax. It reuses the grammar of RFC 9309, the Robots Exclusion Protocol. A group whose User-agent line names the policy token carries the preference, and ordinary Allow and Disallow rules bind it to paths:

User-agent: __usage_ai-training
Disallow: /
Allow: /press/

A disallowed path expresses a preference against training; an allowed path expresses a preference for it. Path matching and longest-match precedence work exactly as they do for crawlers, so in the example above /press/ is open to training and everything else is not.

A User-agent: * group never matches the token, so a site that blocks every bot today has, under this draft, expressed no training preference at all.

The token governs what a provider may do with content, and leaves fetching alone. The draft states it "is not a directive for crawling behavior." A provider may skip fetching content it has been asked not to train on, but it is not required to, and crawl control stays with crawler-specific groups.

Rules for Google-Extended or GPTBot behave as before and are never read as the shared preference. Each provider may document that its own token outranks the shared one, which Google does, and a private agreement between a site and a provider can override the signal.

The token's double-underscore prefix exists, per the draft, to make accidental matches unlikely in deployed parsers that match user-agent names loosely.

What "AI training" means here

The token borrows its definition from the IETF AIPREF working group's vocabulary draft (version 08, September 2026). Training means using content to modify the learned parameters of a generative AI model. Models used exclusively for non-generative tasks, such as classification or ranking, are outside it.

Feeding a page to a model at answer time is a different category in the same vocabulary, "AI Use", and whether it covers retrieval by URL is still an open issue there. The shared token says nothing about it. Blocking training does not, by this definition, keep a page out of AI-generated answers. The vocabulary draft also notes that its usage categories do not yet have working-group consensus, so the definition the token depends on can still move.

How each provider will apply it

The four companies committed on the same day, with different start dates and different readings of "training".

Google says that when a site opts out with the shared token and does not name Google-Extended explicitly, it will treat that as a Google-Extended disallow (Google Search Central). But Google-Extended controls two things at once: Google's crawler documentation says it covers training future Gemini models and grounding. So at Google, the shared opt-out also removes a site from Gemini grounding; Google's post notes that only sites which allow Google-Extended are eligible to appear in Gemini Apps answers via grounding. The only way to stay eligible for Gemini answers is an explicit Google-Extended allow, which also permits Gemini training. Google offers no way to split the two. Its target is "in 2027"; the crawler docs say early 2027.

Microsoft separates the two questions entirely (Bing blog). From February 1, 2027, it honors the shared token for training as pages are recrawled, across Bing, Copilot and Microsoft 365, with a page-level override (<meta name="Bingbot" content="ai-training:disallow">). Grounding is a separate, page-level-only preference with three values: Allow (the default), Disallow, and DisallowExtended, which limits grounding to Microsoft-operated experiences and excludes the grounding services Microsoft sells to third parties. The older NoArchive and NoCache tags stop counting as AI preferences after March 1, 2027. Microsoft adds that "neither choice guarantees inclusion, traffic, or payment."

OpenAI commits to honoring the signal for what it calls foundational generative AI training, with implementation in the coming months, and says the token will not replace its existing robots.txt controls (as reported by PPC Land). GPTBot rules stay in force.

Anthropic has no public launch date. Kelley's message to the working group says the company "has already started implementing support for this draft." Existing tokens such as ClaudeBot continue to apply.

Meta, Amazon and Apple made no commitment on October 30, so Applebot-Extended and other vendor tokens still need their own lines.

Why this design is reasonable, and where it is weak

The draft makes conservative choices that are easy to defend. Reusing RFC 9309 means nothing changes for parsers that ignore it, and no existing file suddenly means something new. Treating wildcards as silence avoids reading training intent into files written before training preferences were a consideration. Separating use from access lets a site stay crawlable for search while stating a training preference.

Its weaknesses are structural too. Nothing makes a crawler comply: robots.txt has never been an access control, and the draft adds no enforcement, so the signal is only as good as each provider's implementation. The preference does not reach back either. The draft ties it to the robots.txt in effect when content was retrieved, so adding the token today says nothing about data already collected or models already trained.

The draft is not a standard yet. It is an individual submission with no IETF standing, it expires in May 2027 unless revised, and Kelley offers it as input to the working group's attachment work, which could change the mechanism.

Providers also read "training" differently. Google bundles grounding into its interpretation, Microsoft keeps the two apart, and OpenAI limits its commitment to "foundational" training. The same robots.txt file can therefore produce different outcomes per provider, which undercuts the main reason for having a shared token.

My take, for the client sites I work on (documentation, marketing pages, app support): if the client doesn't want its content used for training, add the token now. It costs one robots.txt group, does nothing until providers switch it on, and records the preference before the 2027 rollouts begin. The vendor lines stay alongside it.

Gemini grounding is the harder call, and I would not make it the same way for every site. For documentation and support content, being cited in AI answers is how users find the fix, so I would keep grounding and accept Gemini training as the price of an explicit Google-Extended allow. Where the content is the product itself, such as original writing, research or paid material, I would opt out and give up Gemini answers. Google forces one switch for both, so for each site I would ask whether control over training matters more than showing up when someone asks.

What site owners should do now

Parsers that do not implement the draft ignore the new group, so adding it has no effect today. What matters is how it will behave once Google and Microsoft switch it on in 2027. A reasonable approach:

  1. Decide two questions separately: training, and appearing in AI-generated answers (grounding).
  2. If you opt out of training, add a __usage_ai-training group and keep your existing vendor lines next to it. Keep GPTBot, Google-Extended, ClaudeBot and Applebot-Extended rules until each provider confirms it reads the token.
  3. Check for an existing explicit Google-Extended rule. Google maps the shared token only when Google-Extended is not named, so an explicit allow wins over the shared opt-out.
  4. Accept the Google trade-off knowingly: staying in Gemini answers requires allowing Google-Extended, which also permits Gemini training.
  5. If you rely on NoArchive or NoCache to restrict Bing grounding, record a temporary grounding preference in Bing Webmaster Tools (available from December 15, 2026) and move to the new page-level controls before March 1, 2027.
  6. Do not rely on User-agent: * to express a training preference. Under the draft, it expresses none.

What to watch next

OpenAI and Anthropic have not given start dates for reading the token, and Google's announcement does not say how it will translate path-level rules into its single Google-Extended switch. Whether Apple, Amazon and Meta join is also open. The working group may keep this mechanism or reshape it in its attachment draft before this draft expires in May 2027.

Sources