AI Training Opt-Out: What the New Shared robots.txt Signal Means for UK Businesses

Keeping your website out of AI training has meant chasing a growing list of bot names in robots.txt, one provider at a time.

That is starting to change. On 30 October 2026, Google, Microsoft and OpenAI each announced they will honour a single, shared AI training opt-out in robots.txt. The same day, Anthropic's Kevin Kelley, who wrote the proposal, told the IETF mailing list that Anthropic had started implementing it. One line, intended to be read by several of the biggest AI providers. None of them says it is reading the token yet.

The detail matters, because opting out of AI training is not the same as opting out of AI answers. At Google, the planned reading of the new signal covers both. Here is what was announced, how the AI training opt-out works, and how to decide whether to use it.

What was announced on 30 October

Four things landed on the same day:

• Google published "Supporting a shared AI training control for websites" on the Google Search Central blog. John Mueller wrote that website owners will be able to use a shared policy token in robots.txt to set preferences across participating AI providers, and that Google aims to support the control in 2027. Its crawler documentation, updated the same day, says Google intends to support the signal in early 2027.
• Microsoft published a Bing blog post saying it will honour the shared training preference from 1 February 2027, alongside a separate control for what it calls AI grounding.
• OpenAI said it will honour the common signal and aims to implement it in the coming months, with its existing crawler controls still applying. It calls the effort voluntary.
• Anthropic's Kevin Kelley submitted the proposal to the IETF AI Preferences working group. PPC Land reports that he told the group's mailing list Anthropic had already started implementing the draft, and that he did not give a date.

The proposal itself is an IETF Internet-Draft, "Machine-Readable Training Preference Signal". It is a work in progress, not a finished standard, and the draft is due to expire on 3 May 2027 unless it is revised.

How the AI training opt-out works in robots.txt

The shared token

Instead of naming a crawler, you name a purpose. The draft introduces one policy token, __usage_ai-training, and you use it on the User-Agent line like any other robots.txt group. The draft's own example looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That says please don't use this site for AI training, except for anything under /press-kit/, where training is allowed. The draft does not change how robots.txt matches paths, so you can opt out of the whole site or just part of it.

The draft uses the working group's definition of AI training: using content to modify the learned parameters of a generative AI model. Training models that are used only for non-generative tasks, such as ranking, classification or scoring, falls outside it.

What the token doesn't do

• It doesn't block crawling. The draft says the token is not a directive for crawling behaviour. It expresses a preference about how content is used, not whether a bot can fetch it. If you want to stop a crawler visiting, you still need a rule for that crawler by name.
• A wildcard rule doesn't count. Under the draft, a User-Agent: * group must not be read as matching the token. If your robots.txt has no __usage_ai-training line, you have expressed no preference at all, even if you block everything with a wildcard.
• Old vendor tokens don't carry over. A Google-Extended block is not treated as this shared opt-out for other providers. Each existing proprietary token keeps its current meaning only for the company that defined it.
• Private deals can override it. The draft allows private agreements to take precedence over the signal.

And it is a preference, not an enforcement mechanism. OpenAI describes its own participation as a voluntary effort.

Who supports it, and when

PPC Land, which reported the four announcements, notes that Meta and Amazon appear in the draft's acknowledgements but were not named among the companies committing to read the token. The IETF lists those affiliations for identification, not as an endorsement.

Training vs grounding: the part that affects your visibility

There are two very different things an AI company can do with your pages.

Training shapes what future models know. Your content becomes part of the data that adjusts the model.

Grounding happens when someone asks a question right now. The AI retrieves live web pages to build its answer and, often, cites and links to them. For a business, grounding is where AI visibility, and any traffic from AI answers, actually comes from.

The new shared token is defined as a training preference. The providers do not treat grounding the same way.

Google

Google says that when a site uses the shared token to opt out, and does not set Google-Extended explicitly, it plans to treat that as if the site had disallowed Google-Extended. Google's crawler documentation says Google-Extended manages whether crawled content may be used to train future Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. A blanket shared-token opt-out, with no Google-Extended line of your own, is therefore planned to remove you from that Gemini grounding as well. Google's blog puts the other side of the same rule: sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

The same documentation says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews and AI Mode are Search features, so they are governed by your Googlebot rules and by the separate Search Console control covered in our guide to the Search Console AI performance report, not by this token.

Because Google says it will apply the shared opt-out only when Google-Extended is not set explicitly, a site can disallow the shared token and add an explicit Google-Extended allow if it wants to stay eligible for Gemini grounding. Google has not said how it will translate path-level shared-token rules.

Microsoft

Microsoft splits the two. The shared robots.txt token covers training. Grounding has its own page-level preference, set with a meta tag or HTTP header, with three options: Allow (the default), DisallowExtended and Disallow. DisallowExtended limits grounding to Microsoft-operated experiences, which Microsoft lists as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes grounding services it provides to other companies. Microsoft's own example of the split many businesses will want is training set to Disallow and grounding set to Allow.

Microsoft says that choosing DisallowExtended or Disallow for grounding will significantly reduce your content's visibility in AI-generated experiences, and that neither choice guarantees inclusion, traffic or payment. These settings do not change eligibility for ordinary Bing search results.

If your site currently uses NoArchive or NoCache to control AI use in Bing, note the dates. Microsoft says a temporary grounding transition opens in Bing Webmaster Tools on 15 December 2026, and only for sites already using those tags. It says those tags will no longer be read as training or grounding preferences after 1 March 2027.

OpenAI

OpenAI already separates the two through different bots. Its crawler documentation says GPTBot crawls content that may be used to train its generative AI foundation models, while OAI-SearchBot is used to surface websites in ChatGPT search. The settings are independent, so a site can allow OAI-SearchBot and disallow GPTBot. OpenAI says the shared signal will not replace those controls. It describes the signal as covering foundational generative AI training, and it has not said whether "foundational" is narrower than the IETF definition.

Should you opt out of AI training?

There is no single right answer. It depends on how your website makes money.

Reasons to keep allowing training

• You sell services, not content. For most UK SMEs, a website exists to win enquiries. Having AI models understand who you are, what you do and where you operate is usually a benefit, not a loss.
• You want to be recommended. Training does not guarantee an AI assistant will mention you, but a brand that models know little about starts from behind. Our post on AI search optimisation in 2026 covers how citations work.

Reasons to consider an opt-out

• Your content is the product. Publishers, course creators, researchers and anyone who earns money from original writing or data have a stronger case for limiting training.
• You have licensing plans. If you might license content to AI companies, a clear public preference is a sensible baseline, and the draft allows private agreements to override it.
• You want to protect specific sections. Path-level rules mean you can opt out a members area, a resource library or premium guides while leaving marketing pages open.

The middle route most businesses should look at

For many sites, the sensible position will be: allow search, allow grounding, and decide on training section by section. Remember that at Google a blanket shared-token opt-out, with no Google-Extended line, is also planned to take you out of Gemini grounding. An explicit Google-Extended allow is the route Google describes for staying eligible.

What UK businesses should do now

Nothing changes overnight, but this is a good moment to tidy up.

1. Read your current robots.txt. Look for AI-related groups such as GPTBot, Google-Extended, ClaudeBot or Applebot-Extended, and check whether they were added on purpose.
2. Check you haven't blocked search by accident. Rules aimed at AI crawlers sometimes end up catching Googlebot, Bingbot or OAI-SearchBot. Blocking those affects ordinary search or AI search visibility, which is a far bigger problem than training.
3. Write down your position on training and on grounding separately. They are different decisions now.
4. If you use NoArchive or NoCache for Bing, plan the switch before 1 March 2027. Microsoft's temporary Webmaster Tools step opens on 15 December 2026 for sites already using those tags.
5. Hold off on relying on the shared token alone. Until providers say they are reading it, keep any existing vendor-specific rules you still want. The token is meant to sit alongside them, not replace them.
6. Make the change during a planned update. If a site rebuild or migration is coming, fold robots.txt into the checklist. Our guide to website redesigns and SEO covers what else to protect.

Where BlackFire stands

Our own robots.txt says "Search and AI/answer engines are welcome", and explicitly allows GPTBot, OAI-SearchBot, Google-Extended, ClaudeBot and PerplexityBot. For an agency that wants to be found, that is the right call for us. It will not be the right call for every client.

AI visibility is part of our SEO service. AEO and GEO sit on top of the technical, on-page and content work, with regular checks of how AI assistants describe a brand. We are also straight that nobody can guarantee what an AI assistant says. What we can do is make sure your robots.txt reflects a deliberate decision, your important pages are crawlable, and your content is clear enough to be cited. If you want more background first, our post on GEO and AEO for UK SMEs explains why classic SEO still does most of the heavy lifting.

If you also advertise, the same questions are showing up in paid search inside AI experiences. We covered that in Google AI Mode and paid search and ChatGPT ads in the UK.

AI training opt-out FAQs

What is the new AI training opt-out?

It is a shared robots.txt policy token, __usage_ai-training, proposed at the IETF. Adding a Disallow rule under it tells participating AI providers that you would prefer your content not to be used to train generative AI models. Google, Microsoft and OpenAI have said they will honour it, and Anthropic's Kevin Kelley told the IETF mailing list that Anthropic had started implementing the draft.

Does opting out of AI training affect my Google rankings?

Google says Google-Extended, which it plans to map a shared-token opt-out to, does not affect a site's inclusion in Google Search and is not used as a ranking signal. Microsoft says its training and grounding preferences do not change eligibility for Bing search results.

Will the AI training opt-out remove me from AI Overviews?

Google's crawler documentation says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are Search features, controlled by Googlebot rules and by the separate Search Console control covered in our Search Console guide, not by this training token. At Google, though, a shared-token opt-out is planned to be treated as a Google-Extended disallow, which also covers grounding in Gemini Apps unless you explicitly allow Google-Extended.

Does blocking all bots with a wildcard count as an opt-out?

Not under the draft. A User-Agent: * group must not be treated as matching the shared token. If your robots.txt doesn't mention __usage_ai-training, no training preference has been expressed through this mechanism.

When does the shared AI training signal take effect?

Microsoft says it will honour the new preferences from 1 February 2027. Google's crawler documentation says it intends to support the signal in early 2027, and its blog says it aims to support the control in 2027. OpenAI says it aims to implement it in the coming months, and Anthropic has not given a date. The IETF draft expires on 3 May 2027 unless it is revised.

Next step

Not sure what your robots.txt is telling AI companies, or whether an AI training opt-out makes sense for you? Book a call and we'll check what's in your file, flag anything blocking search or AI visibility, and give you an honest view on whether to allow or limit training. You can see the full scope on our SEO service page.