> ## Content Index
> Fetch the complete content index at: https://bielawski.pro/llms.txt
> Use this file to discover other available public pages before exploring further.

# One robots.txt line instead of five - check what you're blocking before you turn off AI training
- URL: https://bielawski.pro/en/one-robots-txt-line-instead-of-five-check-what-youre-blocking-before-you-turn-off-ai-training/
- Published: 2026-11-13T06:09:30.000Z
- Updated: 2026-11-13T06:09:30.000Z
- Description: Google, Microsoft and OpenAI said they will honour a shared robots.txt token for AI training preferences. One line replaces separate instructions for each vendor - but it does not mean the same thing to each of them.
- Author: Bartosz Bielawski
- Tags: SEO, #temat-29747, #en

**Signal in the data**

Independent sources on this topic: **2** in the last week, **4** in 30 days, **4** in 90 days. We have tracked it since 2026-11-04; 6 claims in our database in total.

wk 340

wk 350

wk 360

wk 370

wk 380

wk 390

wk 400

wk 410

wk 420

wk 430

wk 443

wk 451

Number of independent sources per week (last 12 weeks).

## What is happening

Until now, a publisher who did not want its content used to train AI models had to give separate instructions to every vendor. Google has its Google-Extended token, Bing has its own settings, and OpenAI and Anthropic have others. Now a single shared mechanism is emerging.

Google announced support for a shared "policy token" in robots.txt, based on work by the AIPREF group at the IETF. Google, Microsoft and OpenAI said on 30 October that they will honour this shared control for AI training preferences. The mechanism builds on an Internet-Draft by Kevin Kelley of Anthropic.

## Where the sources agree

- **One token, not five.** A publisher declares its AI training policy once, not separately for each vendor.
- **Existing controls stay.** Google-Extended and other vendors' mechanisms keep working and can refine the shared signal.
- **Blocking training is not the same as blocking indexing.** These are two different signals, and confusing them is the most common implementation mistake.
- **Google reads a shared opt-out as an opt-out of Google-Extended** if the site has not named Google-Extended explicitly. That token also covers citation in the Gemini app.
- **Microsoft is cleaning up old signals.** After 1 March 2027 it will stop treating NoArchive and NoCache as training or grounding preferences.
- **The legal footing is getting stronger.** The EU code for GPAI models treats robots.txt as the primary way for publishers to express their preferences.

## Where they disagree

The sources agree on the direction, but not on the scope. Google-Extended covers both training and citation in the Gemini app, while at Microsoft the old NoArchive and NoCache signals worked differently and will only be retired in March 2027\. The same robots.txt line can therefore mean something different to each vendor.

The second point of disagreement is when the shared token will actually be honoured. The mechanism is still going through standardisation, and some announcements are promises about the future. Until standardisation is complete, per-vendor controls remain the only reliable signal.

## What to do about it

Write your site's AI policy in three sentences: whether your content may be used to train models, whether it may be used as a source in AI answers, and whether it should be indexed by search engines. Only then choose your tokens.

Set the shared token as your default declaration, and handle exceptions through Google-Extended and vendor settings. After rollout, check that blocking training has not also cut you off from AI citations - that is the most common side effect.

## Our take

A shared robots.txt token simplifies the declaration, but it does not even out its effects across vendors. Publishers gain one place for their AI policy and lose the excuse that they did not know how to express it. The biggest risk is not a lack of control but unknowingly cutting off AI citations along with training.

**Confidence:** medium - four independent sources and Google's own blog confirm the direction, but the mechanism is still being standardised and vendors' scopes are not equivalent. **We will change our view if:** If vendors honour the token only partially, or Google and Microsoft adopt different definitions of the same line, the shared mechanism would be worth less than separate controls.

## Sources

- Top Online, 2026-11-13: [SEO News #264 – SEOwcy boją się o pracę, spam UGC, Reddit z ograniczeniami](https://toponline.pl/blog/seo-news-264?ref=bielawski.pro)
- Search Engine Journal, 2026-11-08: [Google Rank Data Issues, Lighthouse Update & AI Opt-Out – SEO Pulse](https://www.searchenginejournal.com/seo-pulse-rank-data-problems-pagespeed-scores-ai-training-opt-out/596104/?ref=bielawski.pro)
- Google Search Central Blog, 2026-11-04: [Supporting a shared AI training control for websites](https://developers.google.com/search/blog/2026/10/ai-training-control?ref=bielawski.pro)
- Search Engine Land, 2026-11-04: [Google also gives publishers more control over how AI uses their content](https://searchengineland.com/google-also-gives-publishers-more-control-over-how-ai-uses-their-content-499971?ref=bielawski.pro)