The robots.txt AI training opt-out is one shared line, and it does not turn training off today.
On 30 October 2026, OpenAI, Google, and Microsoft each said they will honor one shared line in robots.txt (a plain-text note a website leaves for visiting programs). The line is supposed to mean “do not use these pages to train a generative model” (a model that writes new text, pictures, or code, instead of only sorting things that already exist). It does not mean that today. Google pointed at 2027. Microsoft pointed at 1 February 2027. OpenAI said “the coming months,” and gave no date. Delete the old per-bot rules because a headline said one line is enough, and the crawlers that already train will keep taking the pages.
What robots.txt Actually Is
A website is just a computer that stores pages and sends a copy when something asks. The asker is often a browser (the app a person uses to look at the web). It is also often a crawler, also called a bot or a spider (a program that visits pages on its own and saves a copy). Search engines (companies that let you look words up across many sites) run crawlers so they can keep a search index (their private, searchable copy of those pages). robots.txt lives at the root of the site, the address that looks like https://example.com/robots.txt. It is not a password and not a law. It is a note on the door. Polite programs read it. Rude programs walk past. The rulebook for that note is RFC 9309 (a finished document from the IETF, the group that writes voluntary internet rules). A finished document is not the same thing as a lock on your server (the computer holding the site).
Old Per-Bot Rules vs the New Shared Token
For years the note had to name each bot. A group in the file starts with User-Agent (the name you are talking to) and then Disallow (please do not use this path) or Allow (this path is fine). OpenAI’s training crawler is still GPTBot. Its search crawler, the one that can surface your site inside ChatGPT search, is OAI-SearchBot, and that one is not the training bot. ChatGPT-User shows up when a person asks ChatGPT to open a page, not when OpenAI is out collecting a training set. Google’s older training switch, still the one that works, is Google-Extended. Anthropic has long told sites to use ClaudeBot for the training crawl. Check Anthropic’s crawler page the day you edit the file, because bot names get added. The new proposal does not invent a new kind of file. It invents one shared name that means a use, not a company. That name is a policy token (a User-Agent line that names a purpose). The token in the 30 October 2026 draft is __usage_ai-training.
The draft is draft-kelley-aipref-training-preference-signal-00, posted 30 October 2026 by Kevin Kelley at Anthropic. It is an Internet-Draft (a proposal), not an RFC, and this copy expires 3 May 2027 if nobody updates it. It adds no new commands. Same groups, same Allow, same Disallow. Its own example is three lines:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Read that slowly. Disallow: / is a preference against training on the whole site. Allow: /press-kit/ is a preference in favor of training on everything under that folder. The second line is a yes. Copying the example without deleting it is how a press kit walks into a training set on purpose. The draft also says that if the file has no __usage_ai-training group at all, no preference has been expressed. Silence is not a no. Case does not matter: User-agent and User-Agent are the same command.
What Training Means in This Draft
“Training,” in the vocabulary draft this token points at, means using a page to change the learned parameters of a generative model. A parameter, also called a weight, is a stored number inside the model. Training is the process that moves those numbers so the model gets better at the examples it was shown. Inference (answering with a model that is already trained) does not move them. So this line is about future baking, not about a chatbot quoting you tomorrow. It also cannot unbake a cake. A model whose numbers were already updated on an older copy of your page does not forget you because a text file changed. And the vocabulary draws a hard edge: training a model that is used only to classify, rank, or score (a sorter, not a writer) is outside this category, even if that model could have been used to generate. A spam filter is not what this line is about. A model that writes articles is.
The same draft says content-usage preferences and traffic rules are orthogonal (they do not control each other). Disallow under __usage_ai-training is not “do not fetch the page.” A company can still crawl you for ordinary search, and still fetch you when a person asks a question, while agreeing not to fold you into the next training run. That is why the first diagram has three doors. The new line only addresses the middle door. Microsoft even split the third door into its own control, called grounding (using a live page to inform a generated answer, often with a citation). Google, in the same 30 October post, said sites that specifically allow Google-Extended stay eligible to show up in Gemini answers and get clicks from that grounding. Opting out of training is not the same knob as opting out of being quoted.
When Each Company Says the New Line Starts
The dates are the whole story, and they are not the same date.
| Who | What they actually wrote | When the new line starts to matter | What still works in the meantime |
|---|---|---|---|
| OpenAI | Will honor the signal. It “will not replace” existing robots.txt controls. Implementation “in the coming months,” with a later post when it is real. | No calendar date | GPTBot for training. Leave OAI-SearchBot alone if you still want ChatGPT search. |
Plans to support the shared control in 2027. If you opt out with the token and you do not mention Google-Extended yourself, Google plans to treat that as a Google-Extended disallow. | 2027, no month named on the Search Central post | Google-Extended | |
| Microsoft | Will honor the token as it crawls, and will also accept a page-level training tag that overrides the file for that one page. Search eligibility does not change. | 1 February 2027 | Whatever you already set, until that crawl, plus the March deadline below |
| Anthropic | Kelley, who wrote the draft, works there. Several news writeups say implementation has begun. Treat that as unverified until Anthropic’s own crawler docs say the token is live. | Unknown | ClaudeBot, after you confirm the name |
One more Google detail, because it surprises people who want a single global no. An explicit Google-Extended group wins for Google. You can disallow the shared token and still allow Google-Extended, or the reverse. “Google plans to interpret the token as Google-Extended” only applies when you did not write a Google-Extended group yourself. If you want Google out, write Google-Extended in your own hand. Do not rely on a mapping that is not switched on yet.
The Trap If You Delete the Old Rules
The failure mode I would bet on is tidiness. Someone adds the new group, deletes GPTBot, Google-Extended, and ClaudeBot, and feels done. Those three names are the switches that work this week. The new name is a switch most of the readers have not installed. A normal crawler does not apply a group unless the group’s name matches that crawler, or matches * (the group that means “everyone else”). __usage_ai-training matches neither GPTBot nor Googlebot until that company writes special code to look for it. Adding the group is safe. Replacing the old groups with it is how you opt back in by accident.
Do not “fix” this with User-Agent: * and Disallow: / unless you also want to disappear from ordinary search. The star is every unnamed bot, including the ones that only want to index you. The new token is the narrow tool. The star is a boarded-up shop.
Microsoft hid a second clock inside a useful post. Starting 1 February 2027, Bing honors the new training preference as it recrawls, and a page-level tag overrides the file for that page. On 1 March 2027, NoArchive and NoCache (older Microsoft tags that people used as a blunt “do not keep a copy”) stop counting as a training or grounding preference. If Microsoft finds either of the new preferences on the site, it drops those old tags immediately, even before March. If a page was relying on NoArchive or NoCache to stay out of grounding, and you do not record a temporary grounding preference in Bing Webmaster Tools (Microsoft’s site-owner dashboard) before 1 March 2027, Microsoft says those pages default to allowing grounding until the new tag is recrawled. Training and grounding are different sentences. Saying no to one does not say no to the other.
Bing’s own examples, for a whole-site training no, and for a single page:
User-agent: __usage_ai-training
Disallow: /<meta name="Bingbot" content="ai-training:disallow">X-Robots-Tag: Bingbot: ai-training:disallowGrounding, if you also want out of generated answers, is a different phrase on the page, not a second robots.txt token. Microsoft’s forms are ai-grounding:disallow (no grounding) and ai-grounding:disallowextended (grounding only inside Microsoft’s own products, not the grounding Microsoft sells to other companies). Allow is the default. You do not need a robots.txt line to permit training. You need one to refuse it.
What to Put in robots.txt Today
What I would paste today, above whatever else is already in the file, and only after reading the file so I do not contradict a group that is already there:
User-Agent: GPTBot
Disallow: /
User-Agent: Google-Extended
Disallow: /
User-Agent: ClaudeBot
Disallow: /
User-Agent: __usage_ai-training
Disallow: /Leave OAI-SearchBot allowed if ChatGPT search is still welcome. Leave the normal Googlebot and Bingbot groups alone if you still want search traffic. Then open https://example.com/robots.txt in a browser and confirm the file you think you uploaded is the file the world gets. Caches (saved older copies) and a CDN (a network of copy-servers in front of your real server) can serve yesterday’s note. The opt-out is not real until the public URL shows it.
None of this is a copyright license, a lawsuit, or a guarantee. OpenAI, Google, and Microsoft each said they intend to read it, on different calendars. Anthropic employs the draft’s author; that is not the same as a dated promise. Companies that never joined the announcement will not read it at all. Pages already inside a trained model stay there. Quotes in a live answer are a separate preference, and only Microsoft has spelled that second preference out with dates. The useful act on 1 November 2026 is boring: add the new group, keep the old groups, and put 1 February 2027 and 1 March 2027 on a calendar if Bing has been using your old tags as a shield.
Common questions about the AI training opt-out
Does __usage_ai-training block crawlers today?
No. OpenAI gave no calendar date. Google pointed at 2027. Microsoft pointed at 1 February 2027. Keep GPTBot and Google-Extended (and Anthropic’s training crawler name) until each company says the new reader is live.
Is silence a refusal?
No. The draft says that if the file has no __usage_ai-training group, no preference has been expressed.
What did we check?
Opened the IETF draft page for draft-kelley-aipref-training-preference-signal-00 (token present) and OpenAI’s bots docs (HTTP 200) on 9 November 2026.
