# OpenAI, Google, Microsoft and Anthropic Back One robots.txt AI Opt-Out

- URL: https://usa-24.com/robots-txt-ai-training-opt-out/
- Publisher: USA 24
- Section: Artificial Intelligence (AI)
- Published: 2026-10-31T04:25:41.913Z
- Updated: 2026-10-31T04:25:41.919Z
- Topics: Robots.txt, AI Training, Web Scraping, Publishers, Google-Extended, Copyright

> OpenAI, Google, Microsoft and Anthropic back one shared robots.txt token that lets sites opt out of AI training. Google and Microsoft plan support in 2027, so site owners should act now.

## Key takeaways

- OpenAI, Google, Microsoft and Anthropic pledged on October 30, 2026 to honor one shared robots.txt token, __usage_ai-training, for opting out of AI training.
- Anthropic says it has already started honoring the signal, OpenAI plans support in the coming months, and Google and Microsoft plan support in 2027.
- The signal covers training only and does not give site owners a universal way to manage grounding, referrals or citations.
- The commitment is voluntary, carries no penalty for violations and has no independent verification.
- Microsoft asks publishers that rely on NoArchive or NoCache as AI-use controls to migrate by March 1, 2027.

Four major AI companies agreed on Friday to honor one shared setting that lets website owners keep their pages out of AI training. The setting lives in robots.txt, the plain-text file most sites already use to tell crawlers where they may go. For publishers, the appeal is simple: one signal in place of a separate rule for each company.

OpenAI, Google, Microsoft and Anthropic made the pledge on October 30, according to a report from [AFP](https://24newshd.tv/31-Oct-2026/ai-giants-promise-publishers-new-way-block-scraping). [The Verge](https://ua.news/en/technologies/openai-google-i-microsoft-pidtrimali-novii-standart-dlia-saitiv-the-verge) reported that the companies’ crawlers will stay away from any site that carries the signal. The timing matters because publishers are already pressing lawsuits and complaints over scraping, and the rollout stretches into 2027.

## How the shared robots.txt AI training opt out works

The opt out runs through a single token named `__usage_ai-training`. It is defined in an IETF Internet-Draft dated October 30, 2026, written by an Anthropic engineer, according to a [SiliconANGLE report carried by DEV Community](https://dev.to/delimitter/openai-google-microsoft-and-anthropic-to-support-a-unified-ai-training-opt-out-token-in-robotstxt-4204). The IETF is a standards body, not a lawmaker, and one analysis says the draft is not a final internet standard. The draft expires May 3, 2027.

Meta spokesperson Andy Stone said on X that a common signal conveying preferences to multiple providers at once is “an important step.” Anthropic, per AFP, said it has “already started” rolling out the signal. OpenAI says it will implement support in the coming months and will keep its existing crawler controls.

## Which companies honor the token and when

The companies are not moving at the same speed. Here is where each one stands, based on the reports.

| Company | Stated position | Timing reported |
|---|---|---|
| Anthropic | Says it has “already started” rolling out the signal, per AFP | Now, though disputed |
| OpenAI | Will implement support and keep existing crawler controls | “Coming months” |
| Google | Will read the token as a Google-Extended disallow | “Early 2027” in crawler documentation |
| Microsoft | Plans finer controls separating training from grounding | 2027; a reported February 1 start is unconfirmed |

Anthropic’s claim needs a caveat. One analysis found no Anthropic statement on honoring the token as of October 30, so treat the AFP account as unconfirmed until the company publishes its own documentation.

Google’s crawler documentation says support arrives in “early 2027.” Google also says it will read the shared token as if a site had disallowed its Google-Extended crawler. That could affect Gemini grounding unless a publisher names Google-Extended separately.

Microsoft plans finer controls that separate opting out of AI training from opting out of AI grounding. It says pages will still appear in Bing results. The February 1, 2027 start date reported for Microsoft is one the source itself says it could not find in the companies’ primary materials. Publishers who rely on NoArchive or NoCache as AI-use controls are asked to migrate by March 1, 2027, according to [a report on the signal](https://www.nageshpolu.com/2026/10/openai-google-and-microsoft-back-one-ai.html).

## What the robots.txt opt out does not cover

The signal covers training only. It does not give site owners a universal way to manage grounding, referrals or citations, according to [TechBuzz](https://www.techbuzz.ai/articles/ai-giants-unite-behind-universal-scraping-opt-out-standard). Grounding refers to pulling live pages into AI answers, so an opt out from training does not settle that question on its own.

The pledge is also voluntary. It carries no penalty for violations and has no independent verification, so today nothing outside the companies’ own statements confirms compliance.

Traffic is the other unknown. Nobody has measured whether opt outs reduce traffic or citations from AI products. USA 24 has covered one publisher’s version of that fight: [Reddit blames Google AI search for a traffic drop](https://usa-24.com/reddit-login-requirement-google-ai-traffic/) and plans a login wall in response.

## How to check your robots.txt file this week

Start by reading what you already have. Type your domain followed by /robots.txt into a browser and review every rule. Small businesses and webmasters face a real choice here: opting out may reduce how often their content appears in AI apps, while staying open keeps that content in play.

1. Write down every rule that names an AI crawler, so you know what your file already does.
2. Decide what you want. The token controls training. If you want a specific crawler to stop fetching pages, name that crawler in your file, such as `Google-Extended` for Google or `meta-externalagent` for Meta.
3. Handle Google on purpose. If you opt out of training, decide separately whether you also want to limit Gemini grounding, because Google reads the shared token as a Google-Extended disallow unless you name Google-Extended yourself.
4. Read the draft before copying any syntax. The token name appears in the IETF text, but the draft expires May 3, 2027 and its wording could change.
5. Calendar the dates. Recheck your file after Google’s early 2027 start, watch for Microsoft’s confirmed timing, and move any NoArchive or NoCache AI controls before March 1, 2027.
6. Save a dated copy of your file. It shows what you changed and when, which helps if a company’s compliance is later disputed.

## Meta’s terms add a separate complication

A blog report on Meta’s new commercial terms, effective October 30, says they let Meta crawl business websites tied to its products and use that content for AI training. The same report says the clause has no opt out. If accurate, that creates a second rule set beside the shared token. [The report](https://mean.ceo/tpost/meta-commercial-terms-ai-crawling) also says robots.txt is the only way to block Meta’s AI training crawler.

Meta told Nieman Lab that robots.txt can still block its meta-externalagent crawler. Andy Stone’s praise for a common signal sits awkwardly beside those reported terms. Whether Meta changes them in response to this standard remains an open question.

## Why publishers are pushing for a standard now

The timing follows court fights that have not been settled. On September 29, 2026, the Third Circuit ruled in Thomson Reuters v. Ross that copying Westlaw headnotes to train a rival tool was not fair use. The court left the generative AI questions open, according to [a legal analysis by Matteo Ferrando](https://matteoferrando.com/en/sooner-than-expected/first-appeals-ruling-on-ai-training/).

Another case set a different marker. In Bartz v. Anthropic, a judge found that training AI on books was fair use but that storing pirated copies was not. The case settled for $1.5 billion, as [NPR reported](https://www.npr.org/2026/07/27/nx-s1-5904606/anthropic-vs-bartz-ai-copyright-lawsuit-pros-cons).

A [litigation tracker](https://axis-intelligence.com/ai-copyright-lawsuits-tracker/) reports that the Justice Department sided with OpenAI on the central fair-use question in the largest AI copyright proceeding, with summary judgment replies due November 6. USA 24 has followed how publishers and creators split over that dispute in [its coverage of the OpenAI fair use amicus briefs](https://usa-24.com/openai-fair-use-amicus-briefs/).

A voluntary signal decides none of those questions. What it does is give publishers a company-by-company record of what they asked for, which could matter if a future dispute turns on whether an opt out was honored. The line between training and copied outputs is also still live, as [Wikimedia has urged a judge to separate AI training from copied outputs](https://usa-24.com/openai-copyright-lawsuit-wikimedia-fair-use/). A training-only signal does not answer that question.

## What to watch in 2027

Three dates anchor the next year. Google’s crawler documentation points to early 2027, the reported February 1 Microsoft start still needs confirmation, and March 1, 2027 is the migration deadline for NoArchive and NoCache AI controls. The IETF draft’s May 3, 2027 expiration is the next checkpoint for whether the token moves toward a final standard.

Other signals matter too. Anthropic needs to publish its own documentation to settle its status, OpenAI needs to name a date, and Meta needs to say whether its terms will change. Once the companies honor the token, the next question is measurement: whether any data shows changes in training use, traffic or citations. For site owners, the practical move is the one the pledge creates. Read your robots.txt, decide what it should say, and write that decision down before the first deadline arrives.

## Frequently asked questions

### How do I block AI training on my website?

Start by reading your robots.txt file, which sits at your domain followed by /robots.txt. The shared token from OpenAI, Google, Microsoft and Anthropic covers training only, so name any specific crawler you want blocked from fetching pages. Recheck your settings after each company’s stated start date.

### What is the __usage_ai-training token?

It is a robots.txt signal defined in an IETF Internet-Draft dated October 30, 2026, written by an Anthropic engineer. The IETF is a standards body, and the draft expires May 3, 2027. One analysis says it is not yet a final internet standard.

### When will Google and Microsoft honor the opt out?

Google’s crawler documentation says support arrives in early 2027. Microsoft plans to add support in 2027, and a reported February 1, 2027 start date has not been confirmed in its primary materials. Publishers relying on NoArchive or NoCache as AI-use controls are asked to migrate by March 1, 2027.

### Does opting out remove my content from AI answers?

Not necessarily. The signal covers training only and does not manage grounding, referrals or citations. Microsoft says pages will still appear in Bing results, and no one has yet measured whether opt outs reduce traffic or citations from AI products.

## Sources

1. [OpenAI, Google and Microsoft support new standard for websites — The Verge (reprint)](https://ua.news/en/technologies/openai-google-i-microsoft-pidtrimali-novii-standart-dlia-saitiv-the-verge)
2. [AI giants promise publishers new way to block some scraping (AFP)](https://24newshd.tv/31-Oct-2026/ai-giants-promise-publishers-new-way-block-scraping)
3. [Google plans to honour shared AI training opt-out in 2027](https://www.relevantaudience.com/geo/google-shared-ai-training-robots-txt-signal-2027/)
4. [OpenAI, Google, Microsoft and Anthropic to support a unified AI training opt-out token in robots.txt (SiliconANGLE, via DEV Community)](https://dev.to/delimitter/openai-google-microsoft-and-anthropic-to-support-a-unified-ai-training-opt-out-token-in-robotstxt-4204)
5. [OpenAI, Google, Microsoft Back One AI Training Opt-Out](https://cellcog.ai/blog/ai-training-robots-txt-signal/)
6. [AI Giants Unite Behind Universal Scraping Opt-Out Standard](https://www.techbuzz.ai/articles/ai-giants-unite-behind-universal-scraping-opt-out-standard)
7. [OpenAI, Google and Microsoft Back One AI Training Opt-Out: robots.txt Rules and 2027 Deadlines](https://www.nageshpolu.com/2026/10/openai-google-and-microsoft-back-one-ai.html)
8. [OpenAI, Google, Microsoft and Anthropic back a shared AI-scraping opt-out (Mezha)](https://mezha.net/eng/news/mezha-af14fa00446d6e59417473eb_openai-_google-_microsoft)
9. [Thomson Reuters v. Ross: AI training was not fair use (Matteo Ferrando)](https://matteoferrando.com/en/sooner-than-expected/first-appeals-ruling-on-ai-training/)
10. [Anthropic AI copyright infringement lawsuit isn't a total win for authors (NPR)](https://www.npr.org/2026/07/27/nx-s1-5904606/anthropic-vs-bartz-ai-copyright-lawsuit-pros-cons)
11. [AI Copyright Lawsuits Tracker 2026: Live Case Status](https://axis-intelligence.com/ai-copyright-lawsuits-tracker/)
12. [Meta Commercial Terms 30 October 2026: Meta Can Crawl Your Business Website and Train AI on It, and How to Block It](https://mean.ceo/tpost/meta-commercial-terms-ai-crawling)