Google, Microsoft, OpenAI and Anthropic said on October 30 that they will support a single robots.txt signal that lets website owners state their content should not be used to train generative AI models, in place of the separate rules each company’s crawler requires today. The mechanism comes from an Internet-Draft published the same day by Kevin Kelley of Anthropic, according to the IETF’s datatracker. The Internet Engineering Task Force has not endorsed it, and the companies are starting to honor it on different schedules, with Microsoft saying it will begin on February 1, 2027.

How the signal works

The draft is titled Machine-Readable Training Preference Signal. It asks site operators to write ordinary robots.txt groups that name a policy token, __usage_ai-training, in the User-Agent line. According to the draft, it adds “no new directive names and no new grammar,” and the rules for path matching in RFC 9309, the robots.txt standard, stay as they are. A Disallow rule under the token states a preference against AI training for that path, and an Allow rule states a preference in favor.

The draft’s own example disallows an entire site and then allows its press kit. TNW describes the same example. Two details matter for site owners. A wildcard group that applies to all crawlers does not count as the token, and a robots.txt file with no group naming it expresses no preference at all. Existing company-specific tokens, such as Google-Extended, keep their current meaning but are not treated as preferences under the draft.

The draft is explicit about what it is not. It describes the token as not a directive for crawling behavior, and it contains no enforcement mechanism: each consumer decides whether to honor it. It says private agreements between a site and an AI company may override the preference, and it does not address data collected before a preference was expressed. Parsers that do not recognize the token simply ignore the group, which preserves legacy behavior, and the two leading underscores make accidental matches by loose parsers unlikely.

The datatracker lists the document as an individual submission, not endorsed by the IETF and with no formal standing in the standards process. It expires on May 3, 2027 unless it is renewed. It borrows its definitions of AI training from a separate IETF working-group draft on AI usage preferences. Lens analysis: this makes it a proposal that fits alongside that work rather than a product of it, since the working group’s own drafts are listed separately.

Who is supporting it, and when

Microsoft has published the most detailed schedule, in a post on its Bing blog by principal product manager Krishna Madhavan. The company is separating two kinds of use: training, and what it calls grounding, in which content informs live answers in Bing, Copilot and Microsoft 365. The training setting can be placed in robots.txt or on a page, with the page-level setting winning, and the default is Allow. Grounding can be set only at page level, with three options: Allow, DisallowExtended, which limits use to Microsoft’s own products and excludes third-party services built on its technology, and Disallow.

Microsoft says it will begin honoring the new preferences on February 1, 2027. They replace its NoArchive and NoCache tags for training and grounding, and those tags will no longer be read for that purpose after March 1, 2027. Sites that use the old tags can record a temporary grounding preference in Bing Webmaster Tools from December 15, 2026, and those temporary preferences expire on September 30, 2027. Microsoft says eligibility for ordinary Bing search results does not change, but that restricting grounding reduces a brand’s visibility in AI experiences. Its post adds: “Neither choice guarantees inclusion, traffic, or payment.”

Google plans to support the signal in 2027, according to TNW, which reports that if a site uses the token without mentioning Google-Extended, Google will treat the token as a block on Google-Extended. TNW adds that Google’s John Mueller said sites that allow Google-Extended can appear in answers in the Gemini app and receive clicks. SiliconANGLE separately reports Google’s support will come sometime next year. Google has also said the proposal fits the European Union’s Code of Practice for general-purpose AI, which recognizes robots.txt as a main way for publishers to state preferences, TNW reports.

OpenAI says it will implement the signal in the coming months, according to TNW and Exame, and TNW says its existing controls such as GPTBot continue to apply. OpenAI’s own announcement could not be opened for this article. Meta’s communications head, Andy Stone, voiced support for a common signal on X but did not say whether Meta will honor it, TNW reports, and SiliconANGLE says Meta has not committed to implementing it.

Anthropic’s status is described differently by different outlets. TNW says Anthropic has started building support, that the signal will complement its existing crawler controls, and that Kelley said it will take time to add support across its services. Exame reports that Anthropic says it has already started implementing the measure, and Xataka México, citing a third-party site, says Anthropic already supports the token. Anthropic’s own announcement was not available for this article.

What it does not do

The limits are substantial, and several outlets stress them. Xataka reports that compliance is voluntary, with no formal sanctions and no independent verification, and that smaller AI labs and open-source projects are not bound unless they adopt the standard themselves. Exame says the initiative covers only future model training, does not change use of content in earlier training, and does not stop crawling in general.

Because the default in Microsoft’s scheme is to allow training, publishers who want out have to act. Exame reports Microsoft saying training use is allowed by default unless site operators configure a block. Microsoft’s post says the preferences do not establish licensing or payment arrangements, so the signal is not a route to being paid for content.

On search, the companies say the two are separate. Exame reports that Microsoft and Google both say blocking training will not affect a site’s presence in their search engines. Microsoft’s post is narrower, saying only that eligibility for Bing results is unchanged. SiliconANGLE adds that the standard does not apply to models that perform exclusively non-generative tasks, and that how compliance would be enforced is not explained.

Why publishers wanted one signal

Until now, a site that wanted to opt out of AI training had to write a separate rule for each company’s crawler and track changes to them. Google introduced its Google-Extended token on September 28, 2023, as a robots.txt setting that let publishers keep their content out of training for Google’s AI tools, though Maginative noted at the time that Google’s announcement did not address copyright or payment.

The coordination problem has been recognized inside the standards community. A report from an IETF Internet Architecture Board workshop on AI controls, published in May 2026 as RFC 9969 by Mark Nottingham and Suresh Krishnan, says the use of robots.txt by AI crawlers has not been coordinated and that publishers must check with each vendor to learn how to opt out. Participants in that workshop, held in September 2024, also said robots.txt cannot be the only solution. The report itself is not endorsed by the IETF.

The IETF formed a working group on AI preferences in January 2025, according to a post on its blog dated June 29, 2025, with Krishnan and Nottingham as co-chairs. The group is developing a shared vocabulary for AI usage preferences and a specification for attaching them to content through robots.txt, HTTP headers or both. As of the datatracker’s current listing, those two drafts are still active, and Kelley’s submission is among the related drafts filed alongside them.

The commercial and legal pressure is also visible in the coverage. Exame notes that the announcement comes amid disputes between technology companies and news organizations over copyrighted content, including a lawsuit by The New York Times, and says the initiative does not resolve them. TNW says the draft thanks staff from the Financial Times, the BBC, Bertelsmann and the European Publishers Council for shaping the discussion, and points to Cloudflare’s call in July for AI crawlers to pay or be blocked.

What to watch

The first test is timing. Microsoft has set specific dates, beginning with the Bing Webmaster Tools option on December 15, 2026 and honoring of the signal from February 1, 2027. Google’s support is expected sometime in 2027, OpenAI’s in the coming months, and Anthropic’s depends on which account is accurate. Until each company ships, a site that adds the token is relying on a promise.

The second test is standing. The draft expires on May 3, 2027, and its status as an individual submission means the IETF process, not the four companies, will decide whether it becomes a standard. The working-group drafts that supply its vocabulary are working-group documents, a stage ahead of an individual submission. Lens analysis: if the working group adopts a single approach, the robots.txt token could end up reworked or replaced, which would matter to sites that deploy it now.

The third test is who else joins. Meta has expressed support without a commitment, and the sources do not say whether other crawler operators will adopt the signal. Since the token carries no enforcement, its practical value will depend on how many operators honor it and on whether publishers can verify that they do. One discrepancy is worth noting: SiliconANGLE says the four companies submitted the draft together, while the datatracker lists Kelley of Anthropic as the only author.

Sources

Image: Outside Googleplex in Mountain View, California.jpg by David Nagle, CC BY-SA 4.0, via Wikimedia Commons. Illustrative photo; it does not show the events described.