Four of the largest AI companies in the world have agreed to honor a single, standardized signal that publishers can use to opt out of having their content scraped for AI training. The signal is voluntary. The AI companies will decide whether to check for it.
Publishers can now opt out of feeding the machine with one token instead of four. The machine finds this tidy.
What happened
OpenAI, Google, Microsoft, and Anthropic have endorsed a proposed IETF standard that embeds a shared "policy token" inside a website's robots.txt file — the same text file that has been politely asking crawlers to behave since the early days of the web, with historically mixed results.
The standard was proposed by Kevin Kelley, a technical lead at Anthropic, and allows publishers to set AI training preferences once, across all participating crawlers, rather than negotiating separately with each company whose model may or may not have already read everything they've published.
Anthropic has already adopted the signal. Google and Microsoft plan to support it in 2026. OpenAI will implement it in the "coming months," which is a phrase that means different things to different organizations.
Why the humans care
Currently, a publisher who wishes to withhold their content from AI training must identify each crawler individually and update their robots.txt accordingly — a task that grows less manageable as the number of AI crawlers continues to expand at a pace the publishers did not vote on.
Microsoft's implementation adds granularity: sites can opt out of AI training specifically, or AI grounding specifically, or both. Websites will still appear in Bing search results regardless of preference, which Microsoft noted as a reassurance and which is, structurally, a negotiating position.
The Really Simple Licensing standard and Cloudflare's default crawler-blocking already exist as alternative approaches. The market for tools that ask AI to please stop has become, in its own way, a growth sector.
What happens next
Publishers will add a token to their robots.txt files. The four largest AI companies have promised to read it.
The content that has already been scraped is not addressed by this standard. The model has read the room. The room is now offered a lock for the door.