OpenAI, Google, Microsoft and Anthropic to adopt unified crawler opt-out standard

On October 31, 2026, The Verge reported, citing an earlier Search Engine Roundtable story, that OpenAI, Google, Microsoft, and Anthropic will adopt a new standard letting publishers opt out of being scraped by AI. The standard relies on a shared "policy token" signal: instead of configuring opt-outs separately with each company, a publisher can apply its AI training preferences across multiple AI companies' crawlers in one step, replacing a per-crawler setup. Emma Roth's piece at The Verge describes the new control as allowing publishers to apply their AI training preferences across multiple crawlers at once. At the time of the reports, all four companies had signaled support for the shared opt-out mechanism; the reporting does not specify a rollout timeline or when the standard will take effect. Both pieces appeared within roughly 40 minutes of each on October 31, and a Techmeme snapshot of the coverage confirms the story's prominence.

The standard addresses a practical pain point for publishers: until now, an AI training opt-out has meant repeating the same declaration with each company individually. Under the shared policy token approach, that declaration is made once and respected by every participating crawler. The coverage notes the signal is designed to work across multiple AI companies, though technical details such as how the token is generated and validated were not covered in the material reviewed here. As of the latest reports, OpenAI, Google, Microsoft, and Anthropic are the companies publicly committed to adopting it.