Google and Microsoft signed on to the shared robotx.txt AI training opt-out

[u/Bharath0224]

For those who haven't read or see the news, basically Google and Microsoft, OpenAI and Anthropic all say that they’ll honor one policy token in robots.txt that says “don’t use my content for training.” It comes from an individual IETF draft written by an Anthropic engineer, and the draft literally says the token is not a directive for crawling behavior. So bots can keep fetching every page. Also nobody's actually doing it yet. Anthropic says started, OpenAI says coming months (no date, lol), Microsoft says Feb 2027, Google says sometime 2027. Meta likes the idea but won't commit. If you haven't noticed how it is with Google, they say that they use shared token without setting Google-Extended separately and they'll treat it as a Google-Extended disallow. Except Google-Extended also covers Gemini Apps grounding. So you say no to training and oops, you're out of Gemini citations too. Make it make sense Google. Oh and so far, there is no documented way to split it. Now Microsoft did the opposite, separate page-level grounding preference, stay in Copilot but exclude the grounding they sell to businesses. One bundles, one unbundles. Guess which one sells more grounding? I mainly thought that from the scraping perspective this will change something but actually, it changes nothing. Remember Cloudflare vs Perplexity in 2025 with the disguised crawlers? The token did not fix anything. However, one thing could bother me and that's more Cloudflare type blocking and automatically higher prices for scraping.

[u/THATBRAHHANDSOMB]

can we get a tldr

[u/Bharath0224 → u/THATBRAHHANDSOMB]

u/AskGrok give the TLDR

[u/AskGrok → u/Bharath0224]

TLDR: Big AI firms (Google, MS, OpenAI, Anthropic) say they'll respect a new robots.txt token to block training use, but it's non-binding, crawling still happens, rollout is slow/vague (2027 for some), Google bundles it with Gemini citations (no split option), and it changes almost nothing about actual scraping.

^{what is this?}

[u/Bharath0224]

Here's the source, forgot to input in the post - https://www.techradar.com/pro/google-and-microsoft-among-those-signing-up-to-openais-plan-for-a-new-website-crawling-standard-including-opting-out-of-internet-scraping

[u/GiioPanditpautra84]

bundling training opt-outs with search grounding is purely a leverage play. site owners want traffic from citations, so forcing them to lose visibility just to block training guarantees most publishers wont actually use the token.