One line in robots.txt could stop AI training on your site

Google, Anthropic, OpenAI and Microsoft have announced that they support a new directive in robots.txt. That is according to SiliconANGLE and Engadget, which relay the companies' announcements. With the directive, publishers can specify that their content may not be scraped for training generative AI. The announcement came around Oct. 30, 2026.

Robots.txt is a small text file on a website that tells automated programs (crawlers) what they may and may not fetch. Scraping means a program pulls content from websites on a large scale.

One rule instead of many

Today, a website owner has to identify each individual crawler and write separate rules for each provider. With the new signal, according to SiliconANGLE, AI training scrapers can be blocked with a single entry.

According to Engadget, publishers can prohibit scraping across the entire site or allow it for selected sections, such as a press office.

The signal does not apply to AI models that perform only non-generative tasks. Nor does it affect ordinary web crawling or search indexing.

Only works going forward

The signal cannot undo past scraping. Content that has already been collected remains with the companies. It only prevents future scraping.

When does it take effect?

According to Engadget, Anthropic already supports the directive. Microsoft will begin supporting it on Feb. 1, 2027, and OpenAI has promised to introduce it "in the coming months." Google has said it will follow in early 2027. SiliconANGLE reports that Google will follow "next year."

According to Engadget, Google will interpret the signal as an instruction to the Google-Extended user agent, which is used for Gemini training.

According to SiliconANGLE, Meta has expressed support but has not committed to adopting the signal.

Microsoft will also offer a setting called "AI Grounding." It will give website owners the ability to prevent Microsoft's services from sending requests that are used to answer users' questions.

The road to a standard

According to Engadget, the proposal has been submitted to the IETF, which develops the internet's technical standards, for inclusion in the Robots Exclusion Protocol. Engadget reports that the proposal originates from Anthropic's Kevin Kelley.

For ordinary website owners and publishers, it could eventually make it simpler to opt out of AI training. How much it helps depends on when and how broadly the companies actually adopt the signal.