A new draft standard would let a website say "do not train on this" to many AI companies with one block in robots.txt. The IETF draft, Machine-Readable Training Preference Signal, was published on 30 October 2026, and OpenAI has said it will honour the signal in the coming months. Reports say Google, Microsoft and Anthropic back it too. It is useful, and it is easy to misread. Here is what it does and what it does not.
How does the signal work?
You add a group to robots.txt under a special name, __usage_ai-training, and use the normal Allow and Disallow rules under it. This is the example from the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That says: do not use my site for AI training, except the press kit, which you may use. One block, read by every company that supports the draft, instead of a separate rule for each company's crawler.
What does "training" mean here?
The draft uses a narrow definition: "Using an asset to modify the learned parameters of a generative AI model." Put simply, building or fine-tuning a model on your pages.
That is the important part, because a lot of AI use is not training. The draft does not say anything about search indexing, about an assistant fetching your page to answer a question, or about a user who pastes your URL into a chat. If your worry is AI answers using your content without sending visitors, this signal is not the tool for it. It covers what goes into the model, not what the model reads later.
What are the traps?
The wildcard does not count
The draft is explicit. A User-Agent: * group "MUST NOT be interpreted as matching" the training token. If your robots.txt says Disallow: / for everyone, that does not count as a training opt-out under this draft. You need the named group.
It only works going forward
Whatever was crawled before you add the line has already been crawled. The signal says what you prefer from now on.
It is a preference, not a lock
robots.txt has always worked on trust. Companies that support the draft say they will respect it. Others can ignore it. Nothing technical stops them.
It is still a draft
This is an early Internet-Draft from the IETF's AI Preferences work, written by an engineer at Anthropic. It expires in May 2027 unless it moves forward. Details, including the token name, can change.
Keep your existing rules
OpenAI says its current crawler controls stay in place. If you already block or allow specific AI crawlers by name, leave those rules alone. The new group adds to them. It does not replace them.
Should you add it?
It depends on what you want, so decide that first, page by page.
- Product pages, documentation, public guides. Many companies want AI models to know about them. If a model has learned your product, it can recommend it. You may want to allow training here.
- Paid content, original research, customer stories you were given permission to publish. These are the pages where an opt-out makes sense.
- Anything private. It should not be public at all. robots.txt is not access control, and anything you list in it is visible to anyone who reads the file.
What would I do this week?
Open your robots.txt and read it. Many are years old and nobody remembers why each line is there. Write down, in one sentence, what you want AI companies to do with your site. Then add the __usage_ai-training group that matches that sentence, deploy it, and check that the live file at /robots.txt shows it. It takes ten minutes, and it puts your preference on record while the companies that respect it are switching it on.
Where Alongside fits
Deciding what a company publishes, what AI companies may train on, and what stays private is the kind of work covered by Alongside's security and compliance engineering.
