Researchers fear OpenAI is now learning from their security work

When security researchers test attacks and defenses together with an AI, the conversations can contain sensitive information. That is exactly the kind of data OpenAI will now start learning from.

Starting Nov. 29, the company will use data from so-called Daybreak Blue sessions to train its models. That is according to The Decoder, which cites an email from OpenAI to members of the company's Trusted Access program. Yoav Alon is named as the source of the information.

Data was previously exempt

Until now, Daybreak Blue data has been kept out of training. That also applied to users who had turned off the setting to improve the model for everyone. In other words: even those who had already said no to contributing will not be covered by the old exemption once the change takes effect.

After swift criticism, OpenAI sent a new email. In it, the company promised a separate opt-out. The sources do not say how it will be designed, when it will arrive, or exactly what it will cover.

Interactions flagged during security reviews can also continue to be used for safety purposes. What that means in practice, and which data it covers, has not been specified.

Fear of a repeat

According to The Decoder, the change is raising concern among researchers because it may recall the Buckmaster-Alpöge case. In that case, mathematicians Tristan Buckmaster and Levent Alpöge uploaded drafts about the Navier-Stokes equations to OpenAI's Codex. Shortly afterward, OpenAI itself presented a proof, but denied having used their input for training.

According to The Decoder, the suspicion alone was enough to show researchers what can go wrong when an AI company both receives and can train on their work.

For ordinary users, the takeaway is that what happens to the data you give an AI service can change, even if you have previously opted out of training. It is therefore worth keeping an eye on your settings once the opt-out becomes available.