AI safety should be built like nuclear power plants, critics say

Nuclear power plants are built so that a single failure can never lead to an accident. Now the demand is that advanced AI be secured in the same way. The debate is about how well that succeeds today.

What are safety layers?

The idea is known in technical terms as "defense in depth". The principle is to have multiple, redundant and mutually independent safety mechanisms. If one layer fails, the next one steps in.

The most important word is independent. Two locks that can be opened with the same key do not give double security.

The companies say it, but do not fully do it

Anthropic, Microsoft and OpenAI have all published safety documents in which they say they will use defense in depth on their most advanced AI systems.

A research analysis by two German researchers, including Leonard Dung of Ruhr-Universität Bochum, reaches a different conclusion, however. They argue that the common safety methods do not live up to the principle. The methods are not independent, and in certain scenarios that are relatively easy to foresee, they can all fail at the same time.

According to the discussion of the analysis, Dung says that the basic way of thinking about risk that one would expect from these companies has not been adopted.

Experience from nuclear power

Among others, the debate highlights Heidy Khlaaf, who was previously a senior systems safety engineer at OpenAI. She also has experience from safety audits of nuclear power plants and is today chief AI economist at the AI Now Institute. It has not been possible to confirm her wording in the specific article the headline is based on.

What does it mean for you?

AI is used today for everything from customer service to health advice and code. As the systems become more powerful, the question of what happens when the safety features fail becomes more pressing. The critics are not saying AI is dangerous in everyday life. They are saying that the protections described today as multiple layers can in practice resemble a single layer if they fail for the same reason.

There are also counterarguments. One commentator in the discussion points out that a very intelligent adversary can trigger failures that are normally unrelated at the same time. That is a problem nuclear power plants do not have in the same way.