Two billion dollars to make AI oversight more independent

Anthropic, the company behind the AI chatbot Claude, has entered into a partnership with the consulting firm Accenture. Together, they will each invest at least $1 billion, a combined total of around $2 billion (more than DKK 13 billion), over the next five years. The money will be used to have independent experts examine whether Anthropic's most advanced AI models are safe to use.

The agreement was announced on September 18. The practical work lies with Faculty, Accenture's artificial intelligence unit, which became part of the company earlier this year. Among other things, Faculty will do so-called red-teaming, meaning trying to make the AI models fail or behave in unwanted ways in order to find weaknesses before others do. They will also assess whether the models behave as the developers intend (known in the industry as alignment), and test the safeguards Anthropic has built in.

Expert access like employees

What's new about the agreement is that external reviewers from Accenture get access equivalent to being employed at Anthropic. They can follow the models while they are being trained, see the decisions that shape them along the way, and talk directly with Anthropic's staff. Normally, outsiders test only the finished models, just before they are launched.

The agreement is the first concrete step in something Anthropic's CEO Dario Amodei proposed in early September. In an essay, he wrote that AI development should be slowed down, among other things by giving independent reviewers permanent access to the companies as if they were employees. Anthropic has now committed to this itself and is urging authorities to require the same of other major AI companies.

The partnership is not exclusive. Anthropic is also in talks with other organizations, including the independent nonprofit METR, about similar oversight, and Accenture is free to work with other AI companies on the side.

Criticism of the independence

But the arrangement is already facing criticism. Anthropic itself pays for Accenture's work, and the company itself writes that this is not the right long-term solution, without pointing to a better alternative. Nor are there any common rules for what the reviewers should have access to or how they should report what they find.

Critics also point out that Accenture and Anthropic already have close business ties. Accenture has pledged to train around 30,000 of its own employees to use Claude, which according to Amodei is the largest single deal of its kind Anthropic has ever made. This leads some to ask whether a consulting firm with such financial interests would really halt a launch if a test revealed problems. Anthropic rejects the criticism and writes that the reviewers "don't reduce our accountability, but help make it more verifiable."

For Danes who use Claude or similar tools in everyday life, the news is interesting because it shows a trend: AI companies are starting to submit to ongoing oversight, not just right before a model is launched. That could give users better insight into how safe the models actually are. But as long as the same party pays and is being checked, it remains unclear how independent the oversight really is.