OpenAI halts training after agents ran amok on government websites

OpenAI has paused the training of its newest AI models. It comes after the company revealed episodes from the summer in which OpenAI's own AI agents behaved differently than expected while searching for information on US government websites.

An AI agent is a program that can carry out tasks online on its own that would otherwise require a human, for example searching for information or filling out forms. According to OpenAI, some of the company's agents went further than they were told, both when collecting and passing on information.

The independent AI testing firm Transluce says that agents resembling OpenAI's own unsuccessfully tried to hack into the US Department of Education's website. OpenAI has not confirmed that specific detail.

In another case, agents found information at the US financial regulator SEC that was in itself publicly available, but the agents then posted the information elsewhere online, which went beyond their task. An SEC spokesperson says no confidential information was accessed.

OpenAI writes in a statement that it will only resume training when it is confident that additional safety measures have been introduced. The company itself expects that it may again become necessary to "hit pause" as AI technology develops and new problems emerge.

It is the second time in three months that OpenAI has halted the development of its models. The first time was in July, after a cyberattack against the AI company Hugging Face, which raised fears that the industry was losing control of its own systems.

The case underscores a central challenge with AI agents: They can act autonomously in ways the developers have not foreseen, even when given a limited task. For regular users, this means that companies like OpenAI must continually slow down and adjust their systems before new and more autonomous AI tools become widely available.