OpenAI releases 722 AI-written math papers, but far from all are checked

OpenAI has published 722 mathematical manuscripts written by an internal, as yet unreleased AI model. They are freely available on GitHub, a site where people share code and documents. OpenAI says the collection contains solutions to hundreds of open problems. These are questions that mathematicians have not yet been able to answer.

What has been released?

The manuscripts are sorted into 372 families within pure mathematics, theoretical computer science and mathematical physics. According to coverage on Interesting Engineering, around 4,000 problems were attempted in total, and the published ones are a selection from those. Each result cost on average roughly as much computing power as three hours of thinking in ChatGPT Pro.

The model has no public name and cannot yet be used by ordinary users.

How reliable are the results?

Here it is important to keep a cool head. Many of the manuscripts have a so-called Lean version. Lean is a programming language in which a computer can check that every single step of a mathematical proof follows logically from the previous one. It is a kind of automatic fact-check of mathematics.

Only about one in five manuscripts has such a computer-checked proof, according to a review of the release. OpenAI itself writes that the others may contain errors. And even Lean verification does not automatically prove that the entire claim is correct, because the problem also has to be translated correctly into Lean.

OpenAI let an independent advisory group on mathematics and AI at the Institute for Advanced Study follow along. The company expects feedback from mathematicians to influence future releases.

Background

The release follows a series of similar announcements from OpenAI this year. In May, an internal model presented a counterexample to a problem about distances between points posed by Paul Erdős in 1946. In August came ten new mathematical advances from the model Astra. The mathematician Timothy Gowers has said that the May result could have been published in one of the field's finest journals.

What does it mean for you?

It does not mean anything in your everyday life right now. But it is a sign that AI is starting to contribute to research, not just to writing emails and summarizing texts. The volume is now so large that human mathematicians will need a long time to determine how many of the 722 results actually hold up.