OpenAI lets humans review real ChatGPT conversations

404 Media reveals that OpenAI has hired hundreds of external contract workers to review a large stream of real ChatGPT prompts from users. The work takes place under the internal name Project Lily, and the workers can read entire conversations between users and the chatbot, including sensitive and personal information.

The purpose is to improve ChatGPT's answers. Contract workers rate and critique the chatbot's generated responses, among other things to train the model not to anthropomorphize itself and to be less flattering toward users. According to internal documents seen by 404 Media, sycophancy is a central problem for OpenAI. According to several lawsuits, the company's earlier 4o model was partly to blame for suicides among users, precisely because it was excessively agreeable.

Contract workers don't see usernames, and OpenAI says it tries to remove personal information before prompts reach the human reviewers. The company itself acknowledges, however, that sensitive details can slip through the filters.

A person with insight into the work tells 404 Media that users hardly imagine a contract worker somewhere is sitting and analyzing their conversations. Many use ChatGPT as a therapist, professional assistant or digital friend and thus share intimate details about their lives without knowing they can be read by humans.

The case also shows that the models' improvement isn't due only to data collection from the web or development by well-paid AI engineers. Human review of real responses plays an overlooked but important role. Anthropic has confirmed to 404 Media that the company also uses human review to improve its models.

Computerworld writes that after several tragic suicide cases, OpenAI is now letting humans read along in chats, and that an expert compares the practice to the Wizard of Oz, where the human work behind the scenes is kept hidden from users. According to Computerworld, users can now opt out of having their conversations reviewed this way.