AI peers that pushed back led to slightly better business plans

Can you get better at running a business by practicing with an AI that disagrees with you? Researchers at Google have explored the question in a trial run in summer entrepreneurship programs for children and teens. The results are preliminary and come from a preprint, a research paper that does not yet appear to have been peer reviewed.

Teammates, not helpers

The trial took place in NFTE's BizCamp summer programs at nine sites, but only five of them randomly assigned students to groups. Students were split at random, roughly 65 to 35, between a treatment group and a control group.

The treatment group completed six tasks in Vantage, an experimental tool from Google Research. In it, students work alongside AI peers in realistic business scenarios. The peers are designed to act as teammates rather than assistants: They bring their own ideas, ask probing questions, disagree and sometimes put forward flawed ideas and incorrect claims. They never solve the task for the student, however.

The control group completed the same six tasks alone on worksheets, but with access to ordinary AI tools.

Seven percentage points higher

The main measure was the students' final business plan, which teachers or judges scored and converted to a percentage of the possible points. The analysis included 52 students from four sites: 36 in the treatment group and 16 in the control group.

The treatment group averaged 67.6 percent, compared with 60.5 percent in the control group. That is 7.1 percentage points more. The treatment group scored above the control group at all four sites, but at only one site was the difference statistically significant on its own.

Among the students in the treatment group, completing more tasks was associated with a higher score. Students who completed five to six tasks averaged 79.7 percent. Those who completed none averaged 59.3 percent. That is only a correlation, though, because students were not randomly assigned to how many tasks they completed.

In a survey of 51 users, 68.6 percent found the feedback helpful, and 74.5 percent saw the tool as useful for their course.

Major caveats

The sample is very small, and the p-value of 0.047 sits just below the usual threshold of 0.05. That means the result is only barely statistically significant. The authors themselves point to small groups at each site, a short intervention lasting a few weeks and limited measurement of lasting skills. Pre- and post-tests in Vantage could not be analyzed at all because of logistical and technical problems.

The difference could also stem from the entire Vantage setup, with its structure, feedback and AI peers, and not only from the peers disagreeing. It is not stated whether the judges knew which group the students belonged to. And Google's own research division both developed the tool and is behind the study.

The authors conclude that the preliminary results suggest AI peers can support performance in the short term on tasks that require complex and lasting skills.