Chatbots answer questions about male infertility better, but not all of them

How good are free AI chatbots at answering questions about male infertility, and do they get better over time? Four Turkish researchers examined this in a new peer-reviewed study in the journal Basic and Clinical Andrology.

The researchers asked the same 88 English-language questions in May 2025 and in May 2026. The questions were derived from recommendations in the European Association of Urology (EAU) guidelines on male infertility. In total, 528 answers from ChatGPT, Perplexity and DeepSeek were evaluated, all from the free web versions.

How the answers were measured

Two urologists with experience in the field rated the answers without knowing which chatbot had written them. The answers received points for quality and an expert score from 1 to 5. Readability was also measured with the Gunning Fog index, where a lower number means the text is easier to read.

Who improved?

In 2025, Perplexity performed best with an expert score of 4.5, followed by ChatGPT with 4 and DeepSeek with 3.

In 2026, the picture had flipped. ChatGPT improved on all four measures and received an expert score of 5. DeepSeek made the biggest gain, going from 3 to 5 in expert score. Perplexity, by contrast, slipped from 4.5 to 4 in both expert score and overall quality. The authors write that in 2026 Perplexity gave broader and at times more generic answers, even though it still cited sources.

Perplexity's texts were the hardest to read in both years. All three chatbots, however, were at a level that is likely to be demanding for most readers.

Safety

In 2025, the experts found possible patient safety problems in 7 of 88 answers from DeepSeek, or about 8 percent. One example was unclear prioritization of methods for measuring testicular volume. Such problems were not found in 2026. None of the 528 answers contained a recommendation that was judged likely to cause immediate or serious harm.

Caveats

The authors themselves were responsible for the questions, scoring and expert assessment, and the study has not been replicated by others. Only one answer per question was examined, so it is not known how stable the answers are. The questions closely followed the wording of the guidelines and were in English only, so the results say less about general, vague questions from ordinary users. The providers may also have changed the models along the way. In addition, the table and the text of the article disagree on whether DeepSeek's overall quality score changed.

The authors conclude that development differed from model to model, and that well-phrased answers and source citations do not in themselves make an answer clinically safe. Verification against authoritative sources and human oversight remain necessary.