Does AI Know More Than Your Doctor?

I have a feeling I am going to be talking about AI and medical advice a lot over the next few years. There is real danger here, and I want to start with a gentle reminder about communication in particular.

The smartest advice is often the least dogmatic. Intelligence about an issue recognizes the complexity and nuance. That is why, when talking to experts, you often hear the words “maybe” and “probably” rather than “for sure” and “certainly.” 

The problem is that this kind of nuanced conversation is not very satisfying, especially for people who are sick. They don’t want guesses; they want answers.

And as it turns out, AI is very good at giving answers on any topic, even though it is not necessarily an expert.

A recent study from MIT reveals a troubling pattern in how people evaluate medical advice: when the source is hidden, patients consistently rate AI-generated health guidance as more trustworthy and satisfying than advice written by physicians, even when the AI’s answers contain clear errors.

Published in NEJM AI, the study presented 300 participants with responses to common medical questions from three sources: physician-written answers from an online healthcare platform, high-accuracy AI responses that had been verified by doctors, and low-accuracy AI responses containing factual mistakes or inappropriate recommendations. Participants were not told which source produced which answer.

The results were striking. AI-generated responses scored significantly higher than physician responses across every measure of trust, validity, and completeness. When the AI was accurate, this preference was understandable. The more concerning finding was that even the flawed AI responses performed on par with, and in some cases outperformed, the doctors’ answers in perceived trustworthiness. Participants indicated they would follow the incorrect AI guidance at rates comparable to physician recommendations.

The researchers attribute this effect largely to how AI communicates. Large language models produce polished, clearly structured explanations that avoid excessive medical jargon. This fluency creates an impression of authority and thoroughness. However, without medical training, most readers cannot detect when AI fabricates details, oversimplifies complex conditions, or offers unsafe recommendations. The confidence of the delivery masks the inaccuracy of the content.

The implications for patient safety are significant. Researchers warned that participants showed a tendency to pursue unnecessary care or unsafe treatments based on erroneous AI output, which could lead to delayed diagnoses, medication misuse, or harm from self-treatment.

Speaking for myself, I find all this a bit troubling.

Here’s a bit of advice. Go to your favorite LLM to get help on your medical condition, but hold its opinions very loosely until you get confirmation from real experts.

Photo by Cash Macanaya on Unsplash