Social media claim says medical accuracy of LLMs drops when the patient communicates
A social media account describes a randomized study reporting a 60% drop in diagnostic accuracy of LLMs and a 12% drop in appropriate management decisions when the patient, not the physician, was the one communicating.
Source: Post claims LLM medical accuracy drops when patients, not physicians, communicate (x.com). Summary written with AI from the source and reviewed by rules; read the original.
What the research shows
According to a social media account, a randomized study reportedly found that large language models (LLMs, AI systems that generate text, such as chatbots) had a 60% drop in diagnostic accuracy and a 12% drop in the share of appropriate management decisions when the patient, not the physician, was the one communicating. The account suggests that chatbots' medical answers may depend on the user's existing knowledge, which would make those answers less reliable for lay people.
Because this finding reaches readers through a social media account rather than a published report, it should be treated as an unconfirmed claim until other researchers have examined the original study. Research on AI in medicine generally needs careful reading, since performance can vary with how a case is presented and with the type of task. Still, the question raised is relevant: someone using an AI tool to understand symptoms may get different answers depending on how they describe the problem.
What it means
If confirmed, the finding suggests that chatbots may give less reliable medical information to lay people, especially when the person describing the case lacks technical knowledge. It is not yet known whether the result holds in other studies and other kinds of cases.
How to read it with care
Treat this claim as unconfirmed and do not use a chatbot alone to decide on diagnoses or treatments. Any question about health, changes to diet, supplements or treatment should be discussed with a doctor or another qualified professional.