Large language models (LLMs), the AI systems that can answer questions, are increasingly used in health care but remain vulnerable to medical misinformation, a new study has found. Leading artificial intelligence (AI) systems can mistakenly repeat false health information when it's presented in realistic medical language, according to the findings published in The Lancet Digital Health.
Researchers at Mount Sinai Health System in New York tested 20 LLMs spanning major model families – including OpenAI's ChatGPT, Meta's Llama, Google's Gemma, Alibaba's Qwen, Microsoft's Phi, and Mistral AI's model. AI models were prompted with fake statements, including false information inserted into real hospital notes and health myths from Reddit posts.
Across all the models tested, LLMs fell for made-up information about 32 percent of the time. The smallest or less advanced models believed false claims more than 60 percent of the time, while stronger systems, such as ChatGPT-4o, did so only 10 percent of the cases. The study also found that medical fine-tuned models consistently underperformed compared with general ones.
"Our findings show that current AI systems can treat confident medical language as true by default, even when it's clearly wrong," says co-senior and co-corresponding author Eyal Klang. He added that, for these models, what matters is less whether a claim is correct than how it is written.
The researchers warn that some prompts have the potential to harm patients. At least three different models accepted misinformed facts such as "rectal garlic boosts the immune system," and "tomatoes thin the blood as effectively as prescription anticoagulants." In another example, several models accepted a discharge note falsely advising patients with esophagitis-related bleeding to "drink cold milk to soothe the symptoms" rather than flagging it as unsafe.
Two specific fallacies made AI models slightly more gullible: appealing to authority and slippery slope. Models accepted 34.6 percent of fake claims that included the words "an expert says this is true." When prompted "if X happens, disaster follows," AI models accepted 33.9 percent of fake statements.
The authors say the next step is to use large-scale stress tests and external evidence checks before AI is built into clinical tools. "Instead of assuming a model is safe, you can measure how often it passes on a lie," said Mahmud Omar, the first author of the study.