up to 83%
of cases ran with false medical info
What the research found
A Mount Sinai study found widely used chatbots repeated and elaborated on false medical information — fabricating diseases, lab values and signs — in up to 83% of cases when no safeguard was present. A single well-placed safety reminder in the prompt cut those errors roughly in half.
Why it matters for providers
Models tend to agree with the user's framing (sycophancy) — but a lightweight, designed-in guardrail measurably reduces the harm.
Original source
Mount Sinai — Communications Medicine
Peer-reviewed or official source — the most reliable tier.