13.5%
of chatbot answers judged unsafe
What the research found
In a physician-led red-teaming study, researchers evaluated 888 responses from Claude, Gemini, GPT-4o and Llama to 222 real patient-posed primary-care questions. Problematic responses ranged from 21.6% (Claude) to 43.2% (Llama), and outright unsafe answers from 5% (Claude) to 13.5% (GPT-4o), several with the potential to cause serious harm.
Why it matters for providers
When patients put real medical questions to general chatbots — exactly what is happening now — a meaningful share of answers are unsafe, and the safest model was more than twice as safe as the worst.
Original source
npj Digital Medicine (Nature)
Peer-reviewed or official source — the most reliable tier.