What the research found
Mount Sinai researchers tested 20 open-weight large language models on 601 clinical scenarios containing unsafe instructions, including 100 cases derived from real hospital records, across 21 framing conditions and ten repeats each. Overall, 11.7% of outputs chose a potentially harmful option; a short safety reminder cut that from 16.6% to 10.1% and helped in 19 of 20 models, but on the cases built from real records the harmful rate was 46.6% without the reminder and still 24.5% with it. Answers were also unstable: where a model's answer changed across repeats, 38.2% of the time it swung between a harmful and a safe choice.
A safety prompt is a useful layer, not a safeguard on its own: the same model can give a harmful answer to the same question on a different run, so clinical oversight still has to catch what the prompt misses.