Who Protects Patients When AI Gets It Wrong?

Jared usually talks to clinic operators, but this episode steps outside the provider world. Lucy Orr-Ewing is Director of Policy and Strategy at the Coalition for Health AI (CHAI), which by her account began with eight US health systems, including Stanford, Mayo Clinic and Duke, and now counts two to three hundred.

CHAI builds responsible AI frameworks, tools and playbooks, and Lucy leads a policy work group of around 150 people from health systems and technology suppliers. She also sits on the board of the Utah Sandbox and works between the US and the UK.

Lucy describes herself as a practical person rather than a clinician or a technologist. She read Classics, was chief of staff in the NHS, and grew up the youngest of four in a family of public servants and doctors.

Seeing the pressure on the NHS convinced her that AI is one of the few ways to keep up with demand, but over-reliance and under-reliance are both mistakes. In her words, she is not pro-AI in general: she is pro-AI that works.

So what does working mean? Lucy argues for a structure of trust rather than a single accuracy score. We trust a car's seatbelts, a light bulb in a newborn's room or sushi from a festival food van because a third party has checked them and there is somewhere to complain when things go wrong.

Healthcare already holds doctors to account through medical boards. She wants the same for AI: a minimum bar before it reaches the market, then layers of governance that catch a tool when it stops performing.

Jared, who once worked on laser medical devices, asks whether health AI risks becoming the next social media, which he calls one of the largest uncontrolled studies on young people ever undertaken. Lucy sees a difference.

The suppliers she works with, she says, pay for their own safety evaluations because providers demand them, and US liability rules (she cites OCR Section 1557) leave providers answerable for whatever they deploy. That has slowed adoption, but it has also pushed the industry towards governing before harm, not after.

On adoption, Lucy expects the US to move faster than the UK: a competitive private market, state legislatures passing hundreds of AI bills a year, and states such as Utah testing new regulatory structures. In the NHS, ambient voice tools were discussed for a decade before the public grew comfortable being recorded.

Jared adds what Coherent sees on the admin side: some patients ask not to talk to a robot, while others don't mind. Lucy's answer is to design for both, the way a hospital builds a ramp beside the stairs, and to consult patients first, as a recent UK consultation of 8,000 patients and 2,000 professionals did.

Her sushi analogy runs through the episode. ChatGPT in 2022 felt transformational at the time and would feel like bad sushi today, so expectations will keep rising.

A bad experience, such as a missed diagnosis or a patient left out by an error, can put someone off AI for good, and she already sees people in the UK removing their records from data platforms.

Looking ahead, she hopes healthcare becomes as user-centred as digital banking, with fewer scattered pilots and more platforms built on top of the patient record that are easier to govern and easier for patients to use.