The fastest way to kill AI in healthcare for a generation is to let it break in a way that makes a great headline, and a much worse hearing.
At Coherent, we are hands on in AI, fintech, and healthcare every day. It's what we do for a living, so it costs me something to say that out loud. But it is the conclusion I keep arriving at, and the rest of this piece is me trying to justify it before the evidence does the job for me.
Let me start with the question I would put to every founder and investor pouring money into this space right now, including the ones who will roll their eyes at this: Would you let today's AI treat your own child, unsupervised, with no clinician in the room?
Would you let it respond to a patient who complains about persistent side effects from a recent surgery?
Not draft a reminder text. Diagnose. Decide.
Sit with the hesitation.
That hesitation is the whole argument.
We have run this experiment before, on a different organ. When The Social Dilemma landed, its most quoted revelation was not a statistic but a behavior: the Silicon Valley engineers who had built the most addictive products on earth were quietly keeping their own children away from screens.
They knew something the rest of us were still finding out. By the time we understood what a decade of engagement-optimised feeds had done to a generation of teenagers, the damage was done. The charts went vertical.
Social media promised to connect the world, and it did. It also lit a global mental-health crisis under a generation of young people who never consented to being the variable group in the study.
We have seen this film: innovation, excitement, disruption, dissolution, correction. Roll credits. Greenlight the sequel.
Healthcare is the sequel.
Same urn, new mechanism
I want to be careful here, because the lazy move is to gesture at social media, gesture at AI, and mutter "second verse, same as the first." It is not the same. The mechanism has changed, and the change is the scary part.
Social media captured your attention. You can claw back attention by putting the phone in a drawer. AI captures your decisions: what gets approved, who gets flagged, which message you see, and which appointment you are offered. A decision made on your behalf, at scale, by a system you cannot interrogate is much harder to put in a drawer, especially when the people deploying it have been trained to treat its outputs as free, true, and basically magic.
So it used to have your eyes.
Now it has your choices.
There is a thought I cannot shake from the philosopher Nick Bostrom. He pictures human invention as a hand reaching into a giant urn. Each ball is an idea, a discovery, a technology. Most come out white: penicillin, the printing press, indoor plumbing.
Some come out grey, useful and harmful at once. Across thousands of years we have pulled out an astonishing number, and, on balance, it has gone alright.
We - people - are still here.
What we have never pulled out is a black ball: a technology that, by default, destroys the civilization that invents it. Bostrom's unsettling point is that this is not because we are wise enough to avoid one. It is simply that we've been lucky. So far.
I have no idea whether AI is a black ball. At some point we will find out. I would prefer we not find out in a clinical setting.
The part nobody is paid to tell you
Every incentive in this system points one way: tell the upside. Founders are paid to tell it. Investors are paid to fund it. The press cycle rewards it.
There is an enormous, well-capitalised chorus singing that AI will democratise medicine, catch the cancers we miss and put a doctor in everyone's pocket. A lot of that is real, which is incredibly exciting.
I am not anti-AI, and I am not anti-AI-in-healthcare. Used well, we can do better for patients and providers. Say it twice, because it is true.
The other half is less headline-worthy. So I'll offer it up from a vantage point that should make me shut up and sell.
Before Coherent, I was involved in developing a cutting-edge laser for skincare and, later, ran day-to-day clinic operations. I stood at the front desk and watched patients come and go. Early in the Coherent journey, I worked the front desk at one of our clinics - uniform and all.
So I've seen a few things that do not make it into the pitch decks. I know that "clear" and "proven" are not the same thing. And I know that the failures here will not arrive with sirens. They will hum.
Three quiet catastrophes
Allow me to ruin your afternoon with three of them. None is dramatic. Each is plausible, individually forgivable, and deeply problematic at scale.
A scheduling agent learns from the data that certain patients no-show more often, and quietly starts deprioritising them. Nobody wrote a line of code that said discriminate against these people. It was inferred. Now the patients who most need care are last to the party, and the dashboard glows a beautiful, reassuring green, because utilization is up and the KPIs are ablaze.
A triage agent takes a call it does not classify as a crisis, because the caller did not use the magic words. It will tell them so - with total confidence and immaculate grammar. The work scales infinitely. The verification does not.
And the human-in-the-loop, our great reassurance, quietly stops being a check and becomes a rubber stamp, because the machine is right enough of the time and clicking approve is faster than reading. The rest of the time is where the harm happens, and the human has been gently trained not to look.
If social media's variable group was a generation of teenagers, healthcare's variable group is whoever happens to be sick this Tuesday.
Two-tier medicine, and the day no one is the doctor
Two consequences of this are already taking shape, and both deserve more alarm than they are getting.
The first is two-tier medicine. The safe version of AI, the one with a human clinician beside it to catch its mistakes, is becoming a premium product.
The version without the human becomes "accessible." We told ourselves AI would level the playing field. Are we building a system where the people with the least support get the version with the most risk?
The second is accountability. When a clinician errs, we know who is responsible. When the algorithm is part of the decision, responsibility scatters… the developer, the clinic, the supervising clinician, the regulator, until it lands on no one.
Where does this end up?
Why I am terrified from the boring end of the building
Here is my confession. At Coherent we work in the least healthcare-y corner of healthcare: non-clinical operations. The unglamorous plumbing. Booking an appointment, routing a message, getting the right person to the right desk. It is admin in a lanyard. No diagnosis, no prescription, no scalpel.
And I am still terrified, precisely because of where I sit. From the boring edge, you can see the patients coming and going.
I should also admit why I am bullish anyway, because so far this reads like a person backing out of his own industry.
The actual prize is enormous, and I believe it is hiding in the boring parts.
Healthcare is drowning. Patients cannot get appointments, cannot get answers, cannot navigate a system that appears to have been designed by a committee that resents them personally.
A huge share of that misery is not medicine. It is logistics: forms, phone trees, authorizations, the seventeen-leg relay race between deciding you need care and actually receiving it.
That is precisely the tedious, high-volume, rules-shaped work that machines are genuinely, unglamorously brilliant at. Done right, we can hand healthcare its time back and give patients a system that actually replies.
That is the white ball, and it can be real. We should reach for it.
The answer to "move fast and break things" is not "move slow and break nothing," because moving slowly in healthcare breaks things too, namely the people still waiting. The answer is to move fast where it is safe, carefully where it is not, and to actually know the difference.
That is the entire argument. Everything else is implementation.
Borrowed from every grown-up industry
We do not need to invent the safeguards. We can borrow them wholesale from every other industry that ships dangerous things for a living.
- Risk-tiering. The model that drafts a reminder text is not the model that decides who gets seen first. Tier by consequence and manage tiers differently. We already do this for drugs, aircraft and lawnmowers. We can manage it for software.
- A human-in-the-loop that is actually a loop. A person who cannot realistically override the machine is set dressing. If you put someone in the loop, give them the time, the information and the authority to say no.
- Auditability. If you cannot reconstruct why the system did what it did, you have an oracle, not a tool. Healthcare does not get to run on oracles. Every consequential decision needs a trail.
- Reversibility. Build the undo button before you build the feature. The social-media generation is proving very hard to un-break. Make sure your worst case is "we caught it and rolled it back," not "we will study the cohort for twenty years and write a moving longread."
I have failed at many hard things before. I once spent a year trying to launch Europe's first all-electric airline. We did not make it; it turns out you cannot simply put hydrogen-electric planes thirty thousand feet in the air by wanting it badly enough. But that failure was contained. Nobody was on board. In healthcare, everyone is on board.
"Easy for tech to say"
Let me make the obvious objection on your behalf, so the comments do not have to. Of course, the health-tech guy wants rules. Nothing entrenches an incumbent like a four-hundred-page compliance regime only the incumbent can afford to read.
But we are not the incumbent. We are the challenger, and challengers do not usually argue for friction. I am arguing for it anyway. Not for more rules. For different decisions by the people touching this thing first-hand. Top-line growth matters a lot. But the real bottom line here is a person. A patient.
It is comfortable to make governance somebody else's job: the slow committee you can blame in the post-mortem. But the regulator shows up after the train is off the tracks. The driver is there before. Let's do the unglamorous work during the joyride, not after the funeral.
Build the carefulness on the way in, while we still can, as the actual product and not a compliance tax bolted on at the end. This is not anti-innovation. It is the most pro-innovation idea we have got, because the single fastest way to kill AI in healthcare for a generation is to let it break someone in a way that makes a great headline and a much worse hearing. Move fast and break patients, and you do not just hurt patients. You turn the entire field into quicksand.
