There is a simple test I use when looking at AI in a healthcare workflow.
If the output cannot be checked, it cannot be trusted.
That sounds obvious. In practice, it is where many AI implementations fall apart.
A healthcare organisation buys an AI tool because it promises to save time. The tool drafts messages, answers questions, summarises notes, processes requests or routes patients. On paper, the workflow becomes faster. In the demo, the answer appears instantly.
Then the real clinic uses it.
Someone has to check whether the patient is the right patient. Someone has to check whether the message is clinically appropriate. Someone has to check whether the AI missed context from another system. Someone has to check whether the patient should have been escalated. Someone has to check whether the summary is accurate enough to be entered into the record.
The time did not disappear.
It moved.
This is the verification tax: the downstream cost of checking an AI system whose output is not reliably bounded, explainable or connected to the right context.
If the verification tax is higher than the time saved, the organisation has not automated work. It has redistributed work to a more stressful part of the workflow.
AI Is Good at Known Problems
AI can be extremely useful in healthcare operations.
It can help with repetitive tasks. It can summarise large amounts of text. It can extract useful information from messy records. It can draft a first response. It can help teams prioritise inbound work. It can reduce the administrative burden that sits between the patient and the clinician.
But AI is best when the problem is already understood.
This is the part that often gets missed. AI is not a substitute for knowing what you are trying to solve. If a clinic has not defined the workflow, the success criteria, the escalation rules and the source of truth, AI does not magically create that structure.
It guesses.
Sometimes the guess is useful. Sometimes it is wrong. The difficult part is knowing which is which.
That is why the first question in AI procurement should not be "what can the model do?" It should be "do we understand the process well enough to know when the model is wrong?"
If the answer is no, the organisation is not ready to automate that process.
It may be ready to study it. It may be ready to map it. It may be ready to use AI in a narrow assistive role. But it is not ready to hand over responsibility.
Undefined Workflows Create Expensive AI
In engineering, short feedback loops expose wrong assumptions quickly. You build, test, learn and correct.
In healthcare operations, feedback loops can be much slower. A clinic might implement a new system over months. A health system might run a transformation programme over a year. By the time the organisation realises it automated the wrong thing, the budget is gone, the team is tired, and the old workaround is still sitting quietly in a spreadsheet.
This is one of the hidden costs of AI hype.
People apply AI to problems they have not solved even once manually. They skip the boring process work because the technology looks powerful enough to absorb the mess.
It usually cannot.
A messy recall process does not become safe because an AI can draft messages. A fragmented patient record does not become unified because an AI can summarise one part of it. A poorly defined triage pathway does not become clinically robust because an AI sounds confident.
The system may look more advanced, but the underlying uncertainty remains.
Now the team has two jobs: do the work and supervise the tool that was supposed to do the work.
That is the verification tax again.
Human-in-the-Loop Is Architecture, Not PR
Many AI products describe themselves as human-in-the-loop.
The phrase can mean almost anything.
Sometimes it means a human approves every output before it reaches a patient. Sometimes it means a human is available if the system escalates. Sometimes it means a human can review logs after the fact. Sometimes it means there is technically a person somewhere in the organisation, but the real workflow is effectively autonomous.
Those are very different safety models.
In healthcare, human-in-the-loop needs to be designed, not claimed.
Who is the human?
What are they checking?
What context do they see?
How much time do they have?
What are they accountable for?
What does the system do when the human does not respond?
What decisions can the AI make without them?
If those questions are unanswered, the human loop may be cosmetic.
This matters because healthcare is inherently unpredictable. Patients arrive when something is not working as expected. They present incomplete information. They use the wrong words. They ask administrative questions that are actually clinical questions. They sound calm when they are distressed.
AI can handle predictable, repeatable work. It can support humans with unpredictable work. But it should not pretend the unpredictable has gone away.
The human layer is what catches the unpredictable.
That makes it a core part of the architecture.
The Difference Between Drafting and Deciding
One practical way to reduce risk is to distinguish between drafting and deciding.
Drafting is often a good use of AI. The system prepares a response, summarises a record, suggests a next step or highlights missing information. A human reviews, edits and approves.
Deciding is different. The system selects the action, sends the message, changes the booking, closes the loop or tells the patient what to do.
The jump from drafting to deciding is where many risks live.
In a well-designed system, that jump is explicit. The organisation knows which tasks can be fully automated, which require approval, and which should never be automated. The boundary is based on risk, data quality and workflow clarity, not on how impressive the model is.
For example, answering "what time do you close?" is low risk. Drafting a follow-up to a patient who cancelled after treatment is higher risk but potentially safe with review. Advising a patient on symptoms is a different category altogether.
The system should not treat those as variations of the same task.
Patients may experience all three as "asking the clinic a question." Technically, they require different controls.
Context Is the Safety Layer
A lot of AI risk in healthcare is really context risk.
The model may be capable. The prompt may be well-written. The interface may be polished. But the system still needs the right context at the right moment.
In private clinics, context is often fragmented across:
- The practice management system
- The inbox
- SMS and WhatsApp conversations
- Phone notes
- Payment history
- Marketing source data
- Free-text clinical notes
- The memory of the practitioner or receptionist
If an AI system acts on one of those sources without understanding the others, it may produce an answer that is technically plausible and operationally wrong.
This is why data integration is not back-office plumbing. It is a safety layer.
Patient communication is especially sensitive because the message is where patients feel whether the clinic understands them. If the system sends a generic message at the wrong time, the failure is not only technical. It affects trust.
Patients can tell when no one is paying attention.
The best AI systems should make the clinic feel more attentive, not less.
The Legal Weight of Non-Determinism
Traditional software is deterministic. Given the same input, it usually produces the same output.
AI systems do not behave in quite the same way.
That is not automatically a problem. It is part of what makes them useful. They can interpret messy input, generate language and adapt to varied situations.
But in healthcare, non-determinism has legal and operational weight.
If the system can produce different outputs, the organisation needs stronger controls around what it can see, what it can do, how outputs are verified, and how decisions are recorded. Otherwise, the clinic is left with a difficult question after something goes wrong: why did the system say that?
An answer like "because the model generated it" is not enough.
This is where auditability becomes central. The organisation needs to know what information was used, what recommendation was made, who approved it, what was sent, and how the patient responded.
Without that, the verification tax becomes a risk tax.
How to Buy AI More Safely
Healthcare buyers do not need to become machine-learning researchers.
They do need to ask better implementation questions.
Start with the problem, not the product.
Can we describe the workflow on paper?
Do we know what good looks like?
Do we know what bad looks like?
Do we know which data source is authoritative?
Do we know who owns the outcome?
Then ask what the AI is allowed to do.
Is it reading, drafting, recommending, deciding or acting?
What happens when confidence is low?
What happens when data is missing?
What happens when two systems disagree?
What happens when the patient asks something outside scope?
Then ask how the work is checked.
Who reviews outputs?
How are reviewers trained?
How much time does review take?
Can reviewers see enough context to make a good decision?
Can the organisation audit the workflow later?
These questions are not anti-innovation. They are what separates useful AI from expensive theatre.
The Better Use of AI
The most valuable AI in healthcare may be the AI the patient never notices.
The system that reconciles fragmented records before a coordinator replies. The tool that finds patients who need follow-up. The assistant that summarises the previous conversation so the human starts with context. The prioritisation layer that helps the team respond to the right patients first. The guardrail that catches missing information before a message is sent.
That kind of AI does not replace care.
It makes care easier to deliver.
It also creates better jobs. If low-value administrative work is removed, teams can spend more time on patient experience, operational improvement and the conversations that actually require judgement.
That is the direction healthcare should be moving.
Not AI as a thin layer of automation over undefined work.
AI as infrastructure for better human decisions.
A Simple Rule
If an AI tool cannot be checked, it should not be trusted.
If checking it takes longer than doing the work, it has not saved time.
If the person checking it does not have the context to know whether it is wrong, the safety model is incomplete.
And if the organisation cannot explain what happened afterwards, the system is not ready for healthcare.
AI will absolutely change how clinics operate. It will automate repetitive work, reveal patterns in messy data and help teams respond faster. The opportunity is real.
But the opportunity is not unlocked by pretending verification is unnecessary.
It is unlocked by designing verification properly.
Clean workflow. Clean data. Clear boundaries. Human supervision. Audit trail.
That is not bureaucracy.
That is what makes the automation worth trusting.
