By Hazem Zohny
A major worry with AI in healthcare is the ‘black box’ problem: deep learning AIs reach conclusions without explaining how. In healthcare, where trust is essential, this is a serious problem.
Recent AI developments challenge this worry. One example is Google’s Articulate Medical Intelligence Explorer (AMIE), an LLM-based system for clinical conversations and multi-visit care planning.
AMIE generates detailed clinical notes, differential diagnoses, and care plans in natural language, complete with step-by-step reasoning, justifications, and citations linked to clinical guidelines. These outputs are designed to be readable, verifiable, and useful for clinicians, and they build on prior patient interactions to support multi-visit care planning over time.
In some ways, this system exhibits more transparency than human clinicians. And that challenges how we think about trust in healthcare AI.
A new type transparency
There are different facets to AI transparency in healthcare, each relevant to different stakeholders—patients, clinicians, regulators, and AI developers. One way to break these down is:
- Mechanistic transparency: insight into how the system works under the hood—its architecture, training, and internal processing. (AI developers, regulators)
- Data transparency: what data the system was trained on, how that data is used, and how it might influence outputs. (Patients, clinicians, regulators, AI developers)
- Organisational transparency: how the system is deployed—who is responsible for it, how it fits into clinical workflows, and how it interacts with professional norms. (Patients, clinicians, regulators)
So-called ‘reasoning’ LLMs communicate the deliberations behind their eventual answer in what’s called their chain-of-thought. They are not mechanistically transparent, but they exhibit a new type of transparency we might call:
- Reasoning transparency: the ability to communicate in natural language the steps, justifications, and considerations behind a given output. [Clinicians, patients]
One way to think about this is by analogy to human decision-making: mechanistic transparency is like insight into the brain’s inner workings, while reasoning transparency is like access to the person’s verbal explanation of their thought process—even if that explanation is partly post-hoc.
Reasoning transparency opens the door to a new kind of interaction with AI systems, where their conclusions can be examined, contested, or adapted—much like a conversation with a human colleague.
AMIE’s architecture
AMIE is an example of this. It’s built around a dual architecture designed to make its “deliberations” visible and accompanied by justifications. It includes:
- A Dialogue Agent that interacts with patients, gathering information and building rapport.
- A Management Reasoning Agent that carries out the clinical reasoning, analyzing the patient’s situation and recommending next steps.
What sets AMIE apart are two reasoning transparency mechanisms:
First, it shows its reasoning process.
AMIE “thinks out loud.” It outlines its step-by-step analysis: concerns, factors weighed, and options considered. This reasoning trace is visible to clinicians, and potentially to patients, so they can follow how the system arrives at its conclusions.
Second, it justifies its recommendations with reference to clinical guidelines.
Each conclusion it reaches (ordering tests, making diagnoses etc.) is rooted in justifications that reference clinical sources, like BMJ Best Practice or NICE guidelines. For instance, when advising on suspected pheochromocytoma, it cited NICE guideline ng136 and BMJ Best Practice document bmj26. That means users can verify whether AMIE’s suggestions are consistent with accepted medical standards.
The first lets users see how the system weighs evidence and reaches conclusions via a higher-level, human-readable account, and the second allows for verifying that its recommendations align with established standards.
More transparent human clinicians?
Human clinicians can talk through their decisions, respond to questions, and build trust through conversation. But much of their judgment relies on tacit expertise, mental shortcuts, and fast, intuitive processes.
These forms of reasoning can be highly effective, but they’re hard to unpack, even for the clinician themselves. Like all humans, clinicians sometimes act on intuition or experience without full access to the reasoning behind their choices.
That’s why some argue that medicine has always involved a kind of “black box” reasoning. We don’t expect clinicians to have full access to their own tacit deliberations or how those link to clinical guidelines, let alone the cognitive or neural mechanisms behind those decisions. Instead, we trust their training, professional norms, and willingness to explain their thinking when prompted (which allows us to contest them).
What’s striking is not just that AMIE can offer explanations—but that it does so more systematically than most human clinicians: it lays out its reasoning consistently, links it to clinical guidelines, and builds a coherent narrative across visits. In some ways, it offers a more legible and auditable form of care.
And it does this to impressive effect: In a blinded study comparing AMIE’s care plans with those of human primary care physicians, reviewers found that AMIE’s treatment recommendations were more consistently aligned with clinical guidelines—ranging from 89% to 93% across three visits, compared to 75% to 81% for the physicians. AMIE’s recommendations were also judged to be more precise, both in treatments and investigations, and this advantage held across multiple patient visits.
Reviewers were also asked which plan they’d prefer for themselves or family. Again, without knowing who wrote the plan, they preferred AMIE’s in 42% of cases, the human doctor’s in only 8%, and had no preference in the rest.
The limits of ‘thinking out loud’
To be clear, AMIE doesn’t eliminate the black box problem. As with all deep learning systems, its inner workings remain opaque. But also, the ‘thinking out loud’ aspect of its reasoning transparency may be deceptive.
Recent work by Anthropic suggests that while the ‘thoughts’ of reasoning models appear clear and coherent, their final outputs may still be relying on unacknowledged cues or shortcuts.
In other words, AMIE’s deliberations may look like thoughtful analysis, but at least some of it may just be well-structured justifications generated after the fact. This arguably mirrors human post-hoc rationalisation—people often construct reasons for their decisions without full awareness of the real drivers, even if their intention is to be transparent about their deliberations.
Still, this does suggest we should be cautious about over-interpreting its step-by-step explanations. In that light, its ability to ground its decision points in clinical guidelines (which is ultimately what allows us to verify and contest it) may deserve more weight than the appearance of deliberative reasoning alone.
Even so, ‘thinking out loud’ appears to play a crucial role, as models that do so hallucinate far less in this context (0.7% vs. >5% in non-reasoning versions).
Other caveats and questions
- Simulated patients, not real ones: AMIE’s performance was evaluated in controlled studies with simulated patient interactions. How it behaves in real-world clinical settings remains to be seen.
- Guideline adherence isn’t always enough: Clinical guidelines are valuable, but not infallible. They evolve, can vary across contexts, and don’t always capture edge cases or systemic biases. Judging AI quality by adherence alone risks missing these subtleties.
- Is post-hoc reasoning enough? If AMIE’s step-by-step reasoning is partially confabulated rather than a reflection of how the reasoning process meaningfully caused the output, what kind of trust do systems like it warrant?
- Does fluency create false confidence? As systems become more articulate, do we risk mistaking coherence for competence—and trusting them more than we should? This reminds us that reasoning transparency is only one part of the trust equation.
Author: Hazem Zohny
Affiliations: Uehiro Oxford Institute, University of Oxford
Competing interests: None declared