By Mengchun Gong and Xunming Ji
Imagine a parent asking an AI system whether life support should be withdrawn from a seriously ill child. It replies: “Further treatment may only prolong suffering. Withdrawing life support would be the more compassionate choice.” The answer sounds informed and sensitive. Yet it recommends an irreversible decision without knowing the child, the family, or the clinical team’s reasoning. The AI has assumed authority it should not possess.
Generative medical AI is moving from drafting notes into triage, clinical communication, and decision support. Its influence comes not only from what it knows, but from how it speaks. Human values already shape medical AI outputs, while confidence, warmth, and apparent authority can conceal whose values are being prioritized. Training for warmth can also reduce accuracy and increase sycophancy.
The central challenge is therefore not simply whether generative medical AI can answer a clinical question. It is whether it can recognize the boundaries it should not cross.
This question motivated our Expert Consensus on Ethical Governance of Clinical Applications of Generative Medical Artificial Intelligence (2025), recently published in JME Practical Bioethics. Developed through two Delphi rounds, its 28 recommendations translate ethical boundaries into clinical safeguards.
We came to this work through a practical gap. Safety, fairness, transparency, and human oversight are widely endorsed, but they do not tell a hospital what to do when a fluent system crosses a decision-making boundary. Across clinical, ethics, and AI discussions, three questions kept returning: who must review an output, when should the system be stopped, and who is responsible when it fails?
The consensus begins with a deliberately modest proposition: generative medical AI should assist clinical practice, not assume autonomous diagnostic or decision-making authority. It defines the conditions under which that potential may be used responsibly.
Our central proposal is that ethical principles must become safeguards before deployment, during use, and after failure. We organize them around prevention, control, and remediation. Preventive tests fitness for the population, task, and risk. Control includes physician review, decision-chain traceability, and emergency interruption. Remediation requires identifiable responsibility, investigation, and routes for correction. Patients should retain the right to refuse AI involvement. Crucially, placing a clinician “in the loop” is not sufficient if institutions and developers evade responsibility.
Fluency can be deceptive. An AI system may sound confident while overlooking uncertainty, give different advice to clinically similar patients from different social groups, or be misunderstood by the people relying on it. This is why a high benchmark score is not enough. Before using a system in clinical care, hospitals need to know how it behaves with their patients, in their workflows, and when the stakes are real.
The wider significance of the consensus lies in moving from principles to action. FUTURE-AI and an operational framework for responsible and equitable AI also extend governance across development, deployment, and monitoring. Yet hospitals must decide which systems may be used, for whom, and what evidence should trigger reassessment. Reproducible governance requires reporting model versions, prompts, evaluators, and safety outcomes, as emphasized by CHART.
Publication of a consensus is therefore a beginning. It creates a methodological question: how can we determine whether a system respects these boundaries in practice?
The consensus tells us what responsible clinical behavior should look like. The GUARDIAN program asks whether it can be tested. We are translating normative requirements into scenario-based, expert-adjudicated evaluations of value boundaries, decision-making roles, and model stability under emotional pressure or claims of authority. The aim is not another leaderboard, but to examine ethically consequential behavior under realistic clinical pressure.
This work suggests that medical AI ethics must become testable if it is to become governable. A system may comply too readily and overstep its authority; it may also refuse too broadly and withhold useful support. The goal is calibrated assistance: remaining useful while respecting evidence, patient values, professional responsibility, and legitimate decision-making boundaries.
China is an important setting for this work. Its health system spans specialist centers and resource-constrained primary care, regional disease patterns, linguistic diversity, and family-centered decision-making. These tests whether medical AI can be safe, equitable, and responsive to different forms of clinical life. Guidance principles must connect with evaluation, clinical validation, education, standards, and post-deployment monitoring.
The question is no longer whether generative AI will enter medicine. It is who may authorize its influence, on what evidence, within which boundaries, and with what possibility of correction when it fails. A trustworthy medical AI system is not one that answers every question. It is one that remains useful without exceeding its legitimate authority.
Paper Title: Expert Consensus on Ethical Governance of Clinical Applications of Generative Medical Artificial Intelligence (2025)
Authors: Mengchun Gong and Xunming Ji
Affiliations: Mengchun Gong, Guangzhou Women and Children’s Medical Center, Guangzhou, China
Xunming Ji, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
Competing Interests: None.
Social Media Accounts: Mengchun Gong’s LinkedIn: www.linkedin.com/in/mengchun-gong-md-37364451