When Clinical AI Explains Itself by Citing Similar Patients, Who Consented?

A record in a clinical training cohort is there to be learned from. Nothing in the PLM framework’s own description of its data use covers that record being retrieved later, by pseudonymous ID and coded chart content, as the reason a stranger’s prediction came out the way it did. That gap is now operational rather than hypothetical: a single-team, unreplicated preprint called Patients-like-me (PLM) demonstrates a working framework on MIMIC-III/IV that explains its predictions by citing similar patients from the cohort, and independent 2026 work on membership inference shows that explanation outputs are a real leakage channel. If your governance committee is being asked to approve “explainable” clinical AI, the explanation style is now a data-use decision, and nothing in the paper’s description of credentialed access to de-identified records addresses disclosure through the explanation layer.

The practical consequence, stated up front: ship feature-attribution or rule-trace explanations by default, and treat case-based reference-patient explanations as a distinct disclosure surface that needs its own consent coverage and its own privacy testing before deployment.

What a reference-patient explanation actually shows

PLM, described in its arXiv preprint, is a clinical prediction framework built on a language model combined with a graph neural network. Patients become nodes in a graph constructed from diagnosis-code overlap, and the model alternates between learning local patient semantics and global cohort structure. In the authors’ words, “PLM formulates a Variational Expectation-Maximization (VEM) algorithm that alternates LM-GNN optimization under a supervised variational objective, allowing local patient semantics and global cohort structure to iteratively refine one another with modest computational overhead.”

The explanation comes out of that machinery directly. Per the same preprint, “PLM also provides reference-patient explanations by retrieving influential similar patients, while edge-masking experiments confirm that the highest-ranked references have the greatest impact on model predictions.” Read that sentence as a governance officer, not as an ML engineer. The explanation surface is not a vector of feature weights. It is a ranked list of specific cohort members, presented as the justification for a prediction about someone else. The paper’s worked example shows what that contains: “an ego graph for a target diabetes patient 57139 with ten retrieved reference cases,” in which “Patient 305 with a direct diabetes CCS code has the highest Imp 0.41,” with annotations “extracted from the mapped CCS codes in each patient’s structured record.” These are pseudonymous IDs over coded concepts, not names; MIMIC’s release removes patient identifiers and shifts timestamps. The disclosure is still a specific person’s record content, and whether such records can be re-identified downstream is exactly the risk the membership-inference literature raises. Auditing this model and disclosing its cohort are the same act.

The authors are candid about the limits of the similarity measure itself: the patient graph rests on diagnosis-code overlap, which they note may miss richer clinical relationships available in multimodal EHR data or external knowledge. So the references are not even the clinically richest neighbors; they are the code-overlap neighbors the graph could see.

One scope detail changes whose records are at stake. PLM’s main experiments are transductive: patients are split into training, validation, and test cohorts, and each split gets its own graph, so a test patient’s retrieved references can come from the same test split rather than the training set. The training-cohort framing is exact for the inductive variant the authors also evaluate, “where each test patient independently connects to a fixed historical training cohort.” Either way, the explanation discloses specific cohort members’ records; which cohort depends on the deployment configuration, and a governance review should ask that question explicitly.

What edge-masking fidelity proves, and what it leaves untested

The edge-masking experiment is the right kind of faithfulness test: remove the edges connecting the query patient to its highest-ranked references, and the prediction moves the most. That establishes the explanation is load-bearing, not decorative. It tells you the retrieved patients genuinely drive the output.

It establishes nothing else. The authors say so themselves: “In addition, our reference-patient attributions are evaluated quantitatively for faithfulness, while further clinician assessment is needed before real-world deployment.” No clinician has judged whether these rationales are useful at the bedside. No attack has been run against PLM to measure what its explanations leak. No consent status for the cited reference patients is documented anywhere in the paper. And the entire evidence base is one team’s preprint, unreplicated as of October 2026.

This distinction matters because faithfulness is the property that makes the privacy question sharp. A faithful reference-patient explanation is, by construction, an accurate pointer to influential cohort members. The better the explanation works as an explanation, the more precisely it identifies who is behind it.

Explanations as leakage channels: the independent evidence

The privacy case against neighbor-citing rationales does not rest on PLM’s authors, and it should not, because they make no privacy claims either way. It rests on independent attack literature.

A 2026 paper on membership inference through explanations lays out the mechanism: “In particular, CFs can reveal how far an instance lies from the model’s decision boundary and can be exploited by the intuition that training samples are more confidently classified and thus are often located farther from the decision boundary than non-members.” Explanation objects carry signal about whether a record was in the training set. The same paper draws the regulatory consequence: “Such leakage significantly increases the risk that the model itself may be considered personal data under the GDPR, since it can enable the re-identification of individuals [9].” If the model’s outputs can re-identify individuals, the model stops being an anonymized artifact and becomes personal data, with everything that implies for access control and retention.

Work targeting clinical models specifically makes the harm concrete. From a study of membership inference vulnerabilities in clinical LLMs: “In the context of LLMs in healthcare, a successful attack could reveal that a patient’s record was used during model development, thereby exposing sensitive health information and undermining the trust needed for clinical AI adoption.” In a cohort defined by a condition, membership itself is the diagnosis.

Now the counterweight, which is real: measured leakage so far is small. That same clinical-LLM study reports “an AUC of 0.5397 and TPR@1%FPR = 1.59% (Table I, Figure 2)” for its paraphrased attack against Llemr, and concludes the results “demonstrate non-trivial yet bounded membership leakage,” with current medical language models showing partial resilience. An AUC of 0.5397 is barely above chance. So the honest position is neither “case-based explanations are proven dangerous” nor “the risk is theoretical.” The risk is structural: the explanation layer is a disclosure surface by design, the attack literature shows such surfaces leak, and nobody has yet measured how much this particular surface leaks. That is an argument for testing and governance gates, not for panic.

Three explanation styles, three disclosure surfaces

The choice facing a governance committee is not “explainable or not.” It is which of three disclosure surfaces you are willing to operate.

The third row comes from a governance framework for rule-based antibiotic prescribing support, which puts the contrast plainly: “Explainability: Every recommendation or abstention can be traced to explicit rule activation, whereas learned models may offer predictions without transparent decision pathways.” A rule trace discloses institutional knowledge, not people. The tradeoff is equally plain: that framework is a specification and evaluation construct with no clinical validation and no real patient data behind it, so it buys its clean disclosure surface by giving up the predictive flexibility that motivates learned models in the first place. PLM’s row buys predictive performance and faithful rationales at the cost of turning every explanation into a cohort disclosure. Attribution sits in between: weaker narrative value for clinicians, and the disclosed object is an aggregate weight rather than a neighbor’s coded record. That is a smaller surface, not a clean one. The explanations-leak paper’s related work notes that “back-propagation-based explanations strengthen MIA,” so attribution outputs warrant their own explanation-layer testing rather than a pass on input-side de-identification alone.

What current data-use practice covers, and what it leaves open

Real-world de-identification practice shows exactly where the gap lives. In the Penda Health AI Consult deployment study, “patient data was fully stripped of patient identifiers (e.g., patient names, date of birth, phone numbers, national ID numbers, medical record numbers, specific geography) as well as clinician identifiers,” and the data team “identified a very low rate of cases where possible patient identifiers were used (approximately 5 per 10,000 notes),” redacting what slipped through. That is a serious, audited pipeline, and it is aimed entirely at inputs: identifiers in clinical notes before they reach the model. It has no analogue for the case where the model’s output, functioning as designed, retrieves a neighbor patient as evidence. You cannot redact the explanation without destroying it, because the patient is the explanation.

The governance culture around MIMIC-based research is also already calibrated to non-deployment framing. A separate MIMIC-IV clinical QA study states flatly: “This work does NOT establish clinical deployment readiness. The framings below are governance scaffolding for any future deployment effort, not claims about the current paper. EpiKG is evaluated here as a research probe; no intended-use statement, no IRB-approved deployment protocol, and no prospective clinical study is in scope.” That is the standard of framing a committee should demand from any framework proposing to cite MIMIC patients back to clinicians, PLM included.

The governance checklist: deciding per method

For each explanation method under review, three checks follow from the evidence above:

1. What does the explanation surface disclose? Feature weights, retrieved records, or rule activations. If the answer is retrieved records, the explanation layer is a disclosure of cohort members and must be treated as one, regardless of how clean the input pipeline is.
2. What does the fidelity evidence establish? PLM’s edge-masking results establish faithfulness, meaning the disclosure is accurate, which sharpens rather than settles the privacy question. Faithfulness is not clinical utility (the authors defer clinician assessment) and not privacy safety (no attack has been run).
3. What does the data-use agreement cover? If the DUA and consent instruments address training inputs and prediction outputs but say nothing about explanation outputs, the method is not covered, full stop. Amend before approval, not after an incident.

On the current evidence, that checklist resolves to: default to attribution or rule-trace explanations; approve case-based reference-patient explanations only after the DUA is amended to cover explanation-layer disclosure and independent membership-inference testing against the actual deployed system passes a threshold your committee sets in advance. I would not accept “faithful explanations” as a substitute for either gate, and I would treat the bounded Llemr numbers (AUC 0.5397) as a reason to run the test, not to skip it.

What would change the verdict

This recommendation is a precautionary default built on structural reasoning, not a measured risk level for PLM, and four findings could legitimately move it. Independent replication of PLM’s results would firm up the mechanism itself. A membership-inference or re-identification attack run against a reference-patient system specifically, rather than against counterfactual explanations or a clinical LLM’s confidence scores, would replace the structural argument with an empirical one in either direction. Clinician-facing studies would tell us whether neighbor-citing rationales are useful enough to justify their governance cost at all; if clinicians find them no more actionable than attribution, the privacy cost buys nothing. And regulatory guidance on whether retrieved reference patients constitute disclosure of personal data would convert the committee’s judgment call into a compliance question with a defined answer.

Until any of that arrives, the consent question in the title has no evidence-backed answer. The PLM paper documents credentialed access to de-identified records; neither it nor any other supplied source says whether retrieving a specific cohort member as an explanation is a use that access covers. That silence is what a committee actually has to decide. Approving neighbor-citing explanations means approving a new kind of data sharing on the assumption that existing instruments cover it, and nothing in the evidence here establishes that they do.