A new benchmark says yes, but not yet well enough to trust on its own. CareTransition-Audit (arXiv:2604.05435) is built to catch the exact failure the question names: an AI-generated discharge summary that drops a follow-up appointment, a medication change, or a red-flag warning while still reading fluently. It scores 11 large language models against a 46-question clinical-completeness checklist, and the result is uncomfortable. The best models reach only moderate agreement with clinicians, and all 11 fail the same way on ambiguous notes.
What does CareTransition-Audit actually measure?
CareTransition-Audit reframes discharge-summary quality around clinical completeness rather than prose fluency, turning a framework the authors call DISCHARGED into a 46-question checklist (arXiv:2604.05435). The distinction is the whole point. A discharge note can be grammatically flawless, well-organized, and clinically eloquent while quietly dropping the instruction to restart a blood thinner, or omitting the return precautions that separate an expected symptom from a dangerous one. Fluency is easy to score. Completeness against an external standard of safe care is not.
Discharge is the handoff where responsibility for a patient moves from the hospital to the patient and their outpatient providers, which is why omissions at that seam propagate instead of getting caught downstream. The authors locate avoidable readmissions and care fragmentation in exactly that gap, and they note that manual review does not scale to cover it (arXiv:2604.05435).
The benchmark runs 11 large language models against 50 discharge summaries drawn from the MIMIC-IV database, with clinician ground-truth labels as the reference standard (arXiv:2604.05435). Each model answers the 46 checklist items per note, and the authors compare those answers to what clinicians marked. The exercise asks whether an LLM can stand in for the human review a hospital cannot afford to run on every discharge by hand. The benchmark’s second arXiv version appeared 2026-06-17, alongside poster acceptances at IEEE-ICHI 2026 and the SD4H workshop at ICML (arXiv:2604.05435).
How well do the 11 models score?
The 11 models land in a modest band: model-assessed mean documentation completeness ranged from 54.9% to 74.2%, and the best-performing reached only about 0.5 Cohen’s kappa against the clinician labels (arXiv:2604.05435). A kappa near 0.5 is moderate agreement by the conventional reading, not strong agreement. A completeness figure in the mid-fifties to low-seventies is, charitably, a draft that still needs a human reading it against the full checklist.
The spread matters as much as the endpoints. A roughly 20-point gap between the weakest and strongest model means completeness is sensitive to model choice, yet even the top of that range leaves better than a quarter of the required checklist items unaccounted for. The two headline metrics also measure different things, and the kappa is the harsher one. The completeness percentage is the share of checklist items a model marked as present or absent; kappa measures agreement with the clinician after correcting for chance. A model can report high completeness and still be wrong about which items are actually present, which is why kappa is the number to trust. When a model marks a note complete and the clinician finds it incomplete, the percentage flatters the model and the kappa exposes it.
In operational terms, an auditor at that level of agreement would both miss genuine omissions and raise false alarms at a rate that erodes clinician trust long before it improves safety. An audit tool that clinicians learn to override is arguably worse than no tool, because it manufactures the appearance of verification without delivering it.
Where do all 11 models fail the same way?
The shared blind spot is ambiguity. All 11 benchmarked models struggled to identify ambiguous documentation, which the authors flag explicitly as a key gap in current automated auditing (arXiv:2604.05435). Under the DISCHARGED checklist, an item can be clearly present, clearly absent, or “Unclear.” The Unclear category is where a clinician reads a note and cannot tell whether an instruction was actually given. It is also the highest-stakes category for patient safety, because an ambiguous note is one a patient or downstream provider has to act on without confidence.
Ambiguity is structurally harder than presence or absence. It requires the model to recognize its own uncertainty and abstain rather than commit, and a model trained to answer confidently has no native mechanism for that abstention. That every model fails the same category is therefore more diagnostic than any single accuracy figure. A model that confidently forces every item into present or absent will look strong on a percentage score and collapse on kappa whenever the ground truth is Unclear. The authors name this failure mode as the gap their benchmark is built to expose (arXiv:2604.05435).
What does this mean for a hospital running an LLM scribe?
The benchmark’s contribution to a deployment decision is a reframing rather than a verdict. Its stated motivation, that incomplete notes drive readmissions and manual review does not scale, points straight at the operational question (arXiv:2604.05435). The second-order effect the benchmark invites is a shift in where the burden of catching omissions lands. If completeness has to happen automatically because no one can review every note by hand, then a generated discharge summary now needs a downstream check before it ships, or the hospital owns whatever the summarizer dropped.
That check, in this design, is a second model answering the 46 questions. The uncomfortable recursion is that the reviewer replacing the clinician is itself only moderately reliable: a best-case kappa near 0.5 says the automated auditor is not yet validated against the very clinicians it is meant to substitute for (arXiv:2604.05435). A health system adopting this approach would be trading an unscalable human review for an unvalidated model review, and the benchmark’s own numbers are the evidence that the trade is not yet clean.
The benchmark also shifts what “verified” is supposed to mean for a generated note. The dimension the authors argue has been missing is whether the summary preserves the steps a safe care transition actually requires (arXiv:2604.05435). Whether any given scribe vendor currently closes that gap is not something this preprint can establish. What it does is define the audit a health system would run to find out.
What is the cost of deploying a summarizer without a completeness check?
The hidden cost is that the omission becomes the hospital’s to own. A discharge summary generated with no downstream completeness check can ship a missing medication-reconciliation instruction, a dropped follow-up appointment, or an absent return precaution, and none of those failures surface in a fluency score or a word count. The benchmark’s existence is the argument that an automated completeness layer is becoming a prerequisite for safe deployment rather than an optional extra.
The caveats on that argument are sharp. This is a non-peer-reviewed arXiv preprint tested on 50 MIMIC-IV notes; arXiv is a moderated but non-peer-reviewed repository (ArXiv, Wikipedia), so the results are preliminary zero-shot baselines rather than a deploy-ready, validated clinical auditor (arXiv:2604.05435). MIMIC-IV is coded to US clinical and billing standards, so generalization beyond that population and coding is not established by this work. A health system that adopted the 46-question checklist as an audit structure would still owe its own local validation before relying on it.
The more durable contribution may be the structure rather than the leaderboard. The DISCHARGED 46-question checklist is a reusable audit template a governance team can adapt today, before the models catch up to it. The work also extends the familiar LLM-evaluation pattern of measuring reasoning, factual accuracy, alignment, and safety into a clinical-documentation completeness dimension (Large language model, Wikipedia). That extension is the part most likely to outlast the specific kappa figure, which is itself preliminary.
The benchmark’s real provocation is simpler than its numbers. It asks anyone generating a discharge note with an LLM to treat that note the way they treat any other clinical artifact: as something that has to be checked for completeness before it leaves the building, not merely proofread for style. A best-case kappa near 0.5 says the machines are not yet ready to do that checking themselves. The fact that someone had to build a 46-question benchmark to make the point says the checking is not yet happening systematically anywhere else.
Frequently Asked Questions
Does CareTransition-Audit apply to outpatient or emergency-department notes?
The 50 reference summaries come from MIMIC-IV, an inpatient ICU dataset coded to US billing standards, so the 46 questions are calibrated to hospital discharge handoffs. Generalization to outpatient, emergency, or non-US coding contexts is not established by the work and would need separate validation.
How does this benchmark differ from the accuracy figures scribe vendors publish?
Vendor numbers usually measure ambient-capture fidelity, whether the note reflects what was said in the room, and clinician time saved per encounter. CareTransition-Audit measures the opposite direction: whether a finished note preserves each required care-transition element against an external clinical standard, regardless of how fluently it reads.
What would a health system need to do before relying on the checklist?
The published scores are zero-shot, meaning the models were not fine-tuned on clinical notes, so the kappa near 0.5 is an off-the-shelf floor rather than a tuned ceiling. An institution would need to label its own notes with clinicians and re-derive the agreement numbers before trusting the audit, and a 50-note sample leaves wide confidence intervals around any kappa estimate.
What if the auditing model and the scribe share the same weaknesses?
Using an LLM to audit an LLM introduces correlated failure risk. If the auditor and the summarizer come from the same model family or training corpus, both will collapse on the Unclear category in the same way, and the audit layer will pass the very omissions it was built to catch.
Would fine-tuning the auditor push agreement above a kappa of 0.5?
A clinic-specific or few-shot auditor would probably score higher than the zero-shot baselines, yet it would sacrifice the vendor-neutral comparability that lets the benchmark rank 11 models on equal terms. The tradeoff is between a deployable local auditor and a reusable public leaderboard.
