Based on one newly revised preprint, the answer for prompted general models is no. The strongest zero-/few-shot LLM reached 84.0% balanced accuracy detecting errors in PET/CT radiology reports, against 94.4% for a 15M-parameter domain-specific model, and only a fine-tuned Llama-3.3-70B matched the compact detector, at substantially higher compute. Read as a procurement question rather than a leaderboard, that split is the most useful thing about the paper.
Those figures come from a single author-reported preprint (arXiv:2608.30021, first posted August 30, 2026 and revised September 7), which its authors describe as the first systematic evaluation of language models for PET/CT report error detection. No independent replication exists in the available evidence, and the paper creates no regulatory obligation for hospitals or vendors. What it does offer is a controlled comparison, on a task clinical informatics teams are actively shopping for, between compact domain-trained detectors and three open-weight LLMs.
What the preprint actually tested
The study assembled 30,633 oncology FDG PET/CT reports collected from 23 radiologists over 10 years, then trained domain-specific BERT models to detect what the authors call “clinically motivated synthetic reporting errors.” Evaluation ran on a held-out benchmark of 11,500 reports, with the domain models compared against zero- and few-shot Qwen3-32B, Gemma-3-27B, and Llama-3.3-70B.
Two design choices matter before quoting any number. Every detection target is synthetic: the errors were constructed to be clinically plausible, but they are not the mistakes radiologists actually make under reading-list pressure. And the corpus is one institution’s oncology FDG PET/CT output. The authors frame the field’s blind spot as a chest X-ray problem, noting that prior LLM work on report verification concentrated there and that capability beyond those datasets was under-explored. The same caution runs forward: radiology spans X-ray, CT, MRI, ultrasound, and nuclear medicine, which is where PET sits (Verywell Health). Nothing here measures behavior on any of the other branches.
The scoreboard, read as a procurement question
The headline comparison is stark. A 15M-parameter domain model reached 94.4% balanced accuracy with a 5.8% false-positive rate; the strongest prompted LLM managed 84.0%. Balanced accuracy averages detection performance across the error and no-error classes, which keeps a detector honest on rare error types, but the flag rate on correct reports cannot be read off that single number: identical balanced accuracy can coexist with very different flag volumes. That second quantity is reported, in the abstract, only for the compact model.
| Approach | Balanced accuracy | False-positive rate | Size | What deployment takes |
|---|---|---|---|---|
| Domain-specific BERT | 94.4% | 5.8% | 15M parameters | Task training; compact footprint |
| Prompted Qwen3-32B, Gemma-3-27B, Llama-3.3-70B (zero-/few-shot) | 84.0% (best arm) | Not reported | 27B to 70B parameters | Prompt engineering; large-model inference |
| Fine-tuned Llama-3.3-70B | 94.4% | Not reported | 70B parameters | Task-specific fine-tuning; large-model inference |
The authors’ own conclusion, that “domain-specific training matters more than model scale” for this task, positions the compact row as the reference point: “an accurate and computationally efficient approach to automated radiology report quality assurance,” in their words.
Two different systems hit 94.4% in this paper. One is a 15M-parameter BERT trained for the task; the other is a 70B Llama fine-tuned for it. Equal headline accuracy, parameter counts a few thousand times apart, and very different hosting requirements. Any summary that quotes that number without saying which system earned it is leaving out the part that decides the purchase.
Can fine-tuning erase the gap?
Yes, and that is the paper’s most instructive internal conflict. Task-specific adaptation of Llama-3.3-70B closed the gap and matched 94.4% balanced accuracy. The domain-versus-general framing then dissolves into a prompting-versus-adaptation framing: general architectures can reach domain-level detection once fine-tuned, with the cost not eliminated but moved, into labeled task data plus what the authors call “substantially greater computational requirements.”
The authors’ own conclusion survives in a different form. Domain training wins, on whichever base model you can afford to adapt. A hospital with GPU capacity and annotation resources has two viable routes to high accuracy; a hospital without them has one.
What the adapted route costs in reviewer burden, the paper does not say. The fine-tuned model’s false-positive rate is not reported in the abstract, so the two systems tied on balanced accuracy may not be tied on workflow impact.
One boundary on the evidence: the evaluation covers three named open-weight models. It cannot settle how a closed frontier chat API would score with better prompting, only that prompting as tested here was the weakest measured approach. The burden of proof sits with the claim that a chat model substitutes for domain-validated detection without adaptation, not with the skepticism.
False positives decide adoption, not accuracy
A report checker runs on every report, clean or erroneous. Every false flag sends a human hunting for a mistake that is not there. The compact model’s 5.8% false-positive rate means roughly one correct report in seventeen gets flagged; whether that is tolerable depends on how much each review costs and how severe a missed error would be, and the paper quantifies neither.
The empty cells in the table are therefore not decorative. With false-positive rates unreported for both LLM arms, no comparison of reviewer workload across approaches is possible from this paper. A prompted model at 84.0% balanced accuracy with a low false-positive rate might still be unusable for other reasons; one with a high rate would bury the reading list in noise. Two systems can post identical balanced accuracy while generating very different flag volumes, and the flag volume is what a radiology department feels. Any procurement comparison that quotes accuracy without per-arm false-positive rates is incomplete.
Detection is not correction
Where would such a checker sit? Radiologists interpret studies on PACS workstations, and in teleradiology the finalized report is transmitted to the requesting clinician (Radiology, Wikipedia). A QA detector belongs between drafting and that transmission: it reads the finished report and flags suspected errors before finalization. The sign-off point is exactly what it precedes.
What it cannot do, on this evidence, is fix. Nothing in the study tests autonomous correction; the task is framed as quality assurance. Even a perfect detector leaves every edit to a human, who must adjudicate each flag, confirm the error, and revise the text.
There is a subtler consequence worth naming. Automating detection moves the point where errors get caught, from the radiologist’s own read-through to the checker, and the checker’s misses inherit the radiologist’s old responsibility. That is a gain only if the system catches what the reader misses often enough to justify the added flags. Because the benchmark errors are synthetic, the study cannot say what fraction of real sign-off errors a deployed system would catch. That is exactly the number a hospital needs.
The governance gap report-QA tools fall into
Report-QA tools sit awkwardly between two familiar oversight categories: decision support, which surfaces information for a human to weigh, and autonomous correction, which changes clinical content. A detector that flags a suspected error but never edits text behaves like the first; hypothetically wire the same model to auto-fix and it behaves like the second. The preprint does not address the question, and nothing in it creates a mandate. It arrives as a useful capability in a category hospitals have not sorted out.
Two practical notes follow. First, no regulator is invoked: this is a research result, not a compliance requirement, and it obligates no one to deploy anything. Second, when vendors claim registered clinical validation for a report-QA tool, that claim is checkable. ClinicalTrials.gov indexes 602,104 studies across 226 countries and territories; a tool pitched as clinically validated either appears in the registry or does not.
Adjacent evidence adds a reason for caution about unsupervised general-model checking. MIRAGE, an evaluation of eight LLM agents on 750 multi-step investigation tasks, found that agents often reached relevant pages without extracting the decisive evidence on them, and that 12.6% of trajectories cited fabricated facts, a pattern persisting across model scale, generation, and reasoning architecture. That is a different task, web investigation rather than report review. But it counsels against assuming a prompted general model asked to verify a report will ground its flags in the study at hand. A confident complaint about a correct report is a false positive with a voice.
What a pilot must measure before anyone changes sign-off
Before a detector touches the sign-off workflow, a hospital should insist on numbers the paper cannot supply:
- Balanced accuracy on local data. The published figure comes from one institution’s PET/CT corpus; another’s reports, error mix, and reporting style will differ.
- False-positive rate for every candidate. Only the compact model’s 5.8% is published here; demand the number for any LLM arm under consideration.
- Review-time burden. Minutes added per flagged report and per shift, measured rather than estimated.
- Performance on naturally occurring errors. Sample real corrections from the institution’s own reporting history; the benchmark’s errors are synthetic, so this is the figure the paper structurally cannot provide.
- Modality and population coverage. Results from oncology FDG PET/CT do not transfer to other settings by assumption.
What would change the verdict
Every headline number rests on synthetic errors injected into 11,500 held-out reports drawn from a single 10-year, 23-radiologist oncology FDG corpus. Detection of real, naturally occurring errors is unmeasured, performance outside PET/CT oncology is unmeasured, no independent replication exists, and false-positive rates are published only for the compact model, so reviewer-workload comparisons across approaches cannot yet be made.
The practical verdict: treat the preprint as a procurement filter, not a scoreboard. Do not install a prompted chat model as the report-error gate; on this evidence it is the weakest option measured. Either validate a compact domain-trained detector, which pairs 94.4% balanced accuracy and a 5.8% false-positive rate with a 15M-parameter footprint, or budget honestly for fine-tuning a large open-weight model plus the inference capacity that parity requires. Pilot locally against the checklist above, and keep a human in every edit.
What would revise this picture: replication on naturally occurring errors at other institutions, false-positive rates for every arm, and a compute-cost accounting that makes the compact-versus-fine-tuned trade explicit. Until then, this is one institution’s synthetic-error benchmark, and a useful one, because it prices the assumption that a general model can simply be prompted into clinical vigilance.
Frequently Asked Questions
What is the false-positive rate for the compact domain-specific model?
The compact model’s 5.8% false-positive rate means roughly one correct report in seventeen gets flagged; whether that is tolerable depends on how much each review costs and how severe a missed error would be, and the paper quantifies neither.
Can the detector automatically fix errors in radiology reports?
What it cannot do, on this evidence, is fix. Nothing in the study tests autonomous correction; the task is framed as quality assurance. Even a perfect detector leaves every edit to a human, who must adjudicate each flag, confirm the error, and revise the text.
How can a hospital verify if a report-QA tool is clinically validated?
Second, when vendors claim registered clinical validation for a report-QA tool, that claim is checkable. ClinicalTrials.gov indexes 602,104 studies across 226 countries and territories; a tool pitched as clinically validated either appears in the registry or does not.
