Someone checks the numbers, and if your review team uses a large language model to extract meta-analysis data, that someone is still you. The clearest recent measurement is a September 2026 arXiv case study on intercropping research, a single-team, unreplicated evaluation, in which the best-performing extraction approach reached a mean similarity-adjusted F1 of 0.577 and the authors state plainly that none of the approaches they tested “is close to fully accurate.” Their conclusion matches what the rest of the available evidence suggests: LLMs work as first-pass extractors whose output accelerates human review, not as systems whose extracted values can enter a synthesis unchecked.
That changes where a review team’s effort belongs. When extraction is automated, accuracy is no longer purchased at the extraction step; it is purchased at the audit step. And the audit is not a formality, because the failure mode these studies document, recovering the direction of an effect while misreading its magnitude, targets exactly the values that pooled estimates are built from.
A case study that turns a worry into a measurement
The anchor evidence is Can Generative AI Automate Data Extraction for Meta-Analysis? A Case Study on Intercropping Research, which tested LLM extraction on agricultural meta-analyses. Before leaning on its numbers, two scope facts matter. First, this is one team’s evaluation, posted as a preprint, with no independent replication in the evidence available here. Second, the domain is unusually demanding: intercropping studies resist meta-analysis because of nested experimental structures and complex relationships among treatments, crops, and values spread across tables. Results from this setting describe a hard case, not a typical one.
Within that hard case, the finding is stark. According to the paper, direct zero-shot prompting was the strongest and most consistent approach, and it peaked at a similarity-adjusted F1 of 0.577. An F1 of 0.577 on extraction means that even under the best configuration, a large share of what the model pulls from papers is wrong, missing, or mismatched to the ground truth. The authors’ own reading is quoted directly:
“These findings suggest that LLMs are currently more suitable for assisting researchers with data extraction than for fully automating the process without human verification.”
That sentence is doing the load-bearing work for this entire topic. It comes from researchers who built and measured an extraction pipeline, and it is a scope statement about assistance, not a dismissal of the technology.
The pressure to ignore it is real. A review of LLM usage in literature reviews found data extraction was the second most cited automation target, appearing in 54 projects (31.4%), just behind searching for publications (60 projects, 34.9%). The same survey reports GPT-based models outperforming BERT-based models on extraction, with mean precision of 83.0% (SD=10.4) and recall of 86.0% (SD=9.8). Those precision and recall figures sound reassuring until you notice they are compatible with the intercropping result: in a review extracting dozens of items per paper across dozens of papers, a method with that level of precision still ships a substantial number of errors into your dataset.
Direction yes, magnitude no
The intercropping study’s most useful contribution is not the aggregate score but the shape of the errors. In its downstream analysis, the authors report that “most model—approach combinations recover the direction of the relationship between the predictor and outcome variables, but do not estimate its magnitude accurately.”
This is the worst possible error profile for meta-analysis, and it is worth understanding why. Direction is cheap information. Whether a treatment helps or harms is often recoverable from an abstract, a conclusion sentence, or the sign of a reported coefficient. Magnitude is the expensive information: the effect size, its variance, the sample sizes and group means that determine weighting. Pooled estimates are weighted arithmetic over magnitudes. A model that reliably gets “positive effect” right while misreading the effect size by a factor of two has given you a confident, well-formatted, wrong number.
One misread magnitude does not stay in one place. It enters the pooled estimate, shifts the confidence interval, and can flip a heterogeneity statistic or a subgroup comparison. Every downstream table and forest plot inherits it. An extraction process that is mostly correct on most items can still be wrong on the one number that carries the most weight in the pooling model, and nothing downstream will flag the problem, because the error is already inside a plausible-looking value.
The pattern is not unique to one agricultural case study. A GPT-4 feasibility study for systematic-review data extraction reported “an accuracy of around 80%, with some variability between domains”: 82% for human clinical, 80% for animal, and 72% for studies of human social sciences. Two things follow. First, a residual error rate near one in five items is the cross-study norm, not an artifact of one hard domain. Second, domain matters: the 10-point spread between human clinical and human social sciences means a verification budget calibrated on one field’s papers may be badly sized for another’s.
Where the errors concentrate
Aggregate accuracy figures like “around 80%” average across very different extraction items, and the averaging hides the actionable structure. The GPT-4 feasibility study found that “causal inference methods and study design were the data extraction items with the most errors.”
Those are not incidental fields. Study design and causal inference method are the items that determine whether two studies can be pooled at all, which model applies, and how effect sizes must be converted or standardized. An error on a categorical descriptor like study design can silently admit an ineligible study into a pool or exclude an eligible one, and it does so without producing a suspicious number for anyone to notice during analysis. A wrong magnitude at least has a chance of looking anomalous in a forest plot; a mislabeled design produces no such signal.
This concentration of errors in specific item types is, practically speaking, good news for workflow design. It means verification effort does not have to be uniform. You can spend human attention where the models measurably fail and spend less where they measurably do not, provided you know which is which for your domain and your extraction schema.
The counter-case, and what it actually claims
The strongest evidence against a blanket “LLMs cannot do this” position is LUMEN, a cost-transparent multi-agent pipeline for automated systematic review and meta-analysis. Its claims deserve precise quotation because the qualifiers carry the meaning. Across 13 ground-truth-comparable outcomes, the authors report “100% directional agreement with published meta-analyses, with effect sizes within 1% for homogeneous study designs.”
The same paper reports that its multi-agent design produced 5.7x more poolable analyses than single-model alternatives and, in ablation, eliminated what it calls clinically dangerous direction errors. It also reports end-to-end costs: a complete automated review and meta-analysis at $19–$29 (median $22.65), with title, abstract screening and data extraction together dominating expenditure. The authors explicitly time-stamp those dollar figures to March 2026 pricing while arguing the cost distribution across stages should be more stable than the absolute amounts.
Read all of that carefully and it qualifies the verification burden rather than removing it. Start with the standard LUMEN meets: directional agreement. Direction is the dimension the intercropping study found models already recover, so perfect directional agreement is fully compatible with inaccurate magnitudes. The within-1% effect-size claim is the stronger one, but it is scoped to homogeneous study designs, and homogeneous designs are precisely the opposite of the nested, table-heavy structures where magnitude estimation failed in the intercropping case. The 5.7x figure measures poolable analyses, a throughput metric, not an accuracy metric. And all of it is self-reported by the pipeline’s builders on 13 outcomes, unreplicated in the evidence available here.
None of this makes LUMEN’s results dismissible. Multi-agent architecture may genuinely reduce certain error classes, and a pipeline that costs tens of dollars per review is a serious economic fact even with dated pricing. The point is narrower: the strongest counter-evidence available does not demonstrate citation-grade magnitude extraction on hard study structures, so it does not lift the audit requirement.
What the economics actually shift
The LUMEN cost breakdown is worth one more look, because it reframes who benefits and who bears risk. If screening and extraction dominate pipeline spend, then automating those stages is where the savings concentrate, and the savings are denominated in reviewer hours. The cost of an error, by contrast, is denominated in the published record: a wrong pooled estimate that gets cited by a guideline, a policy document, or the next review. Those two costs land on different ledgers and often on different people. The team that saves the hours is not always the team whose name is on the synthesis.
This is why “the LLM extracted it and the numbers looked plausible” is not a defensible answer to a methods reviewer, an editor, or a retraction notice. The liability for every value entering the synthesis stays with the human authors regardless of which tool touched the value first.
A stratified verification workflow
Putting the measured error profile together supports a concrete, stratified policy. The stratification itself is an inference from the evidence, not a result any single paper measured, so treat it as a framework to validate on your own corpus rather than a validated protocol.
| Extraction target | What the evidence measured | Verification before the value is citable |
|---|---|---|
| Numeric effect data: effect sizes, variances, group means, sample sizes | Direction usually recovered; magnitude not estimated accurately (intercropping case study, single-team) | Full human re-extraction of every numeric value against the source PDF |
| Categorical descriptors: study design, causal inference method | The items with the most errors in GPT-4 feasibility testing | Stratified spot audit, oversampling exactly these item types, against a defined error taxonomy |
| Other study characteristics | Roughly 80% aggregate accuracy across domains, varying 72–82% by field | Random-sample audit sized to your tolerance for residual error |
| Direction of effect alone | Recovered by most model–approach combinations; LUMEN claims 100% agreement on 13 outcomes | Insufficient as a citation standard on its own |
Three implementation notes follow from the evidence rather than from general good practice. First, define the error taxonomy before the audit starts: wrong value, missing value, value attributed to the wrong study arm, design misclassification, and unit or conversion error are different failures with different downstream consequences, and the item-type concentration finding is only actionable if your audit can tell them apart. Second, size the audit to the domain. A team extracting from social-science studies, where the feasibility study measured 72% accuracy, should not reuse a sampling plan calibrated to human clinical studies at 82%. Third, record which model, prompt, and pipeline produced each extraction. When an error surfaces after publication, that record is the difference between a targeted correction and a full re-extraction of the corpus.
What this evidence cannot settle
Several questions a review team would reasonably ask cannot be answered from these sources, and it is better to name them than to imply answers.
No fetched source quantifies human dual-extraction disagreement rates, so the audit workflow above cannot be cost-compared against the standard dual-human practice it would replace or supplement. The decisive case study is single-team and single-domain; whether intercropping’s nested structures make it an outlier or a preview is an open question. Generalization to medical and social-science reviews is likewise unresolved, and the one datum pointing that direction, the 72% accuracy measured in human social sciences, suggests those fields face a harder problem, not an easier one. Finally, every source cited here is an arXiv posting. None represents independent replication, and LUMEN’s headline numbers in particular are self-reported by the system’s own authors.
Use LLM extraction as a first pass, never as a citable final. Require human re-extraction of numeric effect magnitudes, audit categorical descriptors against a defined error taxonomy with sampling weighted toward study-design and causal-inference items, and refuse direction-only agreement as a sign-off standard, because direction is the one thing the models already get right.
I would calibrate the stringency to what a silent error costs your review. If a wrong magnitude would flow into a clinical guideline or a regulatory submission, full re-extraction of every numeric value is the only posture the current evidence supports. If the synthesis is exploratory, the domain measures near the 80–82% accuracy range, and a second human will independently verify any value before it is quoted, a stratified audit can be a defensible allocation of scarce reviewer time. What the evidence does not support, from any source available, is skipping the check and letting the numbers stand on the model’s say-so.
Frequently Asked Questions
What accuracy did the intercropping case study measure for LLM extraction?
According to the paper, direct zero-shot prompting was the strongest and most consistent approach, and it peaked at a similarity-adjusted F1 of 0.577. An F1 of 0.577 on extraction means that even under the best configuration, a large share of what the model pulls from papers is wrong, missing, or mismatched to the ground truth.
Which data extraction items had the most errors in the GPT-4 feasibility study?
The GPT-4 feasibility study found that “causal inference methods and study design were the data extraction items with the most errors.”
What are the end-to-end costs reported for the LUMEN pipeline?
It also reports end-to-end costs: a complete automated review and meta-analysis at $19–$29 (median $22.65), with title, abstract screening and data extraction together dominating expenditure.
Join the discussion
Share a useful perspective or ask a question about this article.