If your team is weighing a switch from predictive coding to generative AI review, the most useful number in the current record is not the one circulating in headlines. It is the conclusion of a fresh academic reanalysis: once you account for how the two methods were actually run, Relativity’s generative review tool performed comparably to manual review and no better than the machine-learning workflow it was measured against. That finding, from Grossman’s reanalysis of the Redgrave LLP case study (arXiv 2610.16925), reshapes the decision in a specific way. The defensible default remains a validated continuous active learning workflow wrapped in statistical control sampling, and GenAI review earns its place only through a matter-specific pilot scored against the same recall target you would demand of TAR.
The stakes are not abstract. Under Sedona Principle 6, the responding party is best situated to evaluate its own procedures and methodologies, which means the validation burden sits with you regardless of which vendor’s marketing deck triggered the conversation. What follows maps each review stage to the method the evidence actually supports, and specifies what a pre-switch validation protocol has to contain.
The headline that did not survive reanalysis
In June 2026, Keeling et al. at Redgrave LLP published a comparative evaluation reporting that Relativity’s aiR achieved 87.7% recall and 28.6% precision, compared to RAL’s 64.4% recall and 39.2% precision. RAL is Relativity Assisted Learning, its machine-learning tool. The 88-versus-64 spread traveled well. It looks like a generational leap.
Grossman’s reanalysis of the same data reaches a different conclusion, and the reasoning is worth understanding because it is the kind of reasoning an opposing expert will use. In the paper’s words:
“Keeling et al. [9] report on a case study recently conducted by Redgrave LLP in which they assert that Relativity’s Generative AI Review tool (aiR) achieved 88% recall while Relativity Assisted Learning (RAL), its machine-learning tool, achieved 64% recall. These results are not comparable because the methods employed have different objectives, because of inter-assessor disagreement, and because they represent different recall-precision and cost-benefit trade-offs. Our analysis of the Redgrave data indicates that, when these differences are taken into account, aiR and manual review are comparable.”
Three confounds do the work. First, the two runs had different structures: RAL productions were filtered through a team of junior contract lawyers (L1 reviewers) who also had to code each document responsive, while aiR productions were tool-only, with no human checkpoint. Second, the L1 team, the aiR prompt engineer, and the subject-matter expert held measurably different conceptions of responsiveness. Third, the two runs sat at different points on the recall-precision and cost-benefit curve, so comparing recall alone stacks the deck.
The bottom line, again in the paper’s words: “For this particular task, the Redgrave Case Study shows aiR to be comparable to, but not superior to RAL, whether the tool or the L1 reviewer makes the final determination of responsiveness.” One more status caveat belongs here. The underlying Redgrave study is a law-firm working paper, and the bibliography of the Delayed Countdown validation paper lists it explicitly as “Redgrave LLP, Working Paper 2026-01, Jun. 2026, not peer reviewed.” A non-peer-reviewed vendor-adjacent case study is a starting point for a pilot, not a basis for a production protocol.
Filtered vs unfiltered: the human checkpoint hiding inside the comparison
The filtered-versus-unfiltered distinction matters beyond this one study because it is the decision your team actually controls. RAL’s lower recall in the Redgrave data partly reflects that its output was gated by L1 human review; aiR’s higher recall reflects that nothing stood between the tool and the production set. Those are different workflows with different risk profiles, not two readings of the same dial.
The assessor-disagreement literature cited in the reanalysis puts texture on this. The paper quantifies agreement and disagreement between assessors using “overlap,” also known as the Jaccard index, where higher values mean greater agreement; its Table 2 collects the ranges reported in the literature: independent professional reviewers overlapped only 0.158 to 0.350 on responsiveness, while the same subject-matter expert re-judging the same material at different times overlapped 0.602 to 0.806. These are literature-reported ranges, not measurements taken on the Redgrave data, but they make the point the reanalysis needs: independent professionals diverge from each other far more than a single expert diverges from their own earlier judgments, and even the expert’s self-overlap falls well short of perfect agreement. Responsiveness is not a stable property of a document waiting to be read off. It is a judgment call with substantial irreducible variance, and any tool, generative or otherwise, is being fitted to a moving target. Managed L1 review is widely accepted as a reasonable basis for production decisions, which is precisely why a workflow that includes it is easier to defend than one that produces unfiltered tool output.
This is the mechanism behind the defensibility asymmetry. A TAR workflow with a documented human checkpoint fails in ways you can describe to a court: a reviewer coded a document, a sample measured the miss rate, a confidence interval bounded it. A tool-only GenAI production fails in ways you currently cannot describe, because no ruling in the cited case law addresses GenAI-first review; treat judicial acceptance as untested.
What the established TAR evidence still covers
The reanalysis does not merely knock down the GenAI headline; it re-affirms the incumbent. Per the same paper:
“Furthermore, established machine-learning models employing Continuous Active Learning (CAL) and Scalable Continuous Active Learning (S-CAL) show state-of-the-art effectiveness, and superior efficiency, when trained only on a sample of 2,000 first-pass-reviewer responsiveness determinations.”
Two thousand first-pass determinations is a modest seed relative to the cost of a full review, and “superior efficiency” is the phrase a budget owner should hold onto when a vendor’s savings slide deck arrives. The paper also grounds TAR’s track record in the TREC 2009 Legal Track results it cites, where topics ranged in richness from 0.3% to 1.5% (median 0.6%) and achieved precision from 67.3% at 86.5% recall up to 88.4% at 69.2% recall across 569,034 emails. Those are old numbers, but that is the point: the TAR evidence base has been public, replicable, and stress-tested for well over a decade. The GenAI-versus-TAR head-to-head record, by contrast, currently rests on that one contested working paper. Other GenAI legal-review studies exist; the Delayed Countdown bibliography cites an EDRM case study from June 2025 and an IEEE BigData 2024 paper on generative AI in legal document review. None of them is a head-to-head comparison against an established TAR workflow on the same matter, which is the comparison that would change the default.
None of this says GenAI review is unsuitable in general. It says the one head-to-head comparison in this literature shows comparability, not advantage, on a single responsiveness task. Absence of an advantage in the evidence is uncertainty about GenAI’s ceiling, not proof of its floor.
What GenAI shifts onto your team
If per-document economics improve with GenAI review, the savings do not arrive free. They arrive as a transfer of burden. A validated TAR workflow carries a decade of shared understanding about what reasonable validation looks like; a GenAI-first workflow asks your team to invent and defend that validation in-house, on a novel tool, possibly under discovery dispute. And the costliest miss in any review, a privileged document produced, sits in exactly the task category where the cited studies are silent.
Two cautions bear on privilege and PII identification specifically. First, the Redgrave reanalysis covers responsiveness coding only; neither it nor the other cited studies evaluates GenAI tools on privilege or PII. Second, a chapter on the foundations of generative information retrieval notes that hallucination remains a standing problem and that LLM performance is limited “when the understanding of input documents from the external corpus requires domain knowledge that the LLMs do not possess in advance.” Privilege calls are precisely domain-knowledge-dependent: whether a communication is privileged turns on who was copied, in what capacity, and why, facts a general model does not possess about your client. Treating the responsiveness result as transferable to privilege would be an inference the evidence does not support.
The broader adversarial context is Cormack’s 2021 assessment of the eDiscovery market (an abstract-only source):
“Today, many producing parties and eDiscovery vendors similarly promote obsolete technology as well as unvetted tools labeled “artificial intelligence” or “technology-assisted review,” along with unsound validation protocols. This situation will end only when eDiscovery technologies and tools are subject to testing using the methods of information retrieval.”
That was written about the last wave of AI marketing. The labels have changed; the testing gap has not.
The pre-switch protocol: control sampling and confidence intervals
A second October 2026 paper completes the picture: the Delayed Countdown method for validating review-oriented AI workflows, submitted October 1, 2026 to the Tenth Annual Workshop on Applications of Artificial Intelligence in the Legal Industry. Its relevance to the switch decision is direct:
“Our Delayed Countdown method is easily incorporated into existing review-oriented workflows and software, is compatible with arbitrary stopping rules, and produces valid frequentist confidence intervals on effectiveness.”
Translated into practice: whatever tool you run, you should be able to state recall with a statistical confidence interval at the moment you stop, under whatever stopping rule you actually used. That is what “defensible” cashes out to. The paper anchors this in the Federal Rules’ “just, speedy, and inexpensive” standard and Sedona Principle 6: the producing party owns the methodology, so the producing party must be able to show its work. The same validation sampling applies to privilege coding too; the paper notes the underlying coding “might be for responsiveness, relevance, privilege, issues, or some other distinction.”
The same paper supplies a blunt cost warning: “In one study, the worst case over 100 samples for total review cost (review coding plus validation sample coding) was up to 20-fold greater than the median total review cost over those samples.” Validation sampling is not a rounding error you append after the fact. Sample timing and stopping rules interact with total cost in ways that can swamp the per-document savings that motivated the switch. A pre-switch protocol should therefore include, at minimum:
- A recall target set before the pilot runs, the same target your current TAR workflow is validated against, so the comparison is apples to apples rather than headline to headline.
- A control sample sized to produce a usable confidence interval, with the sampling design fixed in advance, not tuned after results arrive.
- Identical production conditions for both methods: either both filtered through human review or neither, because the Redgrave confound shows what happens when they differ.
- A stopping rule declared up front and a validation method, such as Delayed Countdown, that stays valid under that rule.
- A documented conception of responsiveness shared across the L1 team, the prompt engineer, and the SME, given the assessor-variance data above.
- A cost model that includes validation sampling, stress-tested against the worst-case-versus-median spread, not just the median.
If a GenAI pilot matches or beats your recall target under these conditions, supplementing or switching becomes a defensible choice. If the vendor cannot or will not support this design, that is also an answer.
Vendor numbers vs testable numbers
Relativity’s marketing makes strong claims, and they deserve to be stated accurately before being weighed. Its platform page describes aiR output as “Validated, Audited, Governed,” asserting that “Outputs are traceable, decisions stay with legal practitioners, and the validation, audit trails, and access controls hold up to scrutiny.” The same page cites $500K+ saved by bringing e-discovery in-house, a $10M saving on a white-collar investigation, 77% data-volume reduction, and Am Law 200 reliance. The homepage adds $1.6M in annual savings attributed to aiR and quotes a Reed Smith counsel citing more than $2.3 million in cost savings plus faster witness preparation.
None of these figures is independently tested; they are unaudited vendor assertions. That does not make them false, and the audit-trail and governance features described are genuinely relevant to defensibility. But a marketing page asserting that outputs are “validated” is not the same thing as a validation protocol with published confidence intervals, and a savings figure quoted from a satisfied customer is not a per-gigabyte cost model for your matter mix. The governance claims are checkable in a pilot; the economics claims require your own measurement.
Decision map by review stage
| Review stage | What the evidence supports | Recommended posture |
|---|---|---|
| Responsiveness, high-volume | CAL/S-CAL is state-of-the-art and more efficient at a 2,000-determination seed; aiR comparable to, not superior to, RAL on the one reanalyzed matter | Keep validated CAL as default; require confidence-interval control sampling |
| Responsiveness, GenAI supplement | Comparability, not advantage, in the Redgrave reanalysis | Pilot against the same recall target, identical filtering conditions, declared stopping rule |
| Privilege identification | No head-to-head in the cited studies; the Redgrave comparison covers responsiveness only; LLM domain-knowledge limits documented in the GenIR survey | Do not switch on current evidence; retain human privilege review |
| PII identification | Not evaluated in the cited studies | Same posture as privilege: pilot only, with control sampling |
| Validation and reporting | Delayed Countdown yields frequentist confidence intervals under arbitrary stopping rules; worst-case sampling cost up to 20x median | Adopt the protocol before adopting any new tool; budget validation into the cost model |
The pattern across the table is that the burden of proof scales with the cost of a miss. Both responsiveness and privilege error rates are statistically sampleable. I would trial GenAI review on responsiveness first, in matters where a control sample can measure what the tool actually does, and I would not let a recall headline from a differently structured comparison set the target.
Limits of this record
This analysis rests on a single reanalyzed matter, responsiveness coding only, and a non-peer-reviewed underlying study. The Cormack source is available only as an abstract. The cited sources do not cover per-gigabyte cost economics, competing vendors’ GenAI claims, or any judicial ruling on GenAI-first review. Where the record is silent, the honest position is uncertainty with a verification path, not a verdict. The verification path is the one the validation literature already supplies: run the pilot, fix the target in advance, sample with a method that yields real confidence intervals, and let your own matter’s data, not a working paper’s headline, make the call.
Join the discussion
Share a useful perspective or ask a question about this article.