Yes, conditionally. A carefully built grading cascade reached 100% precision (zero of 70 false positives) and 97% recall against human labels when scoring an agentic data-analysis system on 153 numerical tasks, according to Grading the Grader. The conditional is the whole story: the grader only behaves that well after a regex-plus-LLM-plus-human cascade, a keyword-anchored extractor, and a “nudge” that lifts successful grading runs from 36% to 97%. Gating a deploy on a single LLM grader with none of that scaffolding is the mistake the paper implicitly warns against.
What did “Grading the Grader” actually measure?
The paper evaluates a specific question: when an agentic system emits code, a numerical answer, and a verbal diagnosis, how reliably can an automated grader tell a correct run from a wrong one? The target system is LAMBDA, a multi-agent data-analysis pipeline, run over 153 numerical QRData tasks drawn from the DSGym benchmark. QRData tasks have ground-truth numerical answers, which is what makes precision and recall measurable at all.
The reason this matters is the output shape. A single-turn chat response is one string; grade it with an embedding similarity or an LLM rubric and move on. An agentic analyst emits a program, the result that program produces, and prose explaining both. A grader can be fooled by any of the three: the code can be right while the extracted number is wrong, the number can be right while the reasoning is broken, or the reasoning can be plausible while the number is a coincidence. The authors frame the core problem as separating genuine disagreement between agent output and ground truth from grading artifacts introduced by the evaluator itself.
Their answer is a three-layer human-AI grading cascade: strict regex matching, an LLM-based lenient grader, and snippet-based human inspection. The layers are chosen to have different failure profiles. The strict layer is deterministic and catches exact matches. The lenient layer tolerates formatting and presentation differences. The human layer inspects ambiguous snippets the automated layers cannot resolve. The cascade is the design lesson; the headline numbers are what it buys you.
How good are the 100% precision and 97% recall numbers?
The two automated graders in the cascade both hit 100% observed precision, with zero false positives across 70 flagged cases. That is the number that looks reassuring in a CI dashboard. The lenient LLM grader reached 97% recall against human labels, which leaves roughly a 3% gap where the grader’s verdict diverges from ground-truth correctness. The strict regex grader is the weaker of the two on recall, and the paper attributes almost all of the gap to extraction.
The extraction finding is the most useful detail for anyone building one of these. A naive “last number in the output” heuristic is the obvious way to pull an answer out of an agent’s final message. The authors replaced it with a keyword-anchored extraction pipeline that looks for the answer near task-relevant cue words, and it raised the strict grader’s recall by 60 percentage points. The parser/extractor choice dominates strict-grader quality, in other words, more than the matching logic does. The lenient LLM grader is architecturally parser-independent, which is the main structural argument for keeping an LLM in the loop at all: it sidesteps the extraction problem that sinks the regex layer.
There are two asterisks worth attaching to the headline. First, “observed precision” is measured against the 70 cases that reached a human label, not against the full task set, so the denominator is smaller than 153. Second, this is a single case study on numerical QRData tasks, not a general benchmark. A grader that extracts a scalar answer and compares it to a known number is an easier problem than grading an open-ended coding agent or a multi-step research report. The paper says so itself, framing the variable-type field as the task metadata most consistently associated with grading-pipeline dynamics and observed outcome grades. Grader accuracy is task-shape-dependent, and tasks with a clean scalar answer are the easy case.
What is the grader’s real failure mode?
Silence. Before the nudge, grading runs succeeded only 36% of the time, meaning the cascade produced no usable verdict on roughly two-thirds of runs. The lenient-pass rate over those runs was 16%. The nudge mechanism, an iterative re-prompt that coaxes the agent toward a structured answer, raised run success to 97% and lenient-pass rates to 46%.
That 36%-to-97% gap is the number that should change how you read the headline. A grader that is correct 100% of the times it speaks is not very useful if it speaks on only a third of inputs. The precision figure describes the conditional probability of being right given that a verdict was produced. The run-success figure describes how often a verdict gets produced at all. A CI gate built on the raw grader, without the nudge, would fail open on the majority of cases by simply having nothing to fail on.
The nudge finding has a sharper edge. The authors compared nudging with and without re-injecting the original task question, and re-injection offered no benefit. That result is counterintuitive, since the obvious theory is that the nudge helps by restoring context the agent lost. The data says otherwise. The nudge works as an answer-template cue, pushing the agent to emit its result in a shape the extractor can parse, not by re-grounding the agent in the task. For anyone designing an eval harness, that is the actionable point: the lever that moves run success is output formatting, not context recovery.
Why can’t a single LLM grader gate CI on its own?
The Grading the Grader numbers are encouraging precisely because the cascade is doing most of the work. Strip away the regex layer, the keyword extractor, the nudge, and the human spot-check, and a lone LLM judge is a known-weak component. The literature on single-model judging documents systematic biases that the cascade is partly there to absorb.
A survey of agent-as-a-judge methods, When AIs Judge AIs, concludes that agent-based judging can complement but not replace human oversight, and that single LLM judges carry verbosity and self-preference biases and can be gamed by adversarial outputs, including a nonsense “null” response that tricked GPT-4 into issuing high rankings. Separately, practitioner analysis quantifies four biases: verbosity inflates scores by roughly 15%, position shifts pairwise accuracy by 10% or more, self-preference gives a 5% to 7% boost when the same model family generates and judges, and agreeableness produces true-positive rates above 96% alongside true-negative rates below 25% in class-imbalanced settings. The last figure is the dangerous one for an eval gate: a judge that says yes to almost everything correct and almost nothing incorrect is, in a setting where most agent outputs are wrong, a rubber stamp that looks rigorous.
A larger benchmark reaches a compatible conclusion about what “good judge” even means. Judge’s Verdict tested 54 LLMs (43 open-source from 1B to 405B parameters, plus 11 closed GPT, Gemini, and Claude variants) and found that correlation with human labels is insufficient to evaluate a judge. Using a Cohen’s-Kappa agreement analysis with z-scores, the authors report that only 27 of 54 models reached their top tier, with 23 showing human-like agreement patterns and 4 showing super-consistent behavior that could indicate either reliability or oversimplified judgment. Their claim that judge excellence depends on specific training strategies rather than solely model size, plus the “Turing Test for judges” framing, both point the same direction: picking a strong frontier model and calling it your grader is not a validated choice.
What does a trustworthy agentic-eval pipeline look like?
The Grading the Grader cascade is a reasonable template, and the transferable pieces are structural rather than model-specific.
Start with a deterministic layer for the cases where determinism is possible. Strict regex or exact-match grading, backed by a keyword-anchored extractor rather than a last-number heuristic, catches the unambiguous wins and losses with zero LLM cost and zero bias surface. The 60-percentage-point recall lift from better extraction is the cheapest win in the whole pipeline.
Put an LLM lenient grader behind the deterministic layer, scoped to the cases the regex layer could not resolve, so the model’s tolerance for formatting variation is applied where it earns its keep and its biases are confined to a minority of inputs. Keep it parser-independent, as the authors did, so the extractor problem does not re-enter through the LLM’s prompt.
Add a nudge or equivalent output-formatting step that pushes the agent toward a parseable answer shape. The evidence says this is a template cue, not a context restorer, so design it as such: structured answer fields, explicit “final answer:” markers, constrained output schemas. Expect run success to dominate your effective precision more than the grader’s per-verdict accuracy does.
Keep a human spot-check budget on the residual. The whole point of adopting an LLM judge is usually to remove human review, and the second-order effect the paper makes visible is that the judge itself becomes a component with a false-positive rate, a false-negative rate, and a silent-failure rate that all need measurement. Auditing the judge, not just the agent, is what closes the loop.
The honest summary is narrow. On scalar-answer agentic tasks, a cascade-built grader can be trusted enough to act on, with a measured and small error rate and a measured and large silence rate that you have to engineer away. On open-ended coding, research, or multi-step agent outputs, the same numbers do not transfer, and the secondary literature on single-judge biases is the stronger reason not to hand a lone LLM the gate key. Trust the grader to the extent you can show its own confusion matrix, and not a percentage point more.
Frequently Asked Questions
Would the cascade work on non-numerical agent outputs?
Probably worse, and without a measured error rate to lean on. The study covered only 153 numerical QRData tasks, and named variable type as the task field most predictive of grading behavior. String, categorical, or free-text outputs sit outside the evidence base, so a team running them has no precision or recall figure to trust.
How large is the human spot-check layer in practice?
Roughly 46% of runs. In the study, 70 of 153 tasks reached human labeling, which makes the human layer a triage stage handling close to half the input, not a near-removal of review. A team that budgets for occasional spot-checks will underspend on the layer that actually closes the loop.
How is this different from MT-Bench or Arena-Hard judge benchmarks?
Those score single-turn chat responses with rubric or pairwise judging and report judge-human agreement. Grading the Grader scores agentic outputs (code, numerical result, prose) against ground truth through a pipeline with extraction and a nudge. The failure modes differ: chat judges drift on style, agentic judges fail on parsing.
Where does the agreeableness bias actually break a deploy gate?
When wrong answers dominate the input, which is the realistic case for a struggling agent. The measured true-negative rate under 25% means roughly three of four wrong outputs get marked correct. A gate that trusts those verdicts ships bugs while looking rigorous.
How is the nudge different from chain-of-thought or self-consistency prompting?
Both chain-of-thought and self-consistency add reasoning. The nudge does not: re-injecting the original question offered no benefit, which rules out context recovery as the mechanism. It works as an output-schema constraint that pushes the agent toward a parseable answer shape, cheaper than sampling multiple reasoning paths and more targeted than adding reasoning steps.
