Do LLM Judges Need Opus on Every Row? Reading the 4x Cost Claim

Most LLM-as-judge pipelines can stop paying frontier prices on every row, but only under three conditions, and the headline number making the rounds does not prove any of them. Hightouch’s engineering post reports that a hybrid judge, a fast, cheap decision model called Jev that escalates its uncertain rows to Claude Opus, reached frontier accuracy at better than 4x lower cost on one workload: identity resolution. That result is worth taking seriously as an existence proof for judge cascades, and worth discounting as a general guarantee. The 4x figure is vendor-reported, single-workload, and not independently replicated. What it does establish, together with older cascade research and a new preregistered judge study, is a concrete decision rule: cascade judging beats Opus-on-every-row when your small judge’s confidence is well calibrated, you have labeled distribution-matched data to set the escalation threshold, and your volume is large enough to amortize the upfront policy cost. It also changes your failure mode. You stop worrying mainly about biased scores and start worrying about silent routing errors, because agreement between judges turns out to certify less than it looks like it should.

The 4x claim in the vendor’s own words

The claim itself, from the Hightouch post:

“We found that while both models performed okay out of the box (88.8–97.3% accuracy), implementing a hybrid model that combines Jev + Opus achieves frontier accuracy at > 4x lower costs.”

The same post quantifies the small-judge half of the cascade: decision models are “up to 90× cheaper and more than 10× faster out of the box”; they “cost $0.05 - $0.10 per 1,000 identities and answer in about 0.15 seconds,” but “trail Opus accuracy by about 6 percentage points on harder benchmarks” (Hightouch).

Two things in that quote deserve attention. First, the post prices both sides in absolute terms, per 1,000 identities rather than per token: Opus at $3.86 on the easier test and $5.39 on the harder one, Jev at $0.05 - $0.06 (Hightouch). Second, the standalone gap is about six percentage points on harder benchmarks. The cascade claim is that a routing policy closes that gap for a quarter of the cost. Everything below is about when that closure holds.

Scope matters here. A Hacker News discussion of the post made the point bluntly:

“But also, it’s a benchmark/result for a very specific experiment/task: identity resolution”

“Despite the title, the article is not making claims of, or demonstrating general usage improvements or general coding agent improvements”

Identity resolution is a structured classification task with crisp right answers, a sympathetic setting for a small specialized judge. An open-ended rubric judge grading agent trajectories is a different problem, and the Hightouch post does not measure the cascade there.

The break-even math, built from public inputs

The cost question is arithmetic, and you can run it with three inputs you already have: row volume N, the share of rows you escalate e, and the per-row prices of the two judges. Write C_s for the small judge’s cost per row and C_o for the frontier judge’s. Then:

• Frontier-only cost: N × C_o
• Cascade cost: N × (C_s + e × C_o)
• The cascade saves money whenever C_s + e × C_o < C_o, i.e., whenever the escalation rate e < 1 − C_s/C_o

This is a model built from the vendor’s stated price ratio, not a measured finding, and the inputs deserve the label. Using Hightouch’s “up to 90× cheaper” figure, C_s/C_o is roughly 0.011, which means the cascade stays cheaper than frontier-only at any escalation rate below about 98.9% (Hightouch). Cost is rarely the binding constraint. The binding constraint is the accuracy-parity condition: the whole pitch is that the cascade matches Opus, so the escalation rate is set by where your small judge’s confidence stops predicting correctness, not by the break-even line. The escalation rate still sets the size of the saving: a cascade that escalates 40% of rows to preserve parity costs about 0.41 × C_o per row in this model, roughly 2.4x lower and well short of the vendor’s headline (Hightouch).

The vendor publishes both inputs, and its tables make the point with real prices. The hybrid configurations report their achieved escalation rate in a “Share sent” column, from 7% to 74% for Jev. The headline hybrid Jev row escalates 22% of rows at $0.88 per 1,000 identities against Opus’s $3.86 (Hightouch), which is 4.4x lower; that row is the 4x claim. At 40% escalation the same table shows $1.58, about 2.4x lower, and the 74% row lands at $2.92, a saving of about a quarter. The >4x figure also holds only on the easier test: on the harder test, hybrid Jev costs $2.94 against Opus’s $5.39, about 1.8x lower (Hightouch). Substitute prices observed in your own setup before quoting these ratios internally; the post’s own caveat is that “Model costs and latencies are what we observed in our own setup and will vary.”

What “frontier accuracy” parity actually assumes

The parity claim rests on calibration, and Hightouch is unusually direct about this being the enabling condition. Comparing the two decision models it tested:

“Jev’s confidence is better calibrated. Most of its mistakes come with low confidence its right. Most of Luna’s mistakes come with high confidence. This enables Jev to perform better than Luna in a hybrid model harness.”

This is the mechanism that makes a cascade work. A router can only be as good as the signal it routes on. If a small judge’s wrong answers arrive flagged with low confidence, a threshold on confidence separates “handle locally” from “escalate” cleanly, and most escalations catch real errors. If its wrong answers arrive confidently, as the vendor says Luna’s do, no threshold recovers them: those rows get scored wrong and never escalated. Jev is a commercial decision-model system rather than a Hightouch build (arXiv:2610.29052), and Luna is GPT-6 Luna from OpenAI, so the practical lesson is not “use Jev” but “test calibration before you trust the router.” Plot your candidate small judge’s accuracy against its stated confidence on labeled rows from your own traffic. If confidence does not track correctness, the cascade premise fails for your pipeline regardless of what it did for Hightouch’s.

The validation burden is real and the vendor’s own numbers illustrate it. Hightouch reports human review running at roughly 50 identities per hour, and that estimating a rare over-merging error rate to within about 1% requires more than 1,000 samples (Hightouch), which at that throughput is upward of twenty hours of human labeling for one estimate. Routing errors at the low rates a good cascade produces are rare events by construction, and rare events are expensive to measure. Budget for that measurement before you count the savings.

Cascades are not new, and their failure modes are documented

The pattern predates LLM judges. FrugalGPT (arXiv:2305.05176) reported cost savings of 50% to 98% while matching the best individual model, by learning which queries cheaper models can answer. Hightouch’s 4x sits comfortably inside that older range, which is reassuring about plausibility and a reminder that the result is an application of known machinery, not a new effect.

FrugalGPT also states the preconditions plainly:

“And in order for the cascade to work well, the training examples should be from the same or similar distribution as the test examples. Moreover, learning the LLM cascade itself requires resources. We view this as an one-time upfront cost; this is beneficial when the final query dataset is larger than the data used to train the cascade.”

That quote is the volume condition in the decision rule. Learning the escalation policy costs labeled data and engineering time up front; the savings are per-row. If your judge traffic is thousands of rows, the policy may cost more than it ever saves. If it is tens of millions, the same policy is a rounding error. And the distribution-match requirement is not a formality: an escalation threshold tuned on last quarter’s data drifts when your traffic drifts, and the failure is silent, because miscalibrated confidence looks exactly like calibrated confidence until you re-audit.

The sharper warning comes from Learning to Cascade (arXiv:2104.09286), which describes the failure mode that cost models miss:

“Of particular note, in a case in which the prediction of the fast model is correct but the prediction of the expensive model is incorrect, using the expensive model will increase the computational cost and decrease the accuracy.”

Escalation is not automatically an upgrade. When the small judge was right and Opus is wrong, you paid more to get a worse answer. Whether this happens often enough to matter depends on how much better the frontier judge actually is on your escalated rows, which brings up an uncomfortable finding from independent research.

Opus is not automatically the better human proxy

A preregistered cross-provider study (arXiv:2611.04090), an unreviewed preprint observed November 2026, re-judged 4,000 Chatbot Arena battles with five judge models from three providers. Two findings complicate the cascade design. First, verdict-level agreement with human voters was low for every judge, and the hypothesis that judges could disagree with humans on many individual items while still ranking systems similarly was not supported. Second, the two larger Anthropic judges agreed less with Arena votes than the low-cost judges did, though the authors note their data cannot distinguish larger models being worse from larger models simply applying stricter criteria than casual voters.

Read alongside the cascade pitch, that second finding cuts the assumption that escalation is a one-way ratchet toward human judgment. On that Arena task, escalating to a bigger model moved verdicts, not necessarily toward the human. Identity resolution is a different, more objective task, and transferring Arena results to it is an assumption, not a finding. But the study measures what swapping judge size does to verdicts on a human-voted task, and it does not support “bigger judge, better proxy” as a default.

The new failure mode: silent routing errors

The study’s most consequential result for cascade designers is about errors, not averages. On 2,351 confirmation battles where all three economy judges gave a non-tie verdict, all three disagreed with the human vote on 15.8% of battles, versus 3.5% expected if their disagreements were independent (Holm-adjusted p=0.002; author-reported, unreviewed preprint). Disagreement with the human vote co-occurs across small judges roughly four and a half times more often than independence predicts. The authors put a stated limit on that reading: the reference is a single human vote per battle, which “cannot separate shared judge bias from voter noise,” so they describe the result as co-occurring disagreement with the human vote, not as evidence of common judge error.

Even read that carefully, the result undercuts a tempting fallback design. If one small judge is not confident enough, why not run three and escalate on disagreement? Because agreement among them does not certify correctness: judges that look independent can share wrong conclusions, whatever the cause of the co-occurrence, and the study’s replication analysis found this directly. The same preprint reports that when a poorly aligned judge was involved, disagreement with a second judge did flag most of its wrong conclusions, so multi-judge disagreement is a useful screen for an unreliable judge. But:

“But agreement between two judges did not certify a conclusion: the two best-aligned judges shared their one wrong conclusion, and the pair-level precision of replicated conclusions was no higher than that of the better judge alone.”

In a single-judge cascade, the risk lands on the confident rows: the rows your small judge handles confidently include any it is systematically wrong about, and those never reach Opus. Frontier-only judging has a well-studied failure mode, biased scores, that you can audit against a human sample. The cascade adds a second failure mode, routing errors, that you can only measure by sampling the confident, non-escalated rows and checking them. This is the audit that is easy to skip, because the confident rows feel safe.

There is one more reason to keep a human-check budget even after the cascade validates. A judge-hacking study (arXiv:2610.15778), also an unreviewed preprint observed in late 2026, found that optimizing against a frozen, reference-free LLM judge exploits the judge’s false positives: across twelve GRPO cells, judge-based RL did not detectably improve held-out accuracy, while a gold-reward control did for one policy. If your judge scores feed anything that adapts, a fine-tune, a prompt-iteration loop, a routing policy, judge error becomes a target. The authors frame the exploit as arising from the gap between judge-derived and gold reward signals, and a cascade widens that gap on exactly the rows it handles alone.

Putting the decision together

The comparison that matters is not “small judge versus Opus” but the two operating points as systems:

So, do LLM judges need Opus on every row? On the evidence available, conditionally no. I would cascade when three things are true, and stay on frontier-only judging otherwise:

1. Your small judge is calibrated on your traffic. Not on the vendor’s benchmark. If its confident rows are not reliably right on a labeled sample of your data, you have Luna’s problem and the harness will not save you.
2. You have labeled, distribution-matched data to set the threshold, and enough volume that the per-row savings exceed the one-time cost of learning and periodically re-learning the policy. Below that volume, Opus-on-every-row is the cheaper system, all-in.
3. You will measure routing errors before rollout and in production, by sampling confident non-escalated rows for human or frontier re-judgment. Calibration drift makes this ongoing rather than optional: a threshold tuned on last quarter’s data misroutes silently when traffic changes. And the cross-judge findings mean judge agreement is not your safety net; only checking the rows that never escalated tells you what the router is silently accepting.

What the 4x figure does not establish is any of that for your pipeline. It is one vendor’s report on one structured task, with a closed commercial small judge and unaudited parity, published about a problem the vendor says it encounters regularly in its own work. The independent literature supports the cascade pattern in general and documents exactly how it fails, and the most recent preregistered evidence warns that disagreement with the human vote co-occurs across judges, that agreement between judges does not certify a conclusion, and that the quiet failures sit in the rows you stopped looking at. Treat 4x as the top of the range the vendor’s own tables span, roughly 1.8x on the harder test (Hightouch); build the escalation policy like it is the product, and keep a line item in the budget for auditing the confident rows. That audit is the real price of the savings.

Frequently Asked Questions

What are the three conditions required for cascade judging to be effective?

cascade judging beats Opus-on-every-row when your small judge’s confidence is well calibrated, you have labeled distribution-matched data to set the escalation threshold, and your volume is large enough to amortize the upfront policy cost.

How does the cost of the hybrid cascade compare to using Opus on every row?

The headline hybrid Jev row escalates 22% of rows at $0.88 per 1,000 identities against Opus’s $3.86 (Hightouch), which is 4.4x lower; that row is the 4x claim. At 40% escalation the same table shows $1.58, about 2.4x lower, and the 74% row lands at $2.92, a saving of about a quarter.

What is the primary failure mode introduced by using a cascade instead of a single frontier judge?

The cascade adds a second failure mode, routing errors, that you can only measure by sampling the confident, non-escalated rows and checking them. This is the audit that is easy to skip, because the confident rows feel safe.