When LLM Judges Prefer a Corpus That Hurts Retrieval

A production team ran the experiment most corpus budgets never get: they toggled each stage of a speech-to-QA pipeline on and off, asked both LLM judges and human domain experts to rank the results, and then measured what each stage did to downstream retrieval. The judges and the experts agreed completely, and both preferred the corpus that retrieval did worse on. According to the ablation study, enabling the transcript-rewrite stage lowered retrieval accuracy by 0.061, the only stage effect that survived correction for multiple comparisons, even though four LLM judges and six human experts ranked the fully refined corpus highest, by +0.18 and +0.22 respectively on a 1-5 scale. The rewrite stage was also the most expensive one in the pipeline.

One boundary before the numbers do any work: these effect sizes come from a single production pipeline built from 40 Korean medical and finance speech sessions, evaluated with six models under that study’s specific elicitation conditions. They are not base rates for corpus refinement in general, and the counter-evidence below shows refinement paying off elsewhere. What transfers is not the magnitude but the structural finding: rubric preference and retrieval utility decoupled, and the only measurement that caught it was a cheap downstream probe with statistical correction. That is the part worth building into a release process.

The setup: one pipeline, two switches, four corpora

The pipeline converts conference speech, in this case medical and finance sessions in Korean, into question-answer pairs suitable for retrieval or synthetic training. It has two toggleable stages. The preprocessing stage rewrites the raw transcript before pair generation; the refinement stage filters the generated pairs afterward. As the paper puts it, “Switching each on and off gives four corpora from the same 40 Korean medical and finance sessions.”

That design matters more than it looks. Because all four corpora descend from identical source audio, any difference between them is attributable to the stages, not to the underlying material. This is the ablation discipline that teams usually skip in production: corpus changes typically ship as bundles, a new cleaning pass and a new filter and a new prompt all at once, after which no one can say which ingredient did what.

The downstream task is retrieval: given a query, does the system surface the right pair from the corpus? The study also establishes that the corpus itself is doing real work. Supplying it at inference lifts retrieval accuracy from 0.231 to 0.748 in every model and domain tested, and more than a third of the corpus’s question-answer pairs carry knowledge that neither of two frontier models produces from parametric memory alone. So the stakes are genuine. This is not a case where retrieval barely helps and the corpus choice is cosmetic. Getting the corpus right moves accuracy by half a point; the question is which pipeline stages move it, and in which direction.

What the judges said, and what they missed

The evaluation was not lazy. Four LLM judges and six human domain experts each ranked the four corpora on a 1-5 scale, and they converged on the same ordering. The fully refined corpus, the one with both the transcript rewrite and the pair filter enabled, led by +0.18 among the LLM judges and +0.22 among the human experts.

That agreement is exactly what a release gate is supposed to look like. Two independent evaluation populations, one cheap and automated, one expensive and expert, pointing the same direction. Most teams would ship the refined corpus on that evidence and feel good about it. The whole point of spot-checking LLM-judge scores against human experts is to catch the case where the judges are fooled; here the humans concurred, so the check appeared to validate the gate.

Then the retrieval numbers came in. Enabling the transcript rewrite lowered retrieval accuracy by 0.061, and that was the only stage effect that survived correction for multiple comparisons. The stage both evaluation groups preferred was the one stage with a statistically defensible negative effect on the metric the corpus exists to serve.

Two things are worth holding separately here. First, the judge scores were not noise or a small-sample fluke within the study: both groups agreed, with margins that look consistent. Second, agreement is not validity. The judges and experts were answering the question they were asked, which corpus reads better, and the corpus they picked genuinely may read better. The problem is that “reads better” was never the objective. Retrieval accuracy was, and nobody’s rubric was measuring it.

The bill for the judges’ favorite stage

The cost asymmetry makes the inversion sharper. The study reports that the rewrite stage “reads the whole session rather than finished question–answer pairs, so turning it on almost doubles the cost of a session and adds six minutes of wall-clock time.”

Per session, that is roughly twice the token spend and six added minutes of latency, in exchange for a 0.061 reduction in retrieval accuracy. A team gating on rubric scores would have paid the premium specifically to obtain the regression. Across a production corpus of thousands of sessions, the doubled per-session cost compounds into a line item that finance will ask about, and the honest answer to “what did we get for it” would be: a corpus that scores higher on quality rubrics and retrieves worse.

There is also a subtler cost. The paper describes the usual approval path: a judge rates a sample before and after a stage, the score goes up, and the stage stays, sometimes with a human panel confirming the gain. Under that path the rewrite’s negative effect is invisible, because nothing in the procedure looks at retrieval. Rubric-based gates do not just miss regressions of this kind; they actively certify them.

A pattern, not a cause

The obvious hypothesis for the 0.061 drop is lexical: a rewrite that moves the text moves it away from the terms a query would use, and retrieval fails for reasons unrelated to content quality. The study tested that channel directly and rejected it as the driver. Lexical drift, measured as character 3-gram overlap between each question and the passages it retrieves, is real, but “the change in overlap correlates with the change in accuracy at r=0.10, which explains about one percent of the variance,” and both single stages keep their sign after the confound is controlled.

What the paper offers instead is a stated limit. The rewrite edits the transcript, and the generation phase then rebuilds every question-answer pair from the new text, so only 1.3% of the rewritten corpus’s pairs carry a question that also appears in the unrewritten one. Turning the stage on changes the input and regenerates the corpus in a single motion, and the two cannot be separated. In the authors’ words, “What we have is a pattern, not a cause.”

The rewrite’s design also rules out the easy story about what the judges were rewarding. The stage “corrects from evidence rather than from fluency”: it reads two further transcripts of the same audio from different commercial ASR systems and rewrites each sentence with all three in view, recovering proper nouns through official spellings from a domain knowledge base, and “the prompt does not ask it to label speakers or drop fillers.” This is not prose polishing. What rose were the judged dimensions, all five of them, among them coherence from 4.70 to 4.84 and faithfulness from 3.89 to 4.06. The corpus read better on exactly the axes a rubric scores, and that improvement still did not carry to retrieval.

The general lesson survives the missing mechanism. Rubric evaluation and retrieval evaluation probe different properties of the same artifact. A judge can only reward what is visible on inspection; retrieval utility depends on how the text interacts with an embedding model and a query distribution, which is not visible on inspection at all. Expecting one to proxy the other is the category error at the center of most corpus-quality gates.

The gate itself was shakier than it looked

Even if judge scores and retrieval utility usually aligned, there is an independent reason not to hand an LLM rubric the release decision: the scores are less stable than a single number implies. A study of LLM-as-a-judge methodology for open-ended tasks found that “batched evaluation lowers Repeated-Call Agreement in seven of eight settings and Decision-Rule Robustness in six, suggesting greater sensitivity to sampling variation and decision rules.”

In plain terms: the study’s batching comparison scores all of a response’s rubric criteria in one judge call versus one call per criterion, and its decision rules shift the judge’s evidence-threshold instruction between conservative, neutral and permissive wording. Change either protocol choice and the same judge, on the same outputs, can return a different verdict. That finding comes from open-ended task evaluation rather than corpus ranking, so it does not directly indict the four judges in the ablation. But it undercuts the procedural confidence teams place in a judge rubric as a gate. A measurement that wobbles with batching and decision-rule choices is a measurement you calibrate and distrust gracefully, not one you hard-wire into a release pipeline. In the corpus ablation, the judges happened to agree with the human experts; the stability work suggests how easily the judges alone could have produced a different ranking under a different protocol, with no change in the underlying corpora.

Refinement is not the villain; placement and measurement are

It would be easy to overcorrect here and conclude that refinement is wasted money. The evidence does not support that, and one result points firmly the other way.

GraFine, a refinement method applied at retrieval time over corpus graphs, “significantly outperforms Re2, the strongest document retrieval baseline, by an average of 20.0% in R@10 and 18.7% in nDCG@10.” Against the strongest graph-based baseline, GAR, on the ACL-OCL dataset, the margins are 27.5% in R@10 and 30.5% in nDCG@10, and it exceeds 55% average win rates against all baselines in the RAG setting across four datasets. Refinement, in other words, can produce large, measured retrieval gains, when it operates at retrieval time and is evaluated on retrieval metrics.

The contrast with the speech-to-QA ablation is instructive. The rewrite stage failed not because refinement is useless but because it was a build-time transformation gated on corpus appearance, and its downstream effect was never measured until someone ran the ablation. GraFine’s gains were demonstrated on the metrics the system is judged by. Placement matters, and so does what you measure.

A second qualification comes from budget-aware work on when retrieval helps: for Qwen2.5-1.5B on HotpotQA, always retrieving improves accuracy from 15.4% to 43.4%, yet “retrieval benefit occurs on 31.9% of examples.” Downstream utility is heterogeneous. An average effect can hide the fact that retrieval matters enormously for some queries and not at all, or even negatively, for others. A stage-gating probe built on a handful of test queries can easily land in the unhelpful majority and conclude a stage is safe or harmful for the wrong reasons. Probe size and composition are part of the gate’s design, not afterthoughts.

A stage-gating protocol that would have caught this

Putting the pieces together, the procedure the evidence supports is not exotic. It is cheaper than the expert review it replaces as the deciding vote.

1. Abate one stage at a time. Build the corpora with each stage toggled independently, as the study did, rather than shipping bundled changes. This is the only way to attribute effects to stages.

2. Measure downstream utility per variant, not corpus quality. The study’s anchor measurement is instructive: supplying the corpus at inference lifted retrieval accuracy from 0.231 to 0.748 across every model and domain tested, so retrieval accuracy on a held-out query set is a sensitive, decision-relevant probe. Run each corpus variant against the same queries, using the same retriever and model the production system ships with. Two controls from the study keep the probe honest. A wrong-domain control, the same pipeline’s corpus with only the domain wrong, drops accuracy to 0.100, below the 0.231 no-corpus baseline, so a gain over it cannot come from merely having text to read. And a wording filter, with its threshold fixed without reference to any accuracy score, removes questions whose reference answers lean toward one corpus’s phrasing; otherwise scoring simply favors whichever corpus shares the question’s wording.

3. Correct for multiple comparisons. Four corpora, several models, and several metrics produce enough comparisons that some effect will look significant by chance. In the ablation, only the rewrite’s 0.061 decrement survived correction; other stage effects did not. Without correction, a team could just as easily ship a stage on a spurious positive as kill one on a spurious negative.

4. Size the probe for heterogeneity. Given that retrieval benefit can appear on a minority of examples, the probe needs enough queries, spread across the domains the corpus serves, to detect effects of the size the team cares about. A 30-query spot check is an anecdote, not a gate.

5. Keep rubric scores, but demote them. Judge and expert rankings are useful for catching legibility problems and for explaining a corpus to stakeholders. What the ablation argues against is letting them override, or substitute for, the downstream probe. When the two disagree, the probe wins, because the probe measures the thing the corpus is for.

6. Log cost per stage alongside effect per stage. The rewrite nearly doubled per-session cost and added six minutes of wall-clock time. A stage with a small positive downstream effect and a large bill may still fail the gate on budget; a stage with a negative effect and a large bill, like this one, should never survive it.

What this does not prove

The limits deserve the same precision as the findings. The 0.061 decrement, the +0.18 and +0.22 judge margins, the doubled cost, and the six added minutes all come from one production pipeline, 40 Korean medical and finance speech sessions, and six models, under that ablation’s particular elicitation conditions. They are not real-world base rates. A transcript rewrite in another language, another domain, or with a different retriever could help rather than hurt, and nothing here rules that out. The judge-stability evidence comes from open-ended task evaluation, not corpus ranking, so it speaks to the gate’s reliability in general rather than to these four judges specifically. GraFine’s retrieval-time gains, meanwhile, are direct evidence that refinement can pay, which is why the recommendation is about gating on measured downstream utility rather than abandoning refinement.

The burden of proof, though, has moved. Before this ablation, a team could defend refinement spend with a rubric table and an expert sign-off, and the absence of downstream evidence was a shared blind spot. Now there is a published case, with corrected statistics, where exactly that approval process certified the one stage that significantly hurt the metric the corpus exists to improve. If a refinement stage is worth its tokens, a cheap retrieval probe can show it. If the probe shows nothing and the rubric shows polish, the honest reading is that the team is buying a better-looking corpus, and should at least say so on the invoice.