LLM Judge Scores Change Run to Run: Why AI Audits Need Variance Reporting

A single LLM-judge score is not a measurement; it is one draw from a stochastic instrument, and a new preprint revision puts numbers on how wide that spread can be. arXiv:2507.02950, revised 27 August 2026, ran four configured LLM evaluator systems over the same 18 counseling transcripts three times each and found single-run reliability coefficients spanning .33 to .96, with all four judges scoring more generously than a 15-expert human panel on two of five constructs. The practical consequence for QA, audit, and compliance teams: before any judge number enters an evidence file, it needs multi-run variance reporting and calibration against a named human panel, or it is not evidence.

One scope note before the numbers, because the conflation is common. This is a single author-reported preprint about evaluator behavior on a simulated benchmark. Nothing in it establishes what any specific regulatory regime, including EU AI Act conformity assessment, requires of automated evaluation, and this article does not claim otherwise. What it does establish is narrower and still uncomfortable: the instrument most teams reach for to automate quality judgment produces run-dependent scores whose repeatability tells you little about whether they track expert judgment.

What did arXiv:2507.02950 actually measure?

The study compared four configured LLM evaluator systems, GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and Fable 5, against aggregated ratings from 15 counseling experts on 18 complete simulated counseling sessions conducted in Japanese, with each evaluator scoring every transcript three times. The sessions were AI-to-AI: simulated counselors working three counselor conditions across six prespecified client profiles, one session per condition-by-profile cell. Scoring covered four motivational interviewing-informed dimensions plus overall quality.

Two design choices matter for how far the results travel. First, the benchmark is fixed: 18 sessions, one per cell, not fresh stochastic regenerations. The authors state this limit themselves; the results describe these sessions, not the full distribution of sessions the simulation could produce. Second, “three runs per transcript” is the unit of analysis. The paper treats run-to-run reliability, expert-panel alignment, and condition discrimination as separate properties of an automated evaluator, which is the framing most vendor judge evaluations skip.

The preprint’s version history is worth noting because the revision is the news hook. Author Keita Kiuchi posted v1 on 28 June 2025, v2 on 8 July 2025, v3 on 4 July 2026 at 76 KB, and v4 on 27 August 2026 at 205 KB, now 55 pages with 2 figures and 31 tables, with data, code, preregistration, and amendments made available. The jump from v3 to v4 is a substantial expansion, which is why the benchmark is worth covering now rather than as a stale 2025 preprint.

Why did reliability and human alignment come apart?

Single-run intraclass correlation coefficients ranged from .33 to .96 across systems and constructs, and the authors present the spread as evidence that high run-to-run reliability did not ensure closer expert-panel alignment. A judge that gives nearly the same score every time it reads a transcript is precise. That says nothing about whether the score is right.

This is the result the article exists for, because it breaks the default assumption baked into most LLM-as-judge deployments: that a repeatable score is a trustworthy one. An instrument can be reliably miscalibrated. A judge that consistently rewards a surface feature of counseling dialogue, say fluent empathic phrasing, will produce tight ICCs and a stable offset from what trained clinicians would assign. The .96 endpoint of the reported range is not a vindication of whichever system produced it, and the .33 endpoint is not necessarily the least useful judge; the abstract reports a range across systems and constructs, not a per-judge reliability table, so neither endpoint can be attributed to a specific model without the full tables.

The mechanism behind run-to-run variation is not mysterious. Language model inference makes a probabilistic next-token prediction at each step, and temperature controls the randomness of generated text, as IBM’s LLM primer describes. A judge asked to produce a score produces tokens; ask again and the trajectory can differ. Teams that assumed temperature 0 closes this hole should read this alongside our recent piece on non-determinism in the system under audit: that piece covered the audited system, this one covers the instrument doing the auditing, and the same probabilistic machinery sits underneath both.

Which judges were repeatable, and which matched the experts?

The preprint reports that Claude Opus 4.8 had the smallest mean absolute difference from the expert panel, while Fable 5 sat at an intermediate difference; it does not, in the material available, let you map the .33 or .96 ICC endpoints to named systems. That constraint deserves respect rather than workaround. The per-judge story the abstract supports is about alignment ordering, not about a reliability leaderboard.

What the paper does say about all four systems is directional calibration: every judge assigned higher scores than the expert panel for softening sustain talk and for overall quality, though the size of the differences varied across systems and constructs. Judges, in other words, graded on a curve the experts did not use. For a QA team using judge scores to gate releases or flag sessions for review, a systematic upward bias on an “overall quality” construct is the kind of failure that passes silently: every score looks fine, the threshold is never tripped, and the audit file fills with numbers that are all shifted in the flattering direction.

There is one more result worth surfacing because it shows the benchmark doing discriminating work. In a secondary analysis, GPT-4-turbo sessions generated with a Structured Multi-step Dialogue Prompt received higher expert ratings than sessions from the same model under a minimal instruction, on cultivating change talk, partnership, empathy, and overall quality; the contrast on softening sustain talk remained uncertain. So the expert panel could detect a prompting intervention in the underlying counseling agent, and the benchmark is sensitive enough to register it. That is the condition-discrimination property the authors separate from reliability and alignment, and it is the property that makes the benchmark potentially useful for construct-level assessment of evaluators against professional judgment, which is how the authors position it.

When is an LLM-judge score usable as audit or QA evidence?

A judge score is usable as evidence when it arrives with three companions: multi-run variance for the same artifact, calibration against a named human panel for the same construct, and the exact judge system and version that produced it. Absent those, it is an anecdote with a decimal point.

The decision breaks down by stakes:

• Exploratory triage. Using a judge to sort ten thousand sessions into “look at these fifty” is low-stakes prioritization. A single run is defensible if a human reviews everything the judge surfaces. The judge is a recall tool here, and its precision does not carry evidentiary weight.
• Internal QA gating. Once a judge score blocks a release or triggers a remediation, the score has consequences and needs repeatability evidence. Score the artifact multiple times, report the spread, and set thresholds against the distribution rather than a point estimate. The preprint’s method, three runs per transcript with ICCs reported, is the floor, not the ceiling.
• Audit and conformity evidence. When a judge number goes into a file that a third party will rely on, the bar is the practical verdict this study supports: multi-run variance, calibration against a named human panel per construct, and the exact judge system and version recorded. A single unrepeated score is not audit evidence. Note what this does not say: no fetched source for this article specifies what EU AI Act conformity assessment requires of automated evaluators, and nothing here should be read as satisfying or failing any particular regulatory article. The claim is about evidentiary hygiene, not legal compliance.

The counter-evidence deserves equal weight, because it sharpens rather than blunts the guidance. The study’s own headline result cuts against the naive fix of “just pick the most reliable judge”: high repeatability and weak expert alignment coexisted. And the authors bound their inference to 18 fixed sessions, so even their alignment ordering comes with a coverage caveat. The honest read is that judge selection is a per-construct calibration exercise, not a one-time vendor choice.

What should you report instead of a single number?

Report the distribution, the panel comparison, and the instrument identity. Concretely, a judge score entering any QA or audit record should carry four fields:

1. Multi-run variance. Score each artifact at least three times, as the preprint did, and report the spread or an intraclass correlation coefficient, not just the mean. The ICC range of .33 to .96 shows how much this can move across systems and constructs.
2. Human-panel calibration per construct. The judges in this study all ran hot on softening sustain talk and overall quality relative to 15 experts. That bias was construct-specific, which means calibration has to be construct-specific too. One global “judge agrees with humans” figure would have hidden it.
3. Judge identity and version. “GPT-5.5 configured as a counseling evaluator” is a different instrument from the same model with a different rubric, and both differ from next quarter’s model. The preprint’s own v3-to-v4 expansion, 76 KB to 205 KB across 31 tables, is a reminder that even a fixed research artifact moves; production judge configurations move faster.
4. Benchmark coverage. State what the calibration sample actually covered. One session per counselor-condition-by-profile cell is a defensible fixed benchmark and a thin basis for universal claims at the same time. Your calibration set needs the same honesty about its cells.

Why might these results not generalize?

The strongest limitation is the benchmark itself: 18 simulated, Japanese-language, AI-to-AI counseling sessions, one per condition-by-profile cell, rated against 15 experts, reported in a single non-peer-reviewed preprint with no independent replication. Every clause of that sentence narrows the claim.

Simulated AI-to-AI dialogue is not human-to-human counseling, and motivational interviewing constructs scored on synthetic sessions may not behave the same way on clinical transcripts. Japanese-language sessions mean the judge behavior measured here includes each model’s Japanese register handling, which says little about English-language deployments. Fifteen experts is a reasonable panel and a small one. And the fixed-session design, which the authors flag themselves, means the results describe these 18 sessions rather than the full stochastic regenerations the simulation could produce; a redrawn benchmark might reorder the judges.

Then there is the evidence grade. arXiv states explicitly that hosted material is not peer-reviewed and is presented as-is, the submitter’s responsibility. The platform’s moderation checks topicality and scholarly value rather than reviewing content, on a platform hosting more than three million articles and receiving about 24,000 submissions a month as of late 2024, a volume that pushed arXiv to stop accepting unvetted computer-science review articles in November 2025 amid an increase in AI-generated research. The v4 preprint’s available data, code, and preregistration make it more checkable than most preprints, but checkable is not checked. No independent replication appears anywhere in the sources for this article.

The broader LLM limitations literature points the same direction: biased or inaccurate training data degrades output reliability, and benchmark evaluation of reasoning, accuracy, alignment, and safety remains an open measurement problem, per Wikipedia’s LLM entry and GeeksforGeeks’ overview. None of that invalidates the preprint’s central methodological claim, which is about how to report judge behavior rather than about any one judge’s quality. That claim survives the limitations because it is conservative: report more, assume less.

What does this change for compliance pipelines?

Treat every LLM-judge score as instrument output, and make multi-run variance plus human-panel calibration a precondition for any judge number entering an audit, clinical-QA, or conformity file. That is the preprint’s practical verdict, and the cost is real: calibration panels are expensive, repeated scoring triples inference spend at minimum, and every judge upgrade reopens the calibration exercise. Teams that bought LLM judges to eliminate human review costs will find the costs moved rather than vanished.

The specific shift this study forces is in what a compliance file has to contain. A pipeline that records “session 7 scored 4.2 on overall quality” has recorded a number with no known precision and, per this preprint, a likely upward bias on exactly that construct. The same file with three runs, a reported ICC, the judge system and version, and the panel-calibration offset is defensible in a way the bare number is not, regardless of which regulatory text eventually governs it. Teams should get their evidentiary house in order on the technical merits and let counsel map that to specific legal obligations, because the sources here support the first and not the second.

The open question is whether any vendor or standards body ships judge variance reporting as a default rather than a DIY discipline. Mainstream LLM-as-judge coverage is still leaderboard framing, one number per model, which assumes the judge is the measuring stick. This preprint’s contribution is demonstrating, with 31 tables of receipts, that the measuring stick has its own tolerance, its own bias, and its own version history. Audits that ignore that are measuring with a ruler they never checked.

Frequently Asked Questions

Does the .33 to .96 ICC range apply to English-language clinical transcripts?

No. The benchmark consists exclusively of 18 simulated Japanese-language AI-to-AI counseling sessions. The measured reliability coefficients reflect each model’s specific handling of Japanese register and synthetic dialogue structures, so the results cannot be extrapolated to English-language clinical transcripts or human-to-human interactions without independent replication.

How does the v4 preprint’s evidence grade compare to peer-reviewed journal articles?

The v4 preprint is a single-author submission that arXiv explicitly states is not peer-reviewed and is presented ‘as is’ without warranty. Unlike peer-reviewed articles, it lacks independent methodological validation, and no independent replication of its findings appears in the current literature, making it a lower-grade evidence source for regulatory compliance decisions.

What is the minimum inference cost increase for implementing the recommended multi-run variance reporting?

Implementing the preprint’s methodology requires scoring each artifact at least three times, which triples the inference compute cost for every judge evaluation. This cost increase is a fixed overhead that must be budgeted separately from the initial model deployment, as it scales linearly with the volume of sessions being audited.

Why does a high ICC score not guarantee that an LLM judge aligns with human expert ratings?

High ICC indicates precision (consistency across runs) but not accuracy (closeness to ground truth). The study found that judges with ICCs up to .96 still exhibited systematic upward bias on constructs like ‘overall quality’ compared to the 15-expert panel, meaning a judge can be reliably wrong. Selecting judges based solely on repeatability risks deploying instruments that are consistently miscalibrated.

How does the fixed 18-session benchmark limit the generalizability of the judge alignment findings?

The benchmark contains only one session per counselor-condition-by-profile cell, meaning the results describe performance on these specific 18 transcripts rather than the full distribution of possible stochastic regenerations. A redrawn benchmark with different session samples could reorder the judges’ alignment rankings, so the current alignment ordering is a point estimate with limited coverage.