A preprint observed on September 16, 2026 reports that evaluating ChatGPT, Claude, and Gemini through their APIs produces scores that are, on average, 3.4 percentage points higher in accuracy than the same models evaluated through their consumer chatbot interfaces (arXiv:2609.08861). For procurement teams and auditors citing vendor-published eval numbers as evidence about a shipped chatbot, the implication is direct: the benchmarked artifact and the deployed artifact may not be the same system.
The evidence status needs to be stated plainly before anything else. This is an author-reported result on arXiv, moderated but not peer-reviewed, with no independent replication, and it carries no regulatory weight. It is a measurement of one question, done across three vendors, that happens to land exactly on a decision many audit teams make every quarter.
The finding: access surface alone moves the score
The study’s core comparison is simple to state. Run benchmark evaluations against a model through its API, then run the same evaluations against the same model as exposed through its consumer chatbot interface, and compare. According to the preprint, “On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test, retest agreement than corresponding interface evaluations.”
Two numbers are doing different work in that sentence. The 3.4-point accuracy gap means the chatbot answers fewer benchmark questions correctly than the API version of what is nominally the same model. The 2.1-point gap in test, retest agreement means the interface is also less consistent with itself: ask the same question twice and you are more likely to get different answers than you would through the API. Accuracy tells you how good the product is; test, retest agreement tells you how measurable it is. A system that disagrees with its own prior answers is harder to audit at all, because no single run of a test suite characterizes it.
For an auditor, that second number may matter more than the first. An evaluation methodology assumes the thing under test has stable behavior worth measuring. If the interface layer degrades consistency, then even a well-designed audit of the deployed product has a noisier signal to work with than the API numbers imply.
Scale check: a generation’s worth of degradation
Raw percentage-point figures are easy to read past, so the authors provide a calibration. Per arXiv:2609.08861, “For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation.”
That comparison is the sharpest sentence in the paper for procurement readers. A team that paid for a GPT 5.4-class product, justified in part by API benchmark deltas over GPT 5.3, could be receiving effective performance indistinguishable from the older generation once the consumer interface is interposed. Model version is the axis vendors advertise and buyers negotiate over; access surface is the axis nobody puts in the contract. The preprint argues they can be the same size.
This also inverts a common audit instinct. When a deployed chatbot underperforms its published scores, the default explanations are prompt drift, user behavior, or task mismatch. The access-surface hypothesis is rarely on the list, partly because it is invisible from the buyer’s side: there is no field in a model card labeled “interface degradation.”
Why tuning API knobs does not close the gap
The natural rebuttal is that the API and the interface differ only in configuration, and that a careful auditor can reproduce interface behavior through the API by setting the right system prompt, sampling parameters, and reasoning settings. The authors tested exactly that. Their result, from the preprint: “These controls shift behavior in some cases but do not reliably eliminate the gap.”
Both halves of that sentence matter, and the paper’s own remediation test is the strongest caveat on its headline claim. The gap is not absolute. Exposed controls do move interface-like behavior in some configurations, which means a determined evaluator can sometimes approximate the deployed product through the API. But “in some cases” is not a methodology. An auditor who must certify that an API-run evaluation characterizes the shipped chatbot has no reliable recipe for doing so, because the controls that close the gap in one task or configuration do not reliably close it in another.
The mechanism behind this is worth stating even though the preprint frames it as generation-level rather than explaining it exhaustively. A consumer chatbot is not a bare model. It is a model wrapped in a product layer: system prompts the vendor controls, routing logic, safety filters, formatting conventions, tool integrations, and settings that change without notice. Each of those components can alter output distribution. The API exposes a subset of those knobs; the rest are invisible to the caller. Auditing through the API audits the model plus whatever you configured. Auditing through the interface audits the product users actually touch.
Benchmarks were already fragile before this
The access-surface gap lands on top of problems with LLM benchmarks that predate this preprint and apply even to carefully controlled API-side evaluation. As the reference literature summarized in Wikipedia’s LLM entry documents, benchmark results are often sensitive to the prompting method used, which means the same model can post materially different scores under different evaluation harnesses. Shortcut learning lets models guess correct multiple-choice answers from statistical correlations in superficial question wording rather than from the capability the benchmark claims to test. And rapid model improvement regularly renders benchmarks obsolete, as models exceed the human annotators whose performance defined the ceiling.
The practical consequence is a compounding of uncertainties. An API benchmark score is already a noisy estimate of model capability, inflated in places by shortcut exploitation and sensitive to how prompts were formatted. The new preprint adds a second layer: even taking that noisy estimate at face value, it describes the API surface, not the shipped chatbot. A buyer reading a leaderboard is therefore two inferential steps away from the product behavior they care about, not one.
Benchmark construction itself keeps evolving in response. New instruments like MulRobBench, a decision-level benchmark for multimodal UAV agents, evaluate 3,024 samples across 17 task-taxonomy nodes and 12 metric scoring dimensions, and across 17 uniformly audited models the best strict mean scoring-dimension accuracy was 0.1599. The lesson of such work is not that benchmarks are useless; it is that scores only mean something relative to a precisely specified test setup. “The model scored X” is never a complete claim. “This access surface, under this harness, with these settings, scored X” is closer.
How much weight should an audit put on a preprint?
Everything above rests on one paper, and the provenance question deserves more than a ritual caveat. arXiv’s own description of its process states that submissions are moderated for topicality and scholarly value, but that material “is not peer-reviewed by arXiv” and is “presented ‘as is’ without any warranty or guarantee.” The repository itself has been in transition: it separated from Cornell University and became an independent nonprofit on July 1, 2026, and announced its first CEO, Penelope Lewis, and an independent board in August 2026, per Wikipedia’s arXiv entry. None of that changes what moderation does and does not certify.
So the correct epistemic weight is: directional, mechanistically plausible, internally consistent (the authors ran the obvious remediation test and reported its failure), but unconfirmed. No independent lab has replicated the result, and three vendors is a narrow base from which to generalize. The 3.4-point accuracy gap and 2.1-point agreement gap are averages across the tasks and benchmarks the authors chose; they are not universal constants, and they may shrink or grow for different task mixes, different benchmark suites, or future interface versions. Vendors also update their product layers frequently, so even a perfectly replicated result has a shelf life.
The right posture is the one audit teams already apply elsewhere. A single unreplicated study does not rewrite policy, but it can shift the burden of proof. Before this preprint, citing an API eval as evidence about a chatbot product was an untested but convenient assumption. After it, that assumption has a measured counterexample attached to it. The burden now sits on whoever claims the API number describes the deployed product. Groundy’s earlier coverage of bit-exact inference verification made a structurally similar argument from a different direction: governance regimes assume auditors can rerun the system under test and get comparable outputs, and technical reality keeps undermining that assumption. The access-surface gap is another crack in the same foundation.
Split the claims: capability versus product behavior
The actionable change for audit reports, procurement reviews, and compliance filings is a separation of claim types. Vendor-published API evals, model cards, and leaderboard positions are evidence about model capability under a specified access configuration. They are not, on this evidence, evidence about the behavior of the shipped chatbot product. An audit that cites an API accuracy figure as characterizing the consumer product is making an undocumented inferential leap, and this preprint quantifies the size of that leap for three major vendors.
In practice the split looks like this. When reviewing Groundy’s past analysis of vendor-graded benchmark evidence, the flagged problem was that internal vendor scores are the lowest-evidence category available. The access-surface finding extends that skepticism even to numbers a buyer trusts: third-party API evals still describe a surface other than the one end users touch. Where an audit must make a product-behavior claim (does the chatbot answer medical questions reliably, does it refuse appropriately, does it stay consistent across sessions), the test surface should be the deployed interface itself, and the report should say so. Where a capability claim is sufficient (is the underlying model competitive on reasoning tasks), API evals remain usable, labeled as such.
This matters most in regulatory contexts. An AI Act conformity assessment or an internal governance review that leans on leaderboard positions as product evidence now has a published, quantified argument against that practice. Audits that have treated “passed an independent evaluation” as a product-level assurance face the same narrowing problem Groundy identified in bias-audit coverage: the phrase describes a narrower test than it implies, and buyers should price it accordingly.
What an auditor should actually do
| Audit activity | Before this finding | After this finding |
|---|---|---|
| Citing vendor API evals | Treated as product evidence | Capability evidence only, labeled as such |
| Testing the shipped chatbot | Optional or sampled | Required surface for product-behavior claims |
| Closing the API–interface gap | Assumed possible via settings | Not reliably achievable; controls shift behavior “in some cases” per the preprint |
| Reporting structure | Single capability score | Split: model-capability findings vs product-behavior findings |
| Evidence weighting | Leaderboard position accepted | Provenance-weighted; preprints flagged as unreplicated |
The practical verdict follows directly. Treat vendor-published API evals and leaderboards as model-capability evidence only. In any audit, compliance review, or procurement decision about a shipped chatbot, test the actual user-facing interface directly, and report model-capability claims and product-behavior claims as separate findings rather than letting an API score stand in for the deployed product. Teams that already route model selection by workload rather than by aggregate leaderboard, as in Groundy’s axis-based routing analysis, will find this a natural extension: the access surface is one more axis, and it is not the one vendors publish numbers for.
Where the verdict may not hold
The limitation is real and should shape how far any team runs with this. The entire result is one author-reported, non-peer-reviewed preprint covering three vendors. The 3.4-point accuracy gap and 2.1-point agreement gap may not generalize across tasks, benchmarks, or interface versions, and no regulator or independent lab has replicated the finding. The remediation result cuts both ways: since API controls shift behavior in some cases, partial approximation of interface behavior through the API is sometimes possible, and a well-resourced audit team may reduce the gap even if it cannot reliably eliminate it.
What the evidence cannot settle is whether the gap is stable over time, whether it appears in vendors beyond the three tested, and whether it affects safety-relevant behaviors differently from accuracy. Those are empirical questions that independent replication would answer. Until then, the safest reading is also the most defensible one: the benchmarked artifact is not the deployed artifact, an audit that conflates them is relying on an assumption someone has now measured and found wanting, and the cost of testing the real interface is lower than the cost of certifying the wrong system.
Frequently Asked Questions
How much higher do API benchmark scores compare to chatbot interface scores?
On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test, retest agreement than corresponding interface evaluations.
Can API settings be adjusted to match chatbot interface performance?
These controls shift behavior in some cases but do not reliably eliminate the gap.
What is the evidence status of the study comparing API and interface scores?
This is an author-reported result on arXiv, moderated but not peer-reviewed, with no independent replication, and it carries no regulatory weight.
