Testing LLMs for Memorization Before Trusting a Benchmark Score

A benchmark score on a familiar-looking domain does not tell you whether the model reasoned or remembered, and a January 2026 chess study (arXiv:2601.16823) offers a workable way to find out before you trust the number. The method needs no access to the model’s training data: rank your eval items by how much relevant prior material exists in public corpora, score them against an executable oracle, and watch what happens as the priors thin out. In chess, what happens is steep. Base models regress toward random play on low-prior positions, and every model tested starts proposing illegal moves at sharply increasing rates.

One caveat belongs up front, because it shapes how much weight the rest of this article can carry: the central study, Disentangling generalization and memorization in large language models using chess, is an author-reported preprint that has not been independently replicated. Its design is clean and its limitations are honestly stated, but the specific numbers should be treated as reported results, not settled measurements.

The practical consequence for anyone picking a coding or RAG model off a leaderboard is direct. Public benchmarks over-sample high-prior material, because public benchmarks are built from public things. If a model’s advantage concentrates on high-prior items, the gap between its leaderboard score and its behavior on your novel, proprietary inputs is yours to discover, in production, at your expense.

Ranking positions by how much memory can help

Chess works as a testbed for a specific reason: the density of relevant priors is countable. Common opening positions occur thousands of times in the game databases the study draws on; a position counts as high-prior once it appears at least 1,000 times in the Lichess Masters database (arXiv:2601.16823), and such positions likely appear in opening books and other corpora too. A novel midgame state may never have appeared anywhere. The rules are fixed and an engine can score any move, so correctness is mechanically checkable rather than a matter of judgment.

The authors describe the design this way: “We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game’s structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors---ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models’ training data” (arXiv:2601.16823).

That last property is what makes the design transferable to your own eval suite. Most contamination defenses require either the training corpus (to check overlap) or trust in the vendor’s decontamination claims. This one requires neither. It treats memorization as a spectrum you can measure from the outside: how well does the model do as the amount of memorizable material relevant to each item decreases?

The gradient: scores fall as priors thin

Applied to a longitudinal analysis of the GPT lineage plus contemporary models including Claude Opus and Gemini, the taxonomy produced a consistent pattern: “performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline” (arXiv:2601.16823).

Random play is a hard floor. A model that selects legal moves at random is doing nothing a lookup table could not do, and the reported result is that base models approach that floor exactly where memorization can no longer carry them.

The failure mode matters as much as the score. On low-prior positions, “all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions” (arXiv:2601.16823). This is the finding I would carry into internal eval design. An aggregate accuracy number hides the difference between “reasoned to a mediocre answer” and “violated the constraints of the problem.” Invalid-action rates, whether illegal moves in chess, non-compiling code, or schema-breaking outputs in a RAG pipeline, are observable per item and tend to spike where priors run out. They are a better alarm than a blended score.

Reasoning-augmented inference helps, but with a limit that has budget implications: “while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors” (arXiv:2601.16823). In other words, spending more reasoning tokens buys the most exactly where the model needs it least. On genuinely novel inputs, the same token spend returns less. If your production workload is dominated by novel cases, the reasoning-model premium is buying less than the benchmark charts suggest.

A second chess benchmark shows the harness matters too

An independent effort, LLM Chess: Benchmarking Reasoning and Instruction-Following in LLMs through Chess, measured models playing interactively rather than answering position puzzles. The headline number is sobering against the backdrop of math and coding competition results (arXiv:2512.01992): “the LLM we evaluated peaked at Elo 758 against an engine calibrated to chess.com, corresponding to a skill level similar to that of an average online chess player.”

The more useful finding for eval designers is how scaffold-dependent the measurement is. The benchmark lets models query legal moves during play. “When removing this ability, we see a decline in model capabilities of almost 30% for Grok 3 Mini (low)” (arXiv:2512.01992), though the same ablation shows an increase of 10% for o4-mini (low). Same benchmark, same intervention, opposite signs across models.

Two conclusions follow. First, an invalid-action rate measured under one harness does not transfer to another, so when you track illegal moves, malformed tool calls, or constraint violations, record the scaffold as part of the result. Second, a model’s score partly reflects how much the harness props it up, and your production environment is a harness too. If your pipeline validates and retries malformed outputs, the model’s raw invalid-action rate understates what users see; if it does not, the leaderboard’s scaffolded numbers overstate it.

Grading your own eval items by prior density

This is the part the paper does not do for you, so to be clear about provenance: what follows is an adaptation of the chess taxonomy to general eval design, an inference the chess results make plausible but do not test. The five steps:

1. Inventory and tag. For each eval item, record two proxies for prior density: recency (did this problem, pattern, or document exist before the model’s training cutoff?) and public similarity (does a close analog exist in public repositories, contest archives, Stack Overflow, or documentation the model likely trained on?).
2. Stratify into tiers. Group items from high-prior (common public patterns: canonical algorithms, idiomatic CRUD, well-documented APIs) down to low-prior (your proprietary schemas, novel business logic, internal libraries, post-cutoff APIs). The chess study’s insight is that the tiers themselves are the instrument; you do not need to prove contamination, only to vary how much memory could help.
3. Score against an executable oracle. Prefer checks that run: compilers, test suites, schema validators, engines. The chess design works because every position has a mechanically verifiable score. Where your domain only permits human judgment, expect noisier tiers and treat the gradient as suggestive.
4. Track invalid-action rates per tier. Report non-compiling code, constraint violations, and hallucinated API calls per tier, alongside accuracy, and record the harness configuration. The chess evidence suggests these rates are where low-prior failure shows up first.
5. Gate on the lowest tier. Make selection decisions on low-prior performance rather than the aggregate. A model that wins the aggregate by dominating high-prior items is a model whose public benchmark scores will flatter it.

One bounded example to make step 2 concrete: if you are evaluating a coding assistant for a payments platform, “implement a rate limiter” is high-prior; “enforce idempotency across our three internal ledger services, given this interface definition” is low-prior. This is an illustration, not a claim about how any model performs on either task.

Off-the-shelf defenses and where each one leaks

If building tiers is more than you can do this quarter, three existing approaches cover parts of the problem. Each has a measured blind spot.

LiveCodeBench, which collects problems from LeetCode, AtCoder and CodeForces and filters by release date, is the most operationally mature: “Our benchmark mitigates contamination issues in existing benchmarks by introducing live evaluations and emphasizing scenarios beyond code generation to account for the broader coding abilities of LLMs” (arXiv:2403.07974). Its own authors flag the representativeness gap: competition problems “might not be representative of the “most general” notion of LLM programming capabilities.” And older decontamination by matching is weaker than it looks: “it can be a non-trivial task (Team, 2024) and can be evaded using simple strategies like rephrasing (Yang et al., 2023)” (arXiv:2403.07974).

The time-window assumption is quoted plainly in the dynamic-benchmark paper: “A key assumption of this method is that the model has not been trained on data beyond the cutoff, which may not hold as model developers may continue training their models on newer information sources after the cutoff” (arXiv:2505.08389). Vendor training practices are not something you can verify from the outside, so treat cutoff-based defenses as reducing risk, not eliminating it.

CDD, the black-box option, detects memorization from output consistency alone: “CDD measures peakedness: the fraction of temperature samples that are close to the greedy output.” In the paper’s controlled setting, “this consistency only arises when the model has memorized the contaminated training data and reproduces it” (arXiv:2603.03203). But its measured accuracy is dataset-dependent: “full fine-tuning on 410M at cc=10 achieves 0.96 on GSM8K but 0.72 on HumanEval,” with 0.51 on MATH in the same configuration. A detector that is nearly a coin flip on one dataset and reliable on another is a triage tool, not a verdict. The 21–30% relative improvement over baselines reported on 7B models (arXiv:2603.03203) comes from the original CDD evaluation; the dataset dependence above was demonstrated on the 70M, 410M models this paper studies, not on that 7B result.

What the chess result does not prove

Three limitations keep this from being a closed case, and two of them come from the authors themselves.

First, the design is observational. Without access to proprietary training corpora, the authors cannot guarantee that their “novel” positions were actually out of distribution, and they state they cannot empirically verify whether training leakage explains the observed gap (Appendix C.3 of arXiv:2601.16823). Prior density also co-varies with material, board topology, and tactical character (Appendix C.6), so the gradient tracks several things at once. This means the study does not prove that any named public benchmark is contaminated; it proves that performance and prior density are strongly associated in a domain where priors are countable.

Second, chess is closed. Fixed rules, countable priors, an engine oracle. Proprietary code and RAG corpora are open-ended: the priors are not countable, the oracle is often a human, and “illegal move” has no exact equivalent. Whether the same steep gradient appears there is a hypothesis this paper cannot settle. The LiveCodeBench representativeness caveat cuts in the same direction from the other side: even clean, post-cutoff contest problems may not cover open-ended real-world programming.

Third, the “just train on it” response has its own ceiling. Post-training work on chess reasoning found that “all models plateau far below expert levels,” with “evidence that this limitation stems from a deficit in the pretrained models’ internal understanding of chess—a deficit which RL alone may not be able to fully overcome.” If that deficit generalizes, fine-tuning your way out of a low-prior weakness is harder than it sounds, which raises the stakes on picking a base model whose low-prior performance is acceptable in the first place. That generalization, like the gradient itself, is an open question.

What to gate on

My recommendation, with its conditions attached: treat every public leaderboard score as a sample over high-prior material, because that is structurally what it is. Then, before trusting any score for a coding-assistant or RAG selection decision:

• Stratify your internal eval items by prior density using recency and public-similarity proxies.
• Score each item against an executable oracle wherever your domain permits one.
• Report invalid-action rates per tier with the harness configuration recorded, not a single aggregate.
• Gate the selection decision on the lowest-prior tier, and treat reasoning-token costs as tier-dependent rather than flat.

If you cannot do the stratification work yet, the fallback hierarchy is: prefer post-cutoff-windowed results over static benchmark results, treat black-box contamination detectors as triage rather than proof, and discount any score whose domain looks suspiciously like the public internet’s greatest hits.

The strongest honest summary of the evidence is this: in one well-instrumented domain, an unreplicated but carefully limited preprint found that frontier models’ competence degrades steeply and measurably as memorizable priors thin, all the way to random play. Whether your domain behaves the same way is exactly what a prior-density axis in your own eval suite would tell you, and it is the only evidence source in this article that can.

Frequently Asked Questions

How does the chess study define a high-prior position?

a position counts as high-prior once it appears at least 1,000 times in the Lichess Masters database (arXiv:2601.16823), and such positions likely appear in opening books and other corpora too.

What happens to model performance when relevant priors decrease?

performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline