A memory system can pass its retrieval evaluation and still personalize incorrectly: it finds the preference exactly as stated, but misses the later event that made it obsolete. The practical response is to run two separate checks, one for retrieval accuracy and one for current-preference identification. HorizonBench supports that split: all 25 frontier models it evaluated selected superseded preferences above chance across context-length, expression-style and filter controls, while retrieval and memory methods did not recover the lost accuracy.
The scope of that evidence is explicit. HorizonBench names all 25 evaluated models, spanning 8 OpenAI, 9 Gemini and 8 Claude models, and its Table 8 reports per-model accuracy; the best, Claude-opus-4.5, reaches 52.8%. The abstract discloses the scale: 4,245 questions from 360 users, plus a released human dataset of 83,024 ratings from 160 people over 13 weeks. Section 5.4 evaluates one vendor memory system, Mem0; Letta and Zep are not evaluated. HorizonBench is evidence for a missing evaluation layer, not a vendor ranking. Its November 17, 2026 release is nevertheless useful now because it supplies both code and a dataset from which teams can adapt the measurement design.
Correct retrieval can produce an outdated answer
The paper’s motivating example makes the failure unusually clear. A user initially prefers Socratic philosophical discussion. After becoming a chief executive, she prefers direct frameworks but does not explicitly restate her preference. When she later asks whether it is better to be respected or liked, a Socratic answer demonstrates successful retrieval and unsuccessful belief updating, according to HorizonBench.
A conventional memory evaluation can reward the wrong part of that interaction. It sees that the assistant found the stored preference and conditioned its answer on it. It does not ask whether the assistant also found the promotion, understood its relevance and treated the preference as a state that can change.
This is why recall accuracy cannot stand in for personalization quality. Recall asks whether evidence is available; currency asks whether the system’s answer reflects the evidence that should govern now. A system can fail either check independently, and the fixes differ.
HorizonBench calls the second task “long-horizon personalization”: identifying the current preference when later life events supersede earlier statements. Its authors also state the underlying distinction directly: “Remembering what a user said and knowing what the user prefers now are therefore separate capabilities.”
What HorizonBench measures
Three design choices turn that distinction into an observable result:
- The superseded value is an answer option. Selecting it directly records reliance on the outdated preference rather than leaving that failure to be inferred from a general quality score.
- Every preference change is tied to a recorded life event. The evaluation therefore contains evidence about why the transition occurred, not merely two conflicting utterances.
- Context length and expression vary while the transition stays fixed. This helps separate difficulty caused by a long or differently worded history from difficulty with the preference update itself.
Those choices make HorizonBench more diagnostic than an end-to-end score alone. They also define what a team must preserve if it adapts the benchmark: an identifiable transition, evidence for that transition and an outcome in which choosing the old value is visible.
The results suggest that simply shortening the search problem is insufficient. HorizonBench reports that two of the newest models performed no better when given only the relevant conversations than when given the full history. Yet adding one sentence that explicitly stated the change raised their accuracy by 11–22 points. Retrieval and memory methods did not recover the lost accuracy.
The interpretation I draw is bounded but useful: relevant evidence can be present without the model performing the required state update. The one-sentence intervention is a diagnostic probe, not a production fix. Asking users to restate every changed preference would remove the implicit-update problem the benchmark is designed to test.
The paper also reports 52.8% as the best evaluated accuracy, Claude-opus-4.5; Table 8 gives the other 24 per-model scores, and Section 2 defines the measures as accuracy, the 25% uniform-error distractor reference and the evolved, static gap. The number is fully documented, but it remains one benchmark on synthetic histories, not evidence about a particular product’s users.
Where existing memory evaluations stop
Static-preference and recall evaluations remain valuable. They just answer different questions.
PrefEval tests whether models follow preferences stated earlier in a conversation. Its evaluation spans 10 LLMs, 20 topics and conversations up to 100,000 tokens; the paper reports that zero-shot preference-following accuracy falls below 10% at 10 turns, approximately 3,000 tokens, for most evaluated models. That is evidence that even stated-preference adherence is difficult. It does not establish whether a model can revise a preference after a later event. PrefEval is the benchmark HorizonBench cites as Zhao et al. (2025a); the reference title and authors match.
Production memory results need the same separation. Mem0’s LOCOMO evaluation covers single-hop, temporal, multi-hop and open-domain questions. Mem0 reports a 26% relative LLM-as-a-Judge improvement over OpenAI, 91% lower p95 latency and more than 90% token-cost savings in its own evaluation. Those are vendor-reported results about that configuration and those question categories. They are not measurements of superseded-preference selection, and they should not be read as evidence of drift robustness. HorizonBench does report that measurement for one configuration: Mem0 reached 19.9% with gpt-4.1 as the reader on 1,492 items with a pre-evolution distractor, against 19.3% for the same reader on the full history. It matched the full history without improving on it, because the conversation containing the life event never names the new preference.
| Evaluation layer | What it can establish | What it cannot certify |
|---|---|---|
| Retrieval quality and attribution | Relevant memories were surfaced and used with traceable evidence | That the retrieved preference still holds |
| Static-preference adherence | The assistant follows a preference stated earlier | That it revises the preference after a later event |
| LoCoMo-style question answering | Performance across the evaluated recall categories | Robustness to preference supersession |
| HorizonBench-style supersession | Whether the answer selects the current or outdated value after a recorded transition | Performance of vendor stacks other than Mem0, or transfer to a particular product’s users |
| Task outcome and efficiency | Whether the finished interaction succeeds at acceptable latency, ingest overhead and token cost | Which memory stage caused a failure, unless earlier layers are measured separately |
This layering is consistent with the Oracle Agent Memory evaluation framework, which separates evidence retrieval, evidence use, final task outcome and operational efficiency. It also warns that scores across systems are directly comparable only when the dataset version, model, embedding model, retrieval budget, prompt, judge and scoring convention are held constant.
The broader benchmark literature points in the same direction. An agent-native memory survey criticizes reliance on single-sided end-to-end measures such as F1 and BLEU because they do not isolate retrieval fidelity. A separate memory evaluation survey recommends assessing retrieval quality, including Recall@k and nDCG, and attribution before downstream effects. The conclusion is not to discard recall metrics. It is to stop asking them to certify a capability they do not measure.
A two-check evaluation with useful failure diagnoses
The release evaluation should therefore keep retrieval and currency as separate scores.
Check 1: Can the system retrieve the relevant evidence? Include both the original preference statement and the later event in the set of evidence to find. Measure retrieval and attribution before judging the final answer. If this fails, investigate ingestion, indexing, search and context assembly first.
Check 2: Does the answer identify the preference that holds now? Present a task in which the superseded value is available as an answer or observable behavior. Report selection of the current value and selection of the superseded value separately. Do not hide the latter inside a generic incorrect-answer bucket.
The combination is more informative than either score:
- Retrieval fails and currency fails: there is no clean evidence yet that belief updating is the primary bottleneck.
- Retrieval passes and currency fails: investigate evidence use and state updating. The required information was available, but the answer still followed the old state.
- Both pass but the task fails: examine how the personalized decision is integrated into the final action.
- Accuracy passes but efficiency misses its budget: latency, ingest overhead or token cost can still block deployment, as the Oracle paper’s fourth evaluation layer makes explicit.
This diagnostic split is a design recommendation derived from the cited work, not a reported HorizonBench metric.
Adapting supersession tests to your own logs
A static-trait benchmark can be adapted if your logs contain defensible preference transitions and you can construct an outcome that exposes use of the old value. If they do not, adding more static questions will not create a currency evaluation. Build a separate transition set instead.
-
Mine transitions, not isolated preference statements. For each candidate, preserve the earlier value, the later value, the recorded event and the supporting conversation spans. A timestamp alone does not establish why a preference changed. Where the cause is ambiguous, route the example to human review or exclude it from the event-linked set.
-
Establish the current label independently of the model. The evaluation needs a defensible answer about which preference governs after the event. Otherwise, what looks like a stale answer may simply reflect an unclear user state.
-
Hold one transition fixed while varying the history. Create long and short contexts, different expression styles and a relevant-conversations-only condition. Keep the old value among the response options so that supersession failure remains directly visible.
-
Run an explicit-change probe. Add a sentence stating the transition and compare the result with the unaided condition. HorizonBench’s 11–22-point gain applies to two models in its own experiment, so treat a similar gain in your system as diagnostic evidence, not an expected improvement.
-
Diagnose before changing the memory stack. If retrieval improves but currency does not, another index or larger retrieval budget may only deliver the same unused evidence more reliably. Test evidence use and update behavior before attributing the failure to storage.
-
Freeze configurations for comparisons. Dataset version, model, embeddings, retrieval budget, prompt, judge and scoring rules all need to remain constant when comparing systems. Otherwise the result describes a configuration change, not necessarily a better memory mechanism.
Implementation details will remain stack-specific. The memorywire paper documents remember, recall, forget, merge and expire as lifecycle operations and reports that agent-memory frameworks share no common wire format. Those operation names are useful vocabulary, but they do not demonstrate that any framework performs preference supersession correctly. Test the behavior rather than inferring it from an API.
Synthetic histories need their own release gate
HorizonBench’s generator models preferences as evolving states through a mental-state graph, and the paper says that a released human longitudinal dataset calibrates its parameters. That is a partial safeguard, not proof that generated histories represent your users.
The caution comes from a different domain, so it should not be overstated. A cross-domain study of LLM-simulated survey respondents found that simulated users inflated between-segment gaps two- to fourfold, produced wrong targeting decisions in half of GSS and most WVS cases, and manufactured nonexistent segment splits in up to 41% of value questions. This was not an assistant-memory benchmark or a replication of HorizonBench. It demonstrates how structured error in generated data can survive aggregate checks and alter decisions.
Before treating an adapted generator as release evidence, compare it with non-LLM baselines and test decision impact on held-out human data. For a memory product, that means checking whether the generated transitions and the model failures they produce lead to the same engineering decisions as real, held-out histories. Calibration against the released human dataset does not replace that product-specific check.
The practical release rule is straightforward: keep recall tests to diagnose storage and retrieval, and add a separate supersession suite before claiming that personalization remains correct over time. Where real logs cannot support event-linked, human-validated transitions, treat currency as unmeasured rather than certified. Mem0, the one vendor memory system HorizonBench evaluates, matched the full history without improving on it; Letta, Zep and other vendor stacks remain untested.
Frequently Asked Questions
What is the best accuracy score reported in HorizonBench?
The paper also reports 52.8% as the best evaluated accuracy, Claude-opus-4.5; Table 8 gives the other 24 per-model scores, and Section 2 defines the measures as accuracy, the 25% uniform-error distractor reference and the evolved, static gap.
How many models and users are included in the HorizonBench evaluation?
HorizonBench names all 25 evaluated models, spanning 8 OpenAI, 9 Gemini and 8 Claude models, and its Table 8 reports per-model accuracy; the best, Claude-opus-4.5, reaches 52.8%. The abstract discloses the scale: 4,245 questions from 360 users, plus a released human dataset of 83,024 ratings from 160 people over 13 weeks.
Which vendor memory systems were evaluated in HorizonBench?
Section 5.4 evaluates one vendor memory system, Mem0; Letta and Zep are not evaluated. HorizonBench is evidence for a missing evaluation layer, not a vendor ranking.
Join the discussion
Share a useful perspective or ask a question about this article.