When LLM Agent Memory Stops Helping: What to Measure Before Adding More

When your agent’s answers get worse as its memory store grows, the fix is not a bigger store or a bigger context window. A scale-conditioned evaluation of agent memory, When Stored Evidence Stops Being Usable, found that growth does not produce one universal failure mode, so there is no universal limit to cap against. The fix is measurement: run your own task mix against the store at several sizes, for each retrieval topology you use, and track three things at each size, answer accuracy, budget adherence, and routing coverage versus actual QA accuracy. Then cap, prune, or restructure at the crossover you measured, not at a number someone else published.

That recommendation rests on a specific finding. The study tested memory utility as the store grew: flat retrieval, planar memory, single-pass structured retrieval, and multi-hop hierarchical retrieval each fail differently, and on the multi-hop interface a smaller model fails where a larger one holds, while on some planar and flat interfaces stronger models can still exhaust the budget. What there is, per the study, is a procedure you can re-run on your own stack.

This article walks through that procedure, the failure regimes it exposed, and a decision frame for append-only versus curated stores. Every number cited here is author-reported and tied to a specific system, model, and task mix; none has independent replication. Treat them as demonstrations of what to measure, not as thresholds to copy.

What counts as memory, and why the symptom shows up late

First, a scope boundary, because teams routinely argue past each other on this point. A useful definition comes from the MRMMIA paper: agent memory is “a user-specific store that the agent writes to during interaction and retrieves from in subsequent turns or sessions, distinguished from the prompt context and temporary reasoning traces that exist only within a single turn.” Case notes, retrieved facts, session summaries, stored skills. Not the system prompt, not scratchpad reasoning.

The symptom that triggers this article’s question, quality drift as the store grows, tends to appear late because each individual write looks harmless. A support agent that appends every resolved case, a research agent that stores every retrieved fact: nothing breaks at a hundred entries or a thousand. The trouble is that the cost of a bigger store is not only storage. A survey of agent memory designs, A Survey of Agent Memory in the Second Half, notes that “as the system operates for longer periods, with more turns, and when the memory grows, the escalating costs of storage and indexing may further diminish efficiency and performance.” The same survey frames the underlying substrate trade-off as “fast, precise internal state versus scalable external storage that can become noisy as it grows.” Noise, not just size, is what eventually reaches the answer.

What scale-conditioned evaluation measures, and what it deliberately does not

The scale-conditioned study’s contribution is procedural, and its self-declared scope is unusually clear. It evaluates memory utility at several store sizes rather than reporting one aggregate score at one size, and it separates failure modes by retrieval topology and by model-side control. It is equally explicit about what it does not cover: “We do not claim to measure total storage scalability, offline indexing cost, hidden backend graph computation, semantic density of returned evidence, adversarial memory robustness, or stale/contradictory memory management” (arXiv:2605.07313).

That disclaimer matters twice over. First, it means the study’s findings about answer quality in growing stores do not tell you whether your index will hold up, whether returned evidence is semantically dense, or whether your store survives adversarial or stale entries. Those axes need separate measurement. Second, it is a template for how to read any memory benchmark: if a vendor claim does not state which of these axes it measured, at which store sizes, on which task mix, you do not have evidence that memory “works,” you have evidence that something was measured once.

Failure regimes by retrieval topology

Here is the study’s core author-reported finding, quoted in full because it is the load-bearing claim:

Memory growth therefore does not induce one universal observed failure regime: flat retrieval mainly yields wrong-within-budget evidence-use failures, planar memory shows search-and-stopping instability, single-pass structured retrieval returns within-budget wrong answers, and multi-hop hierarchical retrieval can violate the budget when model-side control is weak. (arXiv:2605.07313)

Each regime implies a different fix, which is why a universal cap is the wrong tool:

• Flat retrieval fails by using evidence wrongly while staying inside the retrieval budget. More retrieval calls will not help; the problem is evidence use, so the measurement that matters is accuracy conditioned on what was retrieved.
• Planar memory shows search-and-stopping instability, meaning the agent’s control loop over when to keep searching and when to answer degrades. That is a stopping-policy problem, not a capacity problem.
• Single-pass structured retrieval returns wrong answers within budget, so on the surface everything looks efficient while quality erodes. This regime is invisible to cost dashboards.
• Multi-hop hierarchical retrieval can blow the budget when the model’s control is weak. The study reports that in LiCoMemory, Qwen3-8B produced budget-induced trajectories in 38.6% of cases at its largest tested scale point, s4 (arXiv:2605.07313), a burden that fell with model scale.

That 38.6% figure deserves careful handling (arXiv:2605.07313). It is one system, one model, one scale point, author-reported. Its transferable content is not the number; it is the interaction: the same store was manageable for a larger model and unmanageable for an 8B model. If you swap models without re-running your memory evaluation, you can inherit or lose a failure regime without changing the store at all.

Append-only versus curated: the ~167k-token collapse

The clearest measured case against append-only memory comes from a long-term-memory benchmark in building information modeling, Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval. The authors report that “indiscriminate ingestion collapsing at the upper end (about 167k tokens per task, Sec. 4.4)” in their full-turn ingestion condition. Their framing of why this generalizes is worth quoting too: “production deployments accumulate effectively unbounded history across months and many concurrent users, which cannot be reloaded in full on every query for cost and latency reasons.”

Two caveats before you do anything with 167k. It is an author-reported figure from one benchmark’s task family, not a property of token counts in general; a curated 167k-token store and an indiscriminately ingested one are different objects. And “collapse” here means the ingestion policy failed at the benchmark’s upper end, not that every store fails there. What the result establishes is narrower and more useful: full-reload append-only ingestion has a ceiling somewhere, it showed up in a measured setting, and production history growth is unbounded in a way benchmarks only approximate. One more number keeps the target precise: the same benchmark’s full-context oracle, which bypasses the memory layer entirely and reloads all prior user messages directly, reached 83.2% answer accuracy at current task sizes. So the measured counter-evidence is against indiscriminate ingestion through a memory layer; against full prompt reload in production, the argument is cost and latency, not answer quality at benchmark sizes. If your design assumes you can always ingest everything, that assumption now has measured counter-evidence.

The constructive alternative is not only pruning. SkillMem takes a different route: it “stores each reusable skill as a small set of learned latent memory vectors,” with memory “external to the frozen backbone” and “retrieved independently of other skills.” The point is not that you should adopt latent skill vectors; it is that memory design, not memory volume, is the variable that preserved utility in that system. Compact, structured, selectively retrieved memory is a third option alongside cap and prune, and it changes what “more memory” even means.

The measurement trap: coverage is not accuracy

Before you run any evaluation, one result should shape your metrics. A study of cost-sensitive store routing measured a hybrid routing heuristic at 94% routing coverage but only 70% QA accuracy (Did You Check the Right Pocket?), a gap the authors attribute to factors including needle-in-a-haystack retrieval and conflicting information between stores.

If you instrument memory by asking “did retrieval return the relevant store,” you can ship a system that routes beautifully and answers badly. Coverage is a retrieval metric; accuracy is the product. Both numbers are author-reported and specific to one heuristic, but the directional warning transfers: at growing store sizes, these two curves can diverge, and the gap is exactly where “memory stops helping” hides.

The same discipline applies to cost. The enterprise-memory evaluation in Oracle Agent Memory states the objective plainly: “not simply to minimize prompt length. It is to minimize effective cost and latency after accounting for estimated input tokens, cached-token discounts, cache-hit rate, and answer quality.” A pruning policy that shortens prompts but wrecks cache-hit rate can raise your bill while lowering quality. Measure effective cost, not token count.

The pre-growth measurement checklist

Here is the procedure, assembled from the studies above, to run before you add more memory:

1. Fix your task mix. Use tasks your agent actually runs, with known-correct answers where possible.
2. Vary store size deliberately. Test at several sizes: current, roughly double, and the size your growth rate implies in six months. The scale-conditioned study’s method is the template: utility is a function of store size, so sample the function (arXiv:2605.07313).
3. Measure per topology. If you use flat retrieval and a hierarchical index, test both. The failure regimes differ, so the crossovers will too.
4. Track three quality axes at each size. Answer accuracy; budget adherence (what fraction of runs exhaust retrieval or step budgets); and routing coverage alongside QA accuracy, because 94% coverage co-occurred with 70% accuracy in at least one measured system.
5. Track effective cost. Input tokens, cached-token discounts, cache-hit rate, per the Oracle Agent Memory objective, so a “cheaper” memory policy does not quietly cost more.
6. Repeat after model changes. The 38.6% budget-induced rate for Qwen3-8B falling with model scale (arXiv:2605.07313) means model swaps re-open the question.

When to cap, prune, or restructure

The decision frame, with each row tied to the regime evidence above:

I would treat the last row as the gating check. If coverage and accuracy have already diverged at your current size, growing the store is paying for evidence the agent cannot use, and every other row gets worse from there.

Two honest limits on this frame. Every crossover cited here is an author-reported, unreplicated condition of a specific system: LiCoMemory with Qwen3-8B, IFCMemoryBench’s BIM tasks, one hybrid routing heuristic. They demonstrate that crossovers exist and differ by design, not where yours sits. And the scale-conditioned protocol explicitly does not measure storage scalability, indexing cost, semantic density, adversarial robustness, or stale and contradictory memory (arXiv:2605.07313), so this checklist does not clear those risks; if your store faces untrusted input or long-lived facts, those need their own tests.

The durable takeaway is procedural. Memory that always helps is a design assumption, and the measured evidence says it fails in topology-specific, model-dependent ways. Teams that grow append-only stores without scale-conditioned checks pay storage, indexing, and context costs for evidence the agent increasingly cannot use. The evaluation above is a bounded measurement whose size you set; finding out from your users is slower and more expensive.

Frequently Asked Questions

What three metrics should be tracked at each store size?

track three things at each size, answer accuracy, budget adherence, and routing coverage versus actual QA accuracy.

What is the difference between routing coverage and QA accuracy?

A study of cost-sensitive store routing measured a hybrid routing heuristic at 94% routing coverage but only 70% QA accuracy (Did You Check the Right Pocket?), a gap the authors attribute to factors including needle-in-a-haystack retrieval and conflicting information between stores.

What does the 167k-token collapse indicate about append-only memory?

The authors report that “indiscriminate ingestion collapsing at the upper end (about 167k tokens per task, Sec. 4.4)” in their full-turn ingestion condition.