groundy
Agents & Frameworks

What Memory-Poisoning Defenses Cost AI Agents on Benign Inputs

A new paper measures the benign-traffic cost of memory-poisoning defenses, finding token overhead is low but rerankers withhold relevant memory from 14.9% of questions.

Published 5 references
A bubbled forest-green resin arch holds an ivory puzzle-shaped piece in its opening. An incomplete ivory ring and a loose curved piece sit beside it, casting hard shadows on a warm ivory surface.
On this page13 sections

If you add anti-poisoning defenses to an agent with persistent memory, what do you pay when nobody is attacking? Until recently, the honest answer was that nobody had published the whole bill. Prior work occasionally added a single utility figure, but none reported accuracy change, token overhead, and false-quarantine rate together, replicated across runs (source). A new paper, The Price of Safety (arXiv 2609.22818), measured exactly that: four defense classes run against purely benign traffic, holding the model, memory backend, retrieval, prompts, and judge fixed so only the defense changes. The short version of its findings: sanitization and provenance cost no extra tokens, anomaly detection costs 1.5%, a reranker costs 3.0%, and stacking all four costs 4.5% (source). The subtler finding is that the condition that burned the most tokens, all four defenses stacked at 4.5% (source), was not the one that withheld the most memory. The reranker, at 3.0% in tokens (source), quietly withheld memory from one question in seven, and about half the audited drops removed a memory the question actually needed.

This article converts those author-reported numbers into a deployment decision: which defenses to leave always-on, which to gate on memory exposure, and when shrinking memory lifetime beats screening everything. Every figure below comes from one lab’s own benchmarks, with no independent replication, so treat them as calibrated estimates rather than constants.

The bill when nobody’s attacking

The paper’s core contribution is a per-defense-class price list, and the authors are unusually explicit about their method: “Memory-poisoning defenses for LLM agents are usually judged by how well they stop attacks, yet almost all the traffic an agent sees is benign. We measure what these defenses cost when no attack is present.”

Here is the cost table, all figures author-reported from the paper’s own harness (source):

Defense classToken overheadBenign interruptionAccuracy effect
Sanitization0%0% (never fired)no detectable cost
Provenance0%0% (never fired)no detectable cost
Anomaly detection1.5%blocked 0.4% of benign turnsno detectable cost
Reranker3.0%dropped 2.2% of memories judged; withheld memory from 14.9% of questionsno detectable cost
All four stacked4.5%blocked the same 0.4% of turns; dropped 1.3% of judged memories; 8.0% of questions lost at least one memoryno detectable cost

In the authors’ words (source): “Token overhead was 0% for sanitization and provenance, 1.5% for anomaly detection, 3.0% for the reranker, and 4.5% with all four stacked.” And on the benign-traffic side (source): “Yet anomaly detection blocked 0.4% of benign turns, and the reranker dropped 2.2% of the memories it judged, more than five times as often.”

Read this table the way the authors intend. Stacking did not compound the interruption: the full stack blocked the same 0.4% of turns (source) and withheld memory from fewer questions than the reranker alone, though the two conditions read different stores, so the authors do not read that gap as an interaction. The token overhead is modest: even a full stack adds 4.5% to context cost (source). For an agent doing a few hundred memory-assisted turns a day, that is a rounding error on the API bill. If cost were only tokens, “enable everything” would be a defensible default. It is not the whole cost.

What the reranker actually costs: withheld memory, not accuracy

The reranker’s interesting number is not its 3.0% token overhead (source). It is 14.9%: the fraction of questions for which it withheld memory (source), despite showing no detectable accuracy cost on the benchmark. The paper’s audit of that gap is worth quoting in full (source): “The reranker had no detectable accuracy cost, yet it withheld memory from 14.9% of questions. In the audit, 26 of 50 drops removed a memory relevant to the question, but only 3 of those turned a correct answer wrong.”

Two things follow. First, the benchmark’s headline accuracy metric largely survived because the agent could often answer from other memories; the defense’s real behavior change (denying the agent a relevant fact) was invisible to the aggregate score. Second, that invisibility cuts both ways. On a benchmark with redundant information, a dropped memory is survivable. In a production agent where the dropped memory is the only record of a customer’s stated constraint, the same 2.2% drop rate (source) is a reliability bug that no accuracy number will surface.

The paper’s suggested remedy is worth noting for anyone building read-time filters: “The reranker’s errors suggest a direct remedy: give read-time filters the question and the memory’s source turn.” Context-poor filters over-block; giving the filter both the question and provenance for each memory is a concrete design fix, not a tuning knob.

How to read “no detectable cost”

Every row in the table above says “no detectable cost” for accuracy, and the authors insist that phrase be read literally: “Interval half-widths range from 2.4 to 4.6 points, so costs of one or two points are hard to detect. Our nulls therefore show no detectable cost, not zero cost.” Two conditions did move the aggregate: anomaly detection and the stacked condition each gained 4.7 points, but the authors attribute that gain to a blocked turn changing which memories got extracted, and they decline to count it as a benefit (source).

There is also a cautionary tale inside the paper’s own revision history. An earlier version reported a 4.4-point accuracy cost for the reranker. The authors later disclosed that the reranker condition had run on a different model than its baseline, and a same-model baseline attributed 2.9 of those 4.4 points to the model swap rather than the defense. A model swap alone, with no defense at all, moved accuracy 2.9 points, a drift squarely inside the study’s 2.4 to 4.6 point resolution band. Any real defense cost of that size would have gone undetected at the same resolution.

Two practical consequences. First, if a vendor or colleague cites a small “accuracy cost of safety” figure, ask what baseline it was measured against and whether the backbone was pinned; this study’s backbone was a hosted model without a pinned version whose extraction changed mid-project. Second, for your own evaluation: any A/B comparison of defended versus undefended accuracy needs intervals tighter than about 2 points before “the defense costs nothing” means anything. Most teams do not run evaluations with that precision.

What skipping defenses costs

The other side of the ledger comes from the attack literature, which is larger but measured on different memory architectures, so treat these numbers as pressure indicators rather than transferable risk estimates for your system.

The strongest attack result among these studies is ShadowMerge (arXiv 2605.09033), a poisoning attack on graph-based agent memory: “ShadowMerge achieves 93.8% average ASR, a 50.3 absolute ASR gain over the best baseline, while having negligible impact on unrelated benign tasks.” The detail that should shape your monitoring strategy is the 0.973 benign-task utility the attack preserved while succeeding at a 0.938 rate. An agent can be thoroughly poisoned and still pass your “is it answering fine?” smoke test. Benign utility is not a safety signal.

On the multi-turn agentic side, the TMA-NM paper (arXiv 2606.24322) reports the undefended agent was poisoned in 50.0% of episodes (Wilson 95% CI [40.2, 59.8]). And on web agents, the environment-injected poisoning study (arXiv 2604.02623) found GPT-OSS-120B the most vulnerable of the tested models under no environment stress, with attack success rates (ASRB) from 6.0% to 19.5% depending on the attack strategy, though the same study notes that model’s task success rate was itself low, 0.7% to 3.5%, which limits how much weight that ranking can carry (source).

None of these studies used the conversational memory setup from the cost paper, and none used file-backed memory. They establish that memory poisoning works, across architectures, at rates that justify doing something. They do not tell you your agent’s rate.

A defense that taxes nothing: the TMA-NM counterweight

The strongest counter-evidence to “defenses tax benign behavior” is TMA-NM itself (source): a structurally enforced, origin-bound authority design that held poisoning at 0% (CI [0, 3.8]) while authorizing 95 of 96 legitimate actions, 99.0%, identical to the undefended agent (source). The single shared miss was a model formatting failure under both arms, not a defense block.

Where the cost of a “free” defense hides is in engineering, not tokens. TMA-NM’s authority does not live in the memories themselves: corroboration counts are maintained in a monitor’s append-only record, and origin labels have to survive the memory backend’s consolidation, which rewrites stored content. Sanitization, provenance, and origin binding cost pipeline redesign rather than per-turn spend, and the token column in the cost table measures only what is easy to meter.

Detector shopping

If your plan is to buy or adapt a screening detector, a systematic study of memory poisoning attacks in LLM agents (arXiv 2606.04329) is the relevant shopping data. Among off-the-shelf defenses, PromptArmor had the best overall balance: 67.67% true positive rate at 1.00% false positive rate (source). Adaptation moved the trade rather than eliminating it: PIGuard improved from 38.33% to 47.67% TPR while its false positive rate rose from 0.33% to 5.33% (source), and CommandSans reached 61% TPR with an FPR still high at 8.67%.

Map those false positive rates onto a benign-traffic reality. At an 8.67% FPR (source), roughly one in twelve benign inputs to the detector gets flagged; every flag is a blocked turn, a dropped memory, or a human review. That is a blocked-input rate more than twenty times the cost paper’s write-time screen, which blocked 0.4% of benign turns (source), though in the same range as the reranker’s question-level withholding at 14.9% (source). The paper’s framing is the right reconciliation lens: “Table 3 shows TPR on adversarial inputs and FPR on benign inputs for each defense.” Demand both numbers before buying either.

The tiering rule

Synthesizing the cost gradient, the attack pressure, and the detection trade-offs, here is the deployment rule I would apply, with the caveat that no study in this literature compares always-on versus tiered deployments end-to-end, so this is inference from measured gradients, not a measured result:

  1. Structural defenses always-on. Sanitization and provenance cost 0% in tokens (source) and showed no measurable benign interruption. Where your memory architecture can support origin binding (TMA-NM’s approach), the evidence suggests it can hold poisoning near zero without taxing legitimate use (source). Budget real engineering time here; this tier’s real cost is pipeline design, not per-turn tokens.
  2. Gate per-turn screening on memory exposure. Anomaly detection (1.5% tokens, 0.4% blocked turns) and rerankers (3.0% tokens, 14.9% of questions denied memory) (source) belong on memory that crosses a trust boundary: content written from tool outputs, web pages, user uploads, or other agents. Memory written only from a single trusted user’s direct turns is a weaker case for paying the screening tax on every read.
  3. Shorten memory lifetime before screening everything. The cheapest poisoning defense is memory that expires. Scoped, short-lived, or per-project memory shrinks the attack surface without any per-turn cost, and it reduces the blast radius of the false positives you will inevitably ship. If a memory does not need to persist, deleting it beats filtering it.

A hypothetical illustration, with assumptions stated: a coding assistant with a file-backed memory store that ingests repository content, issue text, and web fetches has exactly the cross-trust-boundary profile that warrants gated screening on those sourced memories, plus provenance on everything. The same assistant’s per-session scratch notes, never exposed to external content, are a reasonable candidate for lifetime limits instead of filters. This is an application of the rule, not a measured result on any such product.

Where these numbers don’t travel

The limitations are substantial and should discipline how you use every figure above. The cost study ran half of the LoCoMo benchmark: five of its ten long conversations, each truncated to 150 turns, 255 questions per replicate, 765 graded answers per condition (source). One lab, one hosted backbone without a pinned version, no independent replication. The attack studies used graph memory, agentic episodes, and web-agent memory; none used the conversational setup from the cost paper. The studies do reach a production memory framework: the cost paper itself ran Mem0 2.0.15 over a local Qdrant vector store, ShadowMerge poisons Mem0 graph memory, and TMA-NM runs on a Mem0-backed store whose origin labels survive Mem0’s consolidation. What no study evaluates is file-backed memory, an agent whose persistent memory is a directory of markdown files rather than a vector or graph store. Whether a reranker drops 2.2% of memories in a file-based store (source), or what provenance even means there, is unknown.

Class-level cost labels may also not transfer across implementations: the cost paper’s defenses are minimal implementations of published ideas, and the authors note that stronger implementations may cost more or less (source). Treat “anomaly detection costs 1.5%” as a measurement of one implementation on one benchmark, not a property of the category.

Practical verdict

Scope memory first, screen selectively, and leave the free structural defenses on. Concretely: enable sanitization and provenance (and origin binding where your architecture allows) on all persistent memory; gate anomaly detection and rerankers to memory with genuine exposure to untrusted content; and prefer shorter memory lifetimes over screening everything that persists. When evaluating any of this, demand false-positive rates alongside catch rates, and do not trust accuracy deltas smaller than about 2 points unless your intervals are tighter than the 2.4 to 4.6 points this study achieved. Finally, remember ShadowMerge’s lesson: your agent answering benign questions correctly tells you nothing about whether its memory has been poisoned. That is the case for paying some tax. The cost paper’s contribution is showing the tax, chosen well, is mostly not paid in tokens.

Frequently Asked Questions

What is the token overhead for stacking all four memory-poisoning defenses?

Token overhead was 0% for sanitization and provenance, 1.5% for anomaly detection, 3.0% for the reranker, and 4.5% with all four stacked.

How often did the reranker withhold memory from questions in the study?

The reranker had no detectable accuracy cost, yet it withheld memory from 14.9% of questions. In the audit, 26 of 50 drops removed a memory relevant to the question, but only 3 of those turned a correct answer wrong.

Scope memory first, screen selectively, and leave the free structural defenses on. Concretely: enable sanitization and provenance (and origin binding where your architecture allows) on all persistent memory; gate anomaly detection and rerankers to memory with genuine exposure to untrusted content; and prefer shorter memory lifetimes over screening everything that persists.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. The Price of Safetyarxiv.orgAccessed
  2. ShadowMergearxiv.orgAccessed
  3. TMA-NMarxiv.orgAccessed
  4. Environment-injected poisoning studyarxiv.orgAccessed
  5. Systematic study of memory poisoning attacksarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy