How AI Agents Remember: Memory Architectures Built for Production

On LongMemEval, a benchmark for long-term interactive memory, Mastra’s observational memory scored 94.87% with GPT-5-mini while holding average context near 30,000 tokens (VentureBeat). That result frames the whole question of agent memory: remembering is an architecture problem, not a model property. IBM’s overview, drawing on the CoALA taxonomy from Princeton, sorts that architecture into short-term, episodic, semantic, and procedural memory, implemented across context windows, vector databases, knowledge graphs, and training itself. Which combination you build decides whether your agent accumulates knowledge across sessions or resets each time the window fills.

What Is AI Agent Memory?

An agent’s memory is every mechanism by which past information influences future behavior. It is not one system but a set of stores with different read and write costs, capacities, and failure modes.

Two baseline facts from IBM’s documentation anchor the field. First, large language models “cannot, by themselves, remember things”; the memory component must be added (IBM). Second, not every agent needs it: a simple reflex agent or a basic thermostat acts on current input alone, while a memory-equipped smart thermostat learns patterns and adapts. Memory earns its cost when personalization, feedback loops, or learning across sessions matter, and IBM flags retrieval efficiency as the central design challenge, since storing too much slows responses.

The CoALA-derived taxonomy practitioners use:

• Short-term memory: recent inputs held for immediate decisions, typically a rolling buffer or context window. Session-scoped; nothing survives after it.
• Episodic memory: records of specific events, actions, and outcomes, logged in a structured format the agent can consult. The basis of case-based reasoning.
• Semantic memory: generalized facts, definitions, and rules, implemented through knowledge bases, symbolic AI, or vector embeddings.
• Procedural memory: skills and learned behaviors, acquired through training, often reinforcement learning, rather than runtime writes.

Long-term memory is the umbrella over the last three, implemented in databases, knowledge graphs, or vector embeddings, with retrieval-augmented generation among the most common techniques (IBM).

How Each Memory Type Works

Short-Term Memory: The Context Window

The context window is the only memory an LLM has natively; every token outside it does not exist for the model. Windows keep growing: Anthropic says Claude Fable 5 “stays focused across millions of tokens in long-running tasks” (Anthropic). Capacity alone does not settle the problem. Weaviate’s context engineering guide is blunt: it is “tempting to assume that shoving everything into bigger context windows solves this problem, but this is generally not the case,” which is why production agents need deliberate memory architecture (Weaviate).

Weaviate names four failure modes that appear as context grows, and none is fixed by size:

• Context poisoning: incorrect or hallucinated information enters the context, and because agents reuse and build on that context, the errors compound.
• Context distraction: too much history and too many tool outputs push the model to repeat past behavior instead of reasoning fresh.
• Context confusion: irrelevant documents or tools crowd the window until the model picks the wrong one.
• Context clash: contradictory information leaves the model stuck between conflicting assumptions.

Episodic Memory: What Happened, When

Episodic memory preserves events in sequence: what the user said, what the agent did, what happened next. MemGPT, the design behind Letta, introduced the memory hierarchy most systems now copy, modeled on an operating system. The agent actively manages what stays in core memory, the context window, versus external layers for conversation history, archival storage, and files, paging between them to get effectively unlimited memory within a fixed window (Letta).

The current version of that idea is the Letta Filesystem. Attach a file of conversation history and the file is parsed and embedded automatically; the agent receives file operation tools (grep, search_files, open, close) and searches iteratively until it can answer. Crucially, the agent generates its own queries rather than searching the original question verbatim, which is the behavior that lets simple tools beat specialized ones (Letta).

Semantic Memory: What Is True

Semantic memory abstracts episodes into durable facts and relationships. This is where vector databases (similarity retrieval) and knowledge graphs (relational structure) live, and where RAG fits as a retrieval technique over a corpus (IBM).

Commercial memory services such as Mem0, LangMem, and Zep offer this layer as a pluggable service, in Letta’s description: tools the agent calls to store and retrieve information, often backed by vector databases or specialized knowledge graphs (Letta). Treat their published comparisons carefully. When Mem0 reported running MemGPT on the LoCoMo benchmark, the Letta team, which built MemGPT, says it could not determine how the numbers were computed and that Mem0 did not respond to requests for clarification (Letta). The lesson is not that the products fail; it is that memory benchmarks are unusually easy to shade.

Procedural Memory: How to Do Things

Procedural memory is skills: routines, workflows, and decision steps that let an agent act without re-deriving everything. IBM places its acquisition in training, often reinforcement learning, which makes it the slowest tier to update (IBM). Weaviate also lists procedural data among the contents of long-term stores, so runtime-writable procedure is possible (Weaviate).

Letta’s benchmarking adds a wrinkle: agents use tools that resemble filesystem operations unusually well because those operations are dense in training data (Letta). The skill lives in the weights; the record the skill operates on lives outside them. That division, model-held skill plus external record, is the practical shape of procedural memory today.

Architecture Comparison

Row details come from VentureBeat, Letta, and IBM.

What Practitioners Are Actually Building

How Observational Memory Works

Mastra’s observational memory, shipped in Mastra 1.0 with plug-ins for LangChain and Vercel’s AI SDK, splits the window into two blocks: a dated observation log and the raw history of the current session (VentureBeat). When unobserved messages hit 30,000 tokens, the Observer agent compresses them into new observations and the originals are dropped; when the log itself reaches 40,000 tokens, the Reflector restructures and condenses it, merging related items and discarding superseded ones. Both thresholds are configurable. Compression runs 3-6x on text and 5-40x on tool-heavy output (VentureBeat).

The mechanism differs from the compaction most coding agents use. Compaction fills the window, then summarizes everything in one expensive pass, producing documentation-style prose that loses specific decisions and tool interactions. The Observer runs more often on smaller chunks and produces an event-based decision log that survives reflection rather than collapsing into a blob.

The economics come from caching. Anthropic, OpenAI, and other providers discount cached prompts by 4-10x, and most memory systems cannot exploit that because injected retrieval results change the prompt every turn (VentureBeat). An append-only observation prefix is stable, so every turn before a threshold is a full cache hit. Mastra’s LongMemEval run averaged around 30,000 tokens of context, and the system scored 84.23% on GPT-4o, versus 80.05% for its own RAG implementation on the same model (VentureBeat), and 94.87% on GPT-5-mini.

Graphs, Vectors, and What the Model Can Use

For questions about relationships, how an account changed, how two entities connect, graphs are the natural store, and IBM lists knowledge graphs alongside vector embeddings as standard long-term implementations (IBM). Letta’s results complicate the easy version of that advice. On LoCoMo, a benchmark built from long timestamped conversations between two speakers, a Letta agent with plain file tools and GPT-4o mini scored 74.0%, above the 68.5% Mem0 reported for its graph variant (Letta). The team attributes the gap to tool usability: agents call filesystem operations well because training data is full of them, while more elaborate machinery can cost more in model confusion than it returns in retrieval precision.

Where the Frameworks Fit

IBM describes LangChain as the integration layer combining memory, APIs, and reasoning workflows, with vector databases doing the storing, and LangGraph as the way to build hierarchical memory graphs that track dependencies and support learning over time (IBM). One recent API detail matters for anyone designing long-running agents: alongside Opus 4.8, Anthropic’s Messages API began accepting system entries inside the messages array, so developers can update an agent’s instructions mid-task without breaking the prompt cache (Anthropic). Instruction updates and cache stability no longer fight each other.

Where Memory Systems Break

Weaviate’s four failure modes describe the window; the long-term store adds its own, and it is the inverse of the obvious one. “The worst memory system is the one that faithfully stores everything,” Weaviate argues: old, low-quality, or noisy entries come back through retrieval and contaminate the context with stale assumptions (Weaviate). Effective systems are selective at write time, letting the model reflect on an event and assign an importance score before storing it. They then maintain what they keep: pruning, merging duplicates, deleting outdated facts, and replacing long transcripts with compact summaries, using recency and retrieval frequency as signals.

The Benchmark Landscape

Two benchmarks dominate this space, and they measure different things. LongMemEval tests long-term interactive memory; Mastra’s scores above are the current reference points. LoCoMo tests retrieval from long conversations (Letta). Numbers do not transfer between them:

All rows come from VentureBeat and Letta. Keep Letta’s methodological warning next to any such table: comparing memory tools is apples to oranges because framework, tools, and model all vary, which is why their Memory Benchmark holds the framework constant and varies only the model (Letta).

Frontier models narrow part of the gap without closing it. Opus 4.8, priced at $5 per million input and $25 per million output tokens (Anthropic), is about four times less likely than its predecessor to let flaws in its code pass unremarked, and early testers report it flags uncertainty more readily. That trims one class of compounding error. Fable 5, at $10 input and $50 output per million tokens, launched June 9, 2026, was suspended June 12, and was redeployed July 1 (Anthropic). It “stays focused across millions of tokens in long-running tasks and improves its outputs using its own notes.”

The memory detail in that announcement is the telling one. When Anthropic had Fable 5 play Slay the Spire, access to persistent file-based memory improved its performance three times more than it improved Opus 4.8’s (Anthropic); Fable also reached the game’s final act three times more often with that memory attached. The model with the most context capacity benefited the most from external storage, the same file-based pattern Letta’s benchmark surfaced. Longer windows raise the ceiling for in-context work, but the payoff from a well-designed external store is growing, not shrinking.

Related reading: Claude Fable 5 vs Opus 4.8: when the 2x price is worth it, plus our breakdowns of Hindsight-style memory architectures and council mode on HaluEval.

Choosing an Architecture

• Single session, bounded context: window management with compression. Accept that compaction-style summaries lose event-level detail (VentureBeat).
• Preferences across sessions: extraction-based memory with conflict handling. Budget for your own evaluation; published tool comparisons are contested (Letta).
• Long-running embedded agents: observational memory. The caching advantage compounds with call volume; a 4-10x discount on cached prompts (VentureBeat) is the difference between a predictable cost curve and an unpredictable one.
• Relational, evolving state: graphs where the queries are relational, paired with an agent that demonstrably drives the tool well.
• Auditable, resumable workflows: graph-orchestrated state of the LangGraph kind, which IBM credits with tracking dependencies and supporting learning over time (IBM).

Mastra’s CEO Sam Bhagwat ranks memory alongside tool use, workflow orchestration, observability, and guardrails as the primitives high-performing agents need (VentureBeat). The benchmarks and the failure modes argue the same thing: the model supplies attention, the architecture supplies continuity.