Compressing LLM Agent History: Pixel Rendering vs Summarization

Pixel rendering beats summarization only if its savings survive verification, and as of 2026-08-24 they have not: the abstract of AgentOCR (arXiv:2601.04786) reports author-claimed numbers (over 95% of text-agent performance preserved, over 50% token reduction) with no independent reproduction, no evaluation tables on the paper’s abstract page, and a “work in progress” tag on the paper itself. The decision that holds up on current evidence is less photogenic: externalize agent state, keep it textual, and inject a compiled briefing at spawn time.

Why do transcript tokens dominate long-horizon agent spend?

Because the standard agent loop re-sends its entire observation-action history with every new turn, the token cost of a run grows faster than the run itself. Each step appends tool output, reasoning, and actions to a transcript that the model must re-read in full on the next call. Carry N turns of history and you pay for roughly N²/2 history tokens over the run, before the model produces a single token of actual work. For short sessions this is noise. For agents that run for hours, history becomes the largest line item and also the largest consumer of context window, starving the task itself.

The pressure is visible in this month’s research output. AgentOCR, whose v3 was posted 21 August 2026 per its submission history, attacks history size directly by rendering it as an image. Consilience (arXiv:2608.20564) attacks the communication side, using round-wise conformal calibration to bound a controller’s one-step regret in multi-agent discussion, and reports improvements in both decision accuracy and communication efficiency on hidden-profile tasks across 12 open- and closed-weight models. Asymmetric structural conditioning (arXiv:2608.20786) attacks context format, and in a tender-response evaluation an LLM judge rated its structurally conditioned answers at least as good as the human-submitted answer on 40 of 55 ground-truth sections. Three papers, three different mechanisms, one shared bottleneck: the transcript.

The demand side explains why the bottleneck matters now. Who Delegates to AI? (arXiv:2608.20425) draws on roughly 53,000 agent skill specifications from the Manus Skills Marketplace and operationalizes exposure as an Agentic Adoption Index measuring how closely an occupation’s tasks match agentic routines practitioners have already built and shared. Delegation concentrates in particular occupations rather than spreading evenly, and every delegated routine inherits the quadratic transcript arithmetic above. That is the problem every strategy below is trying to cut.

What are the five ways to compress agent history?

Every deployed or proposed approach to agent-history compression is one of five strategies, distinguished by where the history lives and what gets lost when it shrinks.

Truncation. Keep the last K turns, drop the rest. Zero infrastructure, zero extra model calls, deterministic cost. The failure mode is silent: the agent forgets its original instructions or an early observation the task later depends on, and nothing in the transcript records what was dropped.

Rolling summarization. Periodically replace older history with a model-written summary. Prompt-only to implement, which is why it is the default in most agent frameworks. The summarizer call costs tokens, the summary is lossy in ways the summarizer chooses rather than you, and summaries of summaries drift. You are also paying a model to decide what the other model needed to know, and it decides without knowing what the task will ask for later.

Structured external state. Move durable facts out of the transcript entirely into a typed store: entity tables, task state, artifact pointers. The transcript carries only working context. This requires schema design and write discipline from the agent, but the state stays textual, queryable, and diffable.

Retrieval over external history. Keep full history outside the context window, index it, and pull relevant slices per turn or per spawn. Cost becomes proportional to what you retrieve rather than what happened. Recall quality is the tax: anything the retriever misses behaves exactly like truncated history, except the omission is harder to predict.

Pixel rendering. Render the transcript as an image and let a vision encoder read it back, on the theory that visual tokens carry more information per token than the rendered text would. This is the newest and least verified of the five, and it is what AgentOCR proposes.

What does AgentOCR actually propose?

AgentOCR represents an agent’s accumulated observation-action history as a compact rendered image, replacing raw text history in the context window, and the paper’s title calls this optical self-compression. Three mechanisms make up the proposal.

First, the rendered history itself. The agent’s transcript is drawn into an image, and the vision encoder consumes image tokens instead of text tokens. The authors claim visual tokens carry superior information density per token. Treat that as the authors’ framing: the abstract reports an over-50% token reduction alongside over 95% preserved text-agent performance on ALFWorld and search-based QA, but not the per-modality density ratio a deployer would need, and the evaluation tables are not on the paper page as of 2026-08-24.

Second, segment optical caching. History is decomposed into hashable segments, and a maintained visual cache means unchanged segments are not re-rendered each turn. The authors report a 20× rendering speedup from this mechanism, which is what they say makes multi-turn rollouts practical, since naive per-turn re-rendering would re-charge you the full visual encoding cost every step. Note what this optimizes: it cuts re-rendering overhead, not history content. The information question, what survives compression, is untouched by caching.

Third, agentic self-compression. The agent actively emits a compression rate, and the system is trained with a compression-aware reward to, in the paper’s language, adaptively balance task success and token efficiency. Read that sentence carefully, because it is the most honest part of the abstract. If pixel compression were free, no learned balance would be needed. The reward exists because compressing harder degrades task success, and someone has to decide how much degradation to buy how many saved tokens. The paper’s answer is: train the agent to decide.

The third mechanism also defines the deployment bar. Agentic self-compression is not a prompt-level trick you can adopt this weekend; it requires model-side training with a custom reward. Unless you fine-tune your own agent model, you are waiting for a vendor to ship it.

How does the token math compare across strategies?

The cost structure differs per strategy in ways you can compute before adopting any of them, using only your own workload’s shape. Take a worked example with explicitly stated assumptions: a 50-turn agent run where each turn appends roughly 1,000 tokens of tool output and 300 tokens of action and reasoning, so 1,300 new tokens per turn. These are illustrative numbers, not measurements from any cited paper; substitute your own logs.

Raw history. At turn t you carry about 1,300t history tokens. Summed across 50 turns, the run spends roughly 1,300 × (50 × 51 / 2) ≈ 1.66 million tokens just re-reading its own past. Nothing was generated yet; this is the cost of remembering.

Truncation at 10 turns. History carried per turn caps at 13,000 tokens. Total history spend drops to roughly 0.6 million tokens, and anything older than 10 turns is gone. Whether that matters depends entirely on whether your tasks need early state late in the run.

Rolling summarization. Every K turns you pay a summarizer call over the retiring segment, plus the summary tokens carried forever after. If a 1,000-token summary replaces each 13,000-token block, carried history grows at roughly a tenth the raw rate, plus the summarizer’s own input and output spend. The catch is that the compression ratio is chosen implicitly by the summarizer, and errors compound as summaries feed summaries.

External state plus retrieval. Carried history per turn is a fixed briefing, say 2,000 to 5,000 tokens of retrieved state, independent of turn count. The 50-turn run spends on the order of 100,000 to 250,000 history tokens, plus whatever the indexing and retrieval infrastructure costs in engineering rather than tokens. This is the only strategy whose per-turn cost does not grow with run length at all.

Pixel rendering. The text-to-visual token ratio is the entire question, and the AgentOCR abstract does not provide it; the reported over-50% token reduction is an end-to-end figure on the paper’s own benchmarks, not the per-turn density ratio this ledger needs. If the density claim holds, rendered history could undercut text history per turn while still growing with run length. Add per-turn vision-encoding cost, offset by segment optical caching (20× rendering speedup, author-reported) for unchanged prefixes. Until the evaluation tables supply the measured ratio, this row of the ledger is blank.

The arithmetic points one way: strategies that bound carried history per turn (truncation, retrieval, external state) convert a quadratic cost into a linear one, and strategies that compress within the window (summarization, pixel rendering) reduce the constant but keep the growth. For genuinely long horizons, bounding beats compressing.

When is lossy visual compression safe, and when does it drop what matters?

Lossy compression is safe exactly when the information it destroys is either irrelevant to the remaining task or cheaply reconstructible from what survives. An image of a transcript will preserve gist, layout, and approximate content. What it cannot be trusted to preserve is exact strings, and agent transcripts are full of strings where exactness is the whole point.

Consider what a coding or operations agent’s history actually contains: error messages with specific codes, file paths, commit hashes, diffs, configuration values, command output whose whitespace carries information, identifiers that must be reproduced byte-for-byte three hours later. A vision encoder reading back a rendered page is performing OCR-adjacent reconstruction on precisely the content least tolerant of reconstruction error. One transposed character in a hash or a flag is not a summary error; it is a wrong action taken confidently.

Pixel rendering is therefore safest for histories dominated by prose: research agents accumulating reading notes, monitoring agents whose early observations only matter as trends, exploratory runs where the path matters less than the destination. It is least safe for histories dominated by artifacts: code, logs, structured output, anything the agent will later need to quote rather than paraphrase.

The same test applies to rolling summarization, honestly applied. A good summary preserves what the summarizer judged important; pixel rendering preserves what the renderer and encoder managed to keep legible. Both fail silently. The difference is that a text summary can be audited against its source with a diff, and a rendered image cannot.

What is the incumbent alternative: external state and spawn-time briefings?

The production-proven alternative to compressing history is to never carry it: keep durable state outside the agent and inject a compiled briefing when a fresh instance starts. PrimeAgentOrchestrator (arXiv:2608.20342) is the concrete reference implementation, and it reports four months of regular deployment from December 2025 through March 2026 as an experience report rather than a benchmark claim.

The architecture is deliberately plain. PAO spawns new instances of Claude Code pre-loaded with relevant memories compiled from the user’s existing personal databases, instead of extending a single long session. At spawn time it queries two independently operated memory backends in parallel: a PostgreSQL entity-observation database for structured state, and a Cloudflare Worker semantic search index for retrieval over unstructured material. The compiled result is delivered by filesystem injection that exploits the host agent’s configuration auto-read behavior, so the agent itself needs no modification. No custom training, no vision encoder, no reward shaping.

Two things about this pattern deserve emphasis. First, it composes the two textual strategies from the taxonomy: structured external state for facts that must be exact, retrieval for everything else. The briefing is bounded per spawn, so cost per session does not grow with the history of the system, only with how much you choose to inject. Second, the state stays text. You can grep the PostgreSQL store, diff two briefings, and reconstruct exactly what the agent was told at spawn, which is the property the next section prices.

Keep the horizons straight, though. PAO’s spawn-time briefing solves cross-session memory: a fresh agent knowing what it needs. In-session compaction solves a different problem: a single long run hitting window limits mid-task. AgentOCR targets the second; PAO sidesteps it by making sessions shorter and state external. Many real workloads need both, and conflating them leads to buying a training pipeline to solve what a database would have.

What breaks when agent state stops being text?

The price of pixel-compressed history is paid in auditability, and the bill arrives long after the tokens are saved. Textual agent state is grep-able, diff-able, loggable, and replayable. When an agent does something wrong in a long run, the debugging workflow is: find the turn where it went wrong, read what it had seen, determine whether the input or the reasoning was at fault. That workflow assumes you can read the history the way the model did.

Once history is a rendered image, every step of that breaks. You cannot grep a bitmap for the error message the agent saw at turn 12. You cannot diff two cached segments to see what changed. You cannot prove to a compliance reviewer, or to yourself, what the agent actually saw, because what it saw is whatever the vision encoder decoded from pixels, which is not guaranteed to be the text you rendered. Segment optical caching makes the cache itself a new piece of state to audit: hashed segments tell you which chunk an image came from, not whether the encoding preserved its content.

This is not a hypothetical compliance nicety. Any environment where agent actions need justification after the fact (production changes, financial operations, anything with an incident review) currently assumes the transcript is evidence. Pixel history converts evidence into a rendering of evidence, with a lossy decode step in between that nobody signed.

The structural-conditioning result from the August cluster cuts in the opposite direction and is worth noting here: in the tender-response evaluation, structured context carried task-critical content well enough for an LLM judge to rate machine answers at least as good as human ones on 40 of 55 sections. Structure preserved what mattered because the structure was explicit and textual. The auditability argument is not against compression; it is against compression into a form you can no longer inspect.

Which compression strategy should you pick?

Externalize first, keep it textual, and treat pixel rendering as a research bet to revisit when its numbers are independently checked. The decision table below compresses the trade-offs; ratings are mechanism-based judgments, since head-to-head measurements across all five strategies do not exist in the cited material.

The practical verdict follows from the last column. PrimeAgentOrchestrator shows the external-state pattern running on plain infrastructure (PostgreSQL plus a serverless search index) with four months of reported regular use, deployable today without modifying the agent. AgentOCR proposes something genuinely novel, and its abstract now reports over 95% preserved performance with over 50% token reduction on ALFWorld and search-based QA, but those are author figures on a work-in-progress paper: agentic self-compression still requires model-side training with a compression-aware reward, the abstract itself concedes the need to trade task success against token efficiency, and no independent reproduction exists to weigh against the auditability you forfeit.

The strongest limitation of this comparison deserves stating plainly. Every AgentOCR number available as of 2026-08-24 is author-reported from the abstract of a work-in-progress paper: the evaluation tables are not on the paper page, the text-to-visual density ratio a deployer needs is not stated, and no independent reproduction or direct adversarial critique of pixel compression has appeared. The cross-strategy verdicts above rest on mechanism reasoning plus those unilateral figures, not measured head-to-head numbers. If the paper’s evaluation tables confirm large savings on real agent transcripts, the calculus shifts for prose-heavy, audit-light workloads. Until then, the boring architecture wins: put the state in a database, retrieve what each run needs, and keep the evidence in a form you can grep.

Frequently Asked Questions

Does AgentOCR’s segment optical caching reduce the total token count of the agent’s history?

No. Segment optical caching only reduces the computational overhead of re-rendering unchanged history segments, which the authors report as a 20x speedup. It does not compress the information content or reduce the number of visual tokens consumed by the vision encoder for the active context window.

How does PrimeAgentOrchestrator’s spawn-time briefing differ from in-session rolling summarization?

PAO addresses cross-session memory by injecting a compiled, bounded briefing from external stores (PostgreSQL and Cloudflare Workers) into a fresh agent instance, whereas rolling summarization compresses the current session’s transcript in-place. PAO avoids the compounding error drift inherent in summarizing summaries by resetting the context window entirely at each spawn.

What specific operational change is required to adopt AgentOCR’s agentic self-compression?

Adoption requires fine-tuning the agent model with a custom compression-aware reward function, as the mechanism relies on the model actively emitting a compression rate. It cannot be implemented via prompt engineering or configuration changes alone, unlike truncation or external state strategies.

Why is pixel rendering less suitable for coding agents than for research agents?

Coding agents rely on exact string matching for commit hashes, file paths, and error codes, where a single transposed character in a vision-decoded image leads to confident but incorrect actions. Research agents, which primarily need to preserve gist and trends from prose, are more tolerant of the lossy reconstruction inherent in optical compression.

What is the primary auditability risk of using rendered images for agent history?

Rendered history destroys the ability to grep, diff, or replay the exact input the model saw, converting textual evidence into a lossy bitmap. This breaks standard incident review workflows that require verifying whether an agent’s error stemmed from flawed input or flawed reasoning, as the original text is no longer directly inspectable.