Context ordering, not window size, is now the binding constraint on long-context agent quality: the ARBIGRAPH benchmark1 shows a tool-assisted agent losing up to 33.3%1 accuracy on dependent task chains even when everything fits in the window. The lesson is mechanical rather than vendor-specific: attention distributes a finite relevance budget across every token, and the order in which context arrives changes what the model can act on.
Why did the bottleneck move from window size to ordering?
Because million-token windows made “does it fit” a solved problem while “does the model use what fits” got measurably worse. The full prompt, conversation history, injected context, and generated response must all fit inside a token budget, and longer prompts cost more while leaving less room for the response, according to Microsoft’s LLM Fundamentals documentation. Once fitting stopped being the constraint, the failure mode moved to what the model actually attends to inside the window.
Three mechanical facts drive this. First, inference is autoregressive: the model reconsiders the entire token sequence for every token it generates, so position and ordering are not cosmetic. Second, attention assigns soft weights via multiple heads, each with its own relevance function. Attention capacity, not raw window length, determines what a model can use. Growing the window by orders of magnitude did not grow attention’s effective selectivity by the same factor.
The result is a mismatch between advertised capacity and usable capacity. The window advertises a million tokens of headroom, but the attention mechanism distributes a finite amount of relevance weight across every one of them. Doubling the window does not double the relevance budget; it halves the average weight per token if the model has not been trained to allocate selectively at that length. Two agents with identical context windows and identical retrievers can post very different accuracy numbers because the better one is not retrieving more, it is retrieving in an order the model can act on.
Third, the economics punish sloppy ordering. Pricing is per-token, windows are measured in tokens rather than words (roughly one token per three-quarters of an English word), and longer prompts cost more while leaving less room for the response. Every token you prepend is a token you pay for on every request. A million-token window is a budget you can spend badly.
What does context engineering actually cover?
Context engineering is the discipline of deciding what goes into the window, in what order, and truncated how. It covers four surfaces: the system prompt, conversation history, tool outputs, and memory retrieved from outside the session. Each changes at a different rate, which is why this is an engineering problem rather than a writing problem.
A useful ordering principle falls out of volatility. Static content (system prompt, tool schemas) belongs at the front, where it stays stable across requests. Volatile content (latest tool output, fresh retrieval) belongs at the back, where it changes without disturbing what came before. Teams that interleave timestamps or session IDs into their system prompt churn a stable prefix for the privilege of a header nobody reads.
The four surfaces have different volatility profiles and earn different treatment. System prompts change between releases, not between requests, so they belong at the front. Tool schemas are static within a deployment and sit alongside the system prompt. Conversation history grows monotonically and should be appended rather than rewritten, so earlier turns stay where the model already saw them. Retrieved documents and tool outputs are the volatile tail: they change every request and should sit last. Mixing a volatile segment into the middle of stable content is the single most common ordering mistake an agent team can make.
Where should you spend your context budget?
Spend it on the axis that your own measurements say is failing, because retriever quality, truncation strategy, segment placement, and ordering are independent knobs with independent failure modes. The decision axes, as the current generation of agent frameworks frames them, are: budget allocation across system prompt, history, and tool outputs; static versus volatile content placement; tool-output truncation and ordering; retriever quality versus ordering quality; and portability of heuristics across model families.
A few of these deserve unpacking because teams routinely conflate them:
Retriever quality versus ordering quality. A retriever can return exactly the right five documents and the agent can still fail because the documents are concatenated in an order that buries the dependency the next step needs. Improving embedding models does nothing for this. Conversely, perfect ordering cannot recover documents the retriever never surfaced. These axes fail independently and must be instrumented independently.
Tool-output truncation. Tool outputs are the fastest-growing and least curated part of most agent contexts. A shell command that returns tens of thousands of log lines gets truncated somewhere; whether you keep the head, the tail, or an error-filtered middle is an ordering decision that changes what the model can conclude. The naive default (keep the head) is usually wrong for logs, where the diagnostic line is at the end, and right for JSON schemas, where the fields declare themselves up front.
History compaction. Long agent sessions accumulate history that must eventually be summarized or dropped. Autoregressive attention means a stale intermediate result retained in history keeps exerting soft attention weight on every subsequent token. Discarding is a feature, not a loss. Production agent codebases already treat this as a structural layer rather than an afterthought: the Claude Code architecture paper documents a five-layer compaction pipeline for context management.
How do you instrument an ordering change?
The tip above is not decorative. Without per-segment token accounting, every ordering change is a coin flip with a dashboard attached. The minimum viable instrumentation is per-request token counts for each segment (system, history, tool outputs, retrieval), plus per-request latency and task-level outcome. With those signals logged you can answer the question of which segment moved when accuracy dropped. Without them, every refactor of the prompt builder looks like progress because nobody can prove it isn’t.
The second piece is an eval that exercises dependency, not just retrieval. Single-turn QA benchmarks will not surface ordering failures because a single turn does not chain. You need a task graph where a later step depends on a value computed in an earlier step, with distractors inserted between them. ARBIGRAPH is one public instantiation of this shape; an internal version can be smaller. The point is to measure whether the agent retains the intermediate result through the distractor and uses it correctly downstream. Run this eval before and after any change to ordering or truncation, and gate the change on the delta. A change that improves latency while degrading chain accuracy is a regression, not a win, even if the latency dashboard looks better.
Several failure patterns show up repeatedly in teams that skip this instrumentation. The first is the silent stability regression: a developer interleaves a dynamic field into the system prompt for observability, the once-stable prefix now shifts on every request, and the team can no longer tell whether an accuracy change came from their fix or from the prompt restructuring. The second is the ordering regression that looks like a retriever regression: a refactor reorders retrieved chunks by recency instead of relevance, chain accuracy drops, and the team spends a week tuning embeddings when the fix is a git revert. A third is truncation drift: a tool’s upstream output format changes, the truncation policy tuned to the old shape now keeps the wrong slice, and chain accuracy degrades with no prompt change on your side. The tool-output segment’s token count stays roughly constant, so a size-only per-segment dashboard will not catch it; a per-segment log that also records a content hash, or a sampled diff against the last good run, will. All three are diagnosable in minutes from per-segment token logs and are invisible without them.
What does the evidence say, and where does it run out?
The strongest current evidence is ARBIGRAPH (arXiv:2607.20764), posted 2026-07-22, which tests whether tool-assisted agents can retain, update, compose, and discard task context across arbitrarily scalable verifiable task graphs. Its headline finding: a Qwen3.5-27B agent’s accuracy dropped by up to 33.3%1 on branching chains of dependent math tasks. These are tasks where every intermediate result fits in the window and every step is individually verifiable, so the accuracy collapse is attributable to context management rather than to capacity or reasoning.
That result matters for two reasons. It isolates retention, update, and discarding as the measurable bottleneck for tool-assisted agents, exactly the surfaces context engineering targets. And it shows the failure is workflow-dependent: heuristics that hold on single tasks or linear chains collapse on branching dependency structures. Any fixed recipe, vendor-supplied or otherwise, is a hypothesis about your workload’s graph shape until you measure it there.
ARBIGRAPH evaluates the agent across four topologies, and the degradation is not uniform across them. Isolated tasks score well, which is the trap: a benchmark that only tests isolated tasks will report that the agent is fine. The collapse shows up on branching chains, where the agent must hold an earlier computed value alive across intervening steps that do not use it, then retrieve it for a later step that does. That is the access pattern a real coding or research agent produces: compute a fact, do unrelated work, come back and use the fact. If your eval does not contain that shape, your accuracy number is a best case, not a typical case.
The counter-evidence deserves equal weight. The LSRM paper is sometimes cited in these discussions: scaling context via native sparse attention let a reconstruction model handle 20x more object tokens and 2x more image tokens with over 2.4dB higher PSNR. But that is a 3D-vision reconstruction result, not text-agent evidence.
Do ordering heuristics port across model families?
No, not blindly. Treat ordering, truncation, and segment placement as first-class, measurable engineering knobs and validate their effect on your own workload and model, rather than importing a recipe tuned elsewhere. Any vendor-published ordering heuristic is tuned to that vendor’s attention profile. Attention-head structure, position-encoding behavior, and training-data ordering conventions differ across model families, and there is no independently measured attention-pattern comparison showing a given ordering prescription survives porting across vendors or to Qwen-class open-weight runtimes.
ARBIGRAPH is the reason this caution is structural rather than pedantic. If a fixed context-management strategy loses a third of its accuracy when the dependency graph branches, then a recipe derived on one set of internal evals encodes assumptions about task shape, tool-output distribution, and model attention that your stack almost certainly violates somewhere. Porting the recipe without re-deriving it is copying the answer key for a different exam.
What differs across model families is not cosmetic. The number of attention heads and their dimensionality change how much parallel relevance the model can hold at once. Rotary and learned position encodings decay differently with distance, so a rule like “put the most important instruction last” can help on one family and hurt on another. Training-data conventions, such as whether the model saw system prompts at the front or instructions interleaved with context, shape where the model learned to look for decisive instructions. A vendor’s published heuristic is a summary of where its own model learned to look, transcribed into prose. Applying it to a model that learned to look somewhere else does not move the model; it moves the context away from where the model is looking.
The actionable path is boring and works: segment your context budget and log it; place static content ahead of volatile content; pick tool-output truncation policies per tool type rather than globally; and run a dependent-chain eval in the spirit of ARBIGRAPH against your actual agent before and after any ordering change. The window stopped being the constraint a while ago. What you do inside it is now the whole game, and the scoreboard is your own eval, not the vendor’s.
What ordering policies survive across model families?
A few ordering rules are robust defaults across vendors because they keep the prompt’s structure stable from request to request. Static content goes first. Volatile content goes last. Tool schemas sit with the system prompt and never move between requests. Tool outputs are truncated by content type, not by a global line limit. Conversation history is appended, not rewritten.
The rules that do not transfer are the attention-tuned ones: how far from the end to place the decisive instruction, whether to repeat the system instruction after a long context, how to order retrieved documents by relevance, and how aggressively to compact history. These are empirical questions for your model, on your workload, measured against your eval. Treat vendor guidance on these points as a starting hypothesis, not a setting.
Frequently Asked Questions
How does ARBIGRAPH measure context retention failure differently from standard benchmarks?
ARBIGRAPH uses arbitrarily scalable verifiable task graphs to test retention, update, composition, and discarding across branching chains. Standard single-turn QA benchmarks fail to surface ordering failures because they do not chain dependent steps, whereas ARBIGRAPH inserts distractors between computation and retrieval to measure whether an agent holds an intermediate result alive across unrelated work.
What is the practical impact of KV-cache mechanics on prompt caching strategies?
Inference engines use a KV-cache to reuse intermediate computations from prior tokens, making the prefill phase slower than the decode phase. This mechanical difference is the basis for prompt caching, where placing stable content at the front allows the engine to skip reprocessing those tokens, directly reducing latency and cost for repeated requests.
Why might a vendor-specific ordering heuristic fail when ported to an open-weight model?
Vendor heuristics are tuned to specific attention profiles, including the number of attention heads, dimensionality, and position-encoding behavior like rotary or learned encodings. These architectural differences change how a model allocates relevance weight across distance, meaning a rule that helps one family can hurt another if the target model learned to look for instructions in a different part of the sequence.
What are the risks of using a global truncation policy for tool outputs?
Tool outputs vary significantly in structure, such as JSON schemas versus log files. A global truncation policy often keeps the head of a log, burying the diagnostic line at the end, while a schema benefits from head retention where fields declare themselves upfront. Truncation must be tailored per tool type to ensure the model sees the most actionable slice of the output.
