The Million-Token Context Window: What Can You Actually Do?

Million-token context windows let you feed entire codebases, legal contracts, and hours of audio to an LLM in a single prompt. In practice, most models lose coherence well before their advertised limit—and the middle of your document is where accuracy degrades most. The sweet spot for reliable use sits around 60–70% of the claimed maximum, with specific failure modes that every practitioner needs to understand before building on top of these capabilities.

The Long-Context Landscape in 2026

Ultra-long context has gone from a Google research demonstration to a mainstream spec in under two years. In early 2024, a 200K window was a premium differentiator. By early 2026, several models offer 1 million tokens or more as standard, and outliers push into double-digit millions.

The open-weight story is notable: Meta’s Llama 4 Scout reaches 10 million tokens—the same as Google’s premium offering—while running on a 17B active parameter mixture-of-experts architecture. Alibaba’s Qwen2.5-1M enables self-hosted million-token processing for organizations with the infrastructure to run it.¹

What You Can Actually Do Well

Full-Repository Code Analysis

The clearest real-world application is codebase comprehension. Gemini 1.5 Pro can process codebases exceeding 30,000 lines of code in a single pass, and tools like Repomix—nominated for the JSNation Open Source Awards in 2025—exist specifically to pack entire repositories into a single AI-readable file targeting Claude, Gemini, GPT-4, and DeepSeek.²

The practical workflow: serialize your codebase to a single file, feed it with a precise question about architecture, dependency chains, or bug location. This eliminates the fragmentation problem of RAG pipelines where relevant context is split across retrieved chunks. For debugging complex interactions between files, a 200K+ context with the full codebase loaded often outperforms a RAG system that retrieves individual files in isolation.

Llama 4 Scout’s 10M window is positioned explicitly for “reasoning over vast codebases”—large enough to handle most real enterprise monorepos in a single context.³

Legal Due Diligence

Long-context LLMs are entering legal workflows as auxiliary tools for due diligence and contract analysis. One documented pattern: investment funds use LLMs to search large document sets for answers using a query set defined by human analysts and lawyers, with the LLM acting as a fast first-pass filter.⁴ Claude 3.5 Sonnet’s 200K context handles complex legal texts—full merger agreements, regulatory filings, multi-party contracts—that previously required chunking, with its accuracy degradation across its full range cited as less than 5% in some evaluations.

The implementation standard across serious legal deployments involves mandatory human review of LLM outputs. The LLM accelerates review, not replaces it.

Multimodal Long-Context Processing

Gemini 1.5 Pro demonstrated the most striking capability here: processing one hour of video and eleven hours of audio in a single inference call.⁵ Near-perfect retrieval (>99.7% recall) on multi-needle tasks across the full 1M token window in Google’s technical report makes this the most validated long-context application in the research literature.

For audio, practical applications span doctor dictation processing, interview transcription at scale, and long-form podcast analysis—workflows where maintaining context across an entire session matters more than per-segment accuracy.

Where Long Context Actually Breaks Down

Lost in the Middle

The foundational research on long-context failure comes from Nelson Liu et al. at Stanford and the University of Washington, published in Transactions of the Association for Computational Linguistics in 2024.⁶ The paper documents a U-shaped performance curve: accuracy peaks when relevant information appears at the beginning or end of the context, and degrades when critical information sits in the middle.

In extreme cases, models with long context prompts performed worse than zero-shot prompting with no context at all. Performance drops by more than 30% when relevant information shifts from position-optimal (start or end) to middle placement.

The root cause is structural: Rotary Position Embedding (RoPE) introduces a long-term decay effect that causes models to systematically de-emphasize middle-context tokens. This isn’t a bug that will be patched—it’s a property of the attention mechanism itself.

Context Rot

Chroma Research’s “Context Rot” study evaluated 18 state-of-the-art models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, finding that reliability decreases significantly with longer inputs even on simple retrieval and text replication tasks.⁷ The degradation accelerates when:

• Question-evidence similarity is low (the answer looks semantically different from the question)
• Distractors in the context are semantically similar to the answer
• The task requires two-hop reasoning rather than direct retrieval

Adobe researchers in February 2025 demonstrated this empirically: multi-hop reasoning across long contexts degrades faster than single-hop retrieval, and the degradation compounds with each additional reasoning step.

The framing that crystallizes the problem: each new token in the context depletes a finite “attention budget,” diluting the model’s capacity to attend to relevant information. A million-token context doesn’t give you a million tokens of reliable attention—it distributes a fixed attention capacity across a much larger pool.

The Effective Window Gap

A 2025 paper (arXiv: 2510.05381) found that even when models can retrieve all relevant information, performance still degrades by 13.9% to 85% as input length increases within the model’s claimed context limits.⁸ The advertised number and the effective number diverge substantially. GPT-4o, with a 128K advertised window, has shown strong primacy and recency effects with only approximately 8K effective tokens for certain tasks—the middle 70–80% of the prompt is largely ignored.

The Performance Gap: Benchmarks Confirm the Theory

RULER (arXiv: 2404.06654), NVIDIA’s synthetic benchmark released in 2024, evaluates long-context models across 13 tasks at lengths from 4K to 128K tokens.⁹ The key finding: “Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as context length increases.” Simple needle-in-a-haystack retrieval flatters model capabilities; any task requiring aggregation, multi-hop reasoning, or variable tracking reveals the degradation earlier.

The canonical Needle-in-a-Haystack test results illustrate the spread:

• Gemini 1.5 Pro: 99% recall at 1M tokens; >99.7% on multi-needle tasks (100 unique needles) at 1M tokens; 99.2% at 10M tokens⁵
• GPT-4: Accuracy drops significantly when processing sequences beyond 10% of maximum capacity (BABILong benchmark)
• Sequential-NIAH (EMNLP 2025): A harder variant requiring sequential needle extraction saw the best-performing model reach only 63.5% accuracy on 8K–128K contexts¹⁰

The gap between Gemini’s retrieval performance and the field illustrates that long-context quality varies enormously by model and task. Advertised context length is not a proxy for long-context capability.

The Cost Reality

Pricing as of early 2026; check provider documentation for current rates.

The cost structure creates a meaningful architecture decision. A 500K token prompt to Claude Sonnet 4 crosses into the extended-context pricing tier, costing $3.00 for that single input call. For high-volume applications, that arithmetic favors retrieval-augmented generation over full-context loading—even when the context window technically fits.

The infrastructure cost is just as significant. A 128K context prompt on Llama 3.1-70B consumes roughly 40GB of HBM just for KV cache storage. At 1M tokens, KV cache requires approximately 15GB per concurrent user. This means million-token context serving is GPU-memory-intensive per session, not just per-token computationally.

Latency compounds the challenge. A 100K token prefill on a 70B model takes 8–10 seconds of GPU compute. Prefill latency exceeds two minutes for maximum context lengths without optimization on current hardware. Microsoft Research’s MInference technique (NeurIPS 2024 Spotlight) reduces 1M-token prefill on LLaMA-3-8B from 30 minutes to 3 minutes on a single A100 GPU by exploiting sparse attention patterns—still not instant, but a 10× improvement.¹¹

Engineering Around the Limits

Given the failure modes, production long-context applications use a set of mitigations:

# Structuring prompts to maximize long-context reliability
system_prompt = """
[CRITICAL CONTEXT - READ FIRST]
{most_important_facts}

[BACKGROUND DOCUMENTS]
{long_document_body}

[TASK - REFER BACK TO CRITICAL CONTEXT]
{question_with_reference_to_key_facts}
"""

Placing critical information at both the beginning and end of the context—the “bookending” pattern—directly counters the lost-in-the-middle effect. For tasks requiring synthesis across the full document, chunked analysis with a final aggregation pass often outperforms single-pass full-context processing.

For cost-sensitive applications, Microsoft’s LLMLingua achieves up to 20× prompt compression with minimal accuracy loss—reducing a 1M token input to 50K before inference while preserving the semantic content that matters.¹²

Frequently Asked Questions

Q: What’s the practical maximum context length for reliable LLM performance? A: At time of writing, approximately 60–70% of a model’s advertised maximum. For a 200K token window, budget for reliable performance up to roughly 130K tokens. Gemini 1.5 Pro is the notable outlier, with near-perfect retrieval demonstrated at 1M tokens in its technical report—but for most models and most tasks, the effective window is substantially shorter than the spec sheet.

Q: Is a million-token context window better than RAG for codebase analysis? A: For moderate-sized codebases that fit within the effective context window, direct ingestion outperforms RAG by preserving cross-file relationships that retrieval systems miss. For enterprise monorepos exceeding even 10M tokens, RAG or intelligent snapshot tools remain necessary. The right approach depends on codebase size and query type.

Q: Why does LLM performance degrade in the middle of long contexts? A: Rotary Position Embedding (RoPE), which most transformer models use to encode token positions, introduces a long-term decay effect that systematically de-emphasizes middle-context tokens. The architecture is optimized for primacy and recency. Research mitigation strategies include sparse attention mechanisms and modified position encodings, but no production model has fully eliminated this property.

Q: How much does a 1M token prompt cost across major providers? A: At current early 2026 rates: approximately $6.00 for input on Claude Sonnet 4 (in the extended-context tier), $1.25 on GPT-4.1, and $0.075 on Gemini Flash-Lite. Output tokens are priced 4–8× higher than input tokens across the market, so for analysis-heavy workloads with short outputs, input cost dominates. DeepSeek’s self-hosted pricing at ~$0.48/M tokens disrupts the economics for on-premise deployments.

Q: Can open-source models handle million-token contexts effectively? A: Meta’s Llama 4 Scout officially supports 10M tokens, and Alibaba’s Qwen2.5-1M supports 1M for self-hosting. The infrastructure cost is the barrier: a 1M token context requires approximately 15GB of KV cache memory per concurrent session, making this practical only for organizations with dedicated GPU infrastructure. Optimization techniques like NVFP4 KV cache quantization cut memory requirements by up to 50%, but the hardware floor remains significant.