2-Bit KV Cache Quantization: Memory You Save, Recall You Risk

A 2-bit KV cache is one of the largest single memory levers a serving team can pull without touching model weights: the KVQuant paper reports running a 1-million-token LLaMA-7B context on one A100-80GB GPU at that precision. The catch, measured by a rival method’s evaluation, is that direct 2-bit quantization measurably degrades long-context accuracy, and it can fail without any signal in your serving dashboards. Whether to enable it is a cache-configuration decision, and the answer depends on your workload’s recall sensitivity, not on the bit-width fashion cycle.

Why the KV cache, not the weights, caps your concurrency

Serving economics are batch economics. Weights are read once and amortized across every concurrent sequence; the KV cache is per-sequence, and it grows with context length. A 70B-class model at INT4 weights might occupy under 40 GB, leaving the remaining VRAM to be divided among concurrent requests’ key and value tensors. When each request carries a 128K or 1M-token history, that per-request cache, not the parameter count, decides how many sessions fit on a card. Groundy’s batching economics on a single RTX 3090 walks through the same arithmetic at smaller scale: as concurrency grows, per-sequence KV reads become the bandwidth tax.

That is why KV-cache compression attracts more attention from capacity planners than another round of weight quantization. The 2-bit weights question matters most for fitting a model on a card at all; the KV-cache question matters for how many requests that card can serve at once. Two routes compete for the same budget. One is architectural, replacing standard attention with linear or sparse variants that shrink the cache structurally, as in the Kimi Linear approach. That route is a model migration. The other is the subject here: quantize the cache of a model you already serve. It is a configuration change rather than a model migration, though not a stock engine flag; KVQuant and OptR both arrived with custom kernels rather than engine settings. The tradeoff it introduces is the subject of this article.

The capacity math: what 2-bit actually buys

The headline capacity figures come from KVQuant, which reports that compressing the KV cache to 2-bit precision enables 1M-token context inference with LLaMA-7B on a single A100-80GB GPU, and 10M-token inference on an 8-GPU system. Those are author-reported results on one model family, but the arithmetic behind them generalizes: keys and values stored at 2 bits instead of 16 is an 8× reduction in the per-token cache footprint, which translates roughly into 8× the concurrent sequences, 8× the context length, or some mixture of both, at fixed VRAM.

For a capacity planner, that ratio reframes procurement. If cache bytes are the binding constraint, moving from FP16 cache to a working 2-bit scheme is the equivalent of buying several times the GPU fleet for the same request load. Even the more modest measured envelopes matter. The WitCert paper ran its end-to-end serving evaluation on SGLang with an RTX 4090, batch 16 × 64 tokens and page_size=16, reporting capacity and throughput as medians of three runs. Consumer-card serving of long contexts is exactly the regime where cache compression decides whether a deployment exists at all.

The recall bill: direct 2-bit KIVI/KVQuant, measured

The strongest counter-evidence to a blanket 2-bit rollout comes from ReST-KV, a cache-eviction paper that benchmarked its competitors. Its evaluation reports that applying aggressive 2-bit KIVI or KVQuant directly to the full KV cache “results in significantly lower accuracy.” ReST-KV’s own results come from two separate experiments, and the paper does not fuse them. In its needle-in-a-haystack evaluation, a Llama3.1-8B-Instruct configuration matches full-cache accuracy while storing only 1/32 of the original tokens. On LongBench, the paper reports that combining ReST-KV with moderate 4-bit quantization retains high average accuracy at a 6.25% total compression ratio.

Two caveats belong with this finding, and both cut the same way for a reader. First, the negative result on 2-bit KIVI/KVQuant comes from a competing method’s paper, not a neutral benchmark; the authors had every incentive to show the direct-2-bit baseline at its worst. Second, even taking the finding at face value, it establishes a comparison between method classes on the paper’s test sets, not a universal law. What it does establish firmly is that the bit-width number alone does not determine quality: an eviction-plus-4-bit pipeline can match or beat the memory savings of naive 2-bit while preserving more accuracy. The compression ratio is not the decision variable. What you discard, and how, is.

Method map: three ways to shrink the cache

The methods compared here fall into three classes, and the differences are mechanistic, not cosmetic.

Distribution-aware quantization (KVQuant, KIVI) studies the statistical shape of keys and values and designs quantization grids around them: per-channel keys, per-token values, outlier handling, residual precision for recent tokens. These are the methods with the strongest published memory results and, per ReST-KV’s evaluation, the measured accuracy cliff at 2 bits when applied naively to the full cache.

Rotation-based INT2 is the newest entry. OptR, a preprint posted in August 2026, argues that prior rotation schemes optimize the wrong objective. Instead of minimizing key/value reconstruction error, OptR minimizes post-W_O attention-output error, decomposing that error into key-induced and value-induced terms and learning per-head orthogonal corrections through the full INT2 quantization and attention path. The intuition: what matters to the model is not whether the cache entries are individually faithful, but whether the attention output after the output projection is close to the unquantized result. It is a single unreplicated preprint, but it does report numbers: OptR improves AIME25 accuracy on Qwen3-8B from 17.33% to 66.67% with QuaRot and from 54.67% to 66.00% with OSCAR (source), against 68.00% for BF16, at an effective cache cost of 2.32 bits per element at a 64K-token context, with decode latency, throughput and prefill time reported within 2% of the base pipeline (source). Treat those as author-reported results from one preprint: a promising data point on where the field is heading, not a ship-ready technique. The rotation idea itself is not exotic; Groundy’s 4-bit KV cache analysis describes Walsh-Hadamard rotation as one of the design decisions that makes a robust low-bit path work.

Eviction plus moderate quantization (ReST-KV) attacks a different variable: how many tokens you keep at all. In its needle-in-a-haystack evaluation, a Llama3.1-8B-Instruct configuration matches full-cache accuracy while storing only 1/32 of the original tokens; on LongBench, the paper reports high retained average accuracy at a 6.25% total compression ratio with moderate 4-bit quantization. The risk profile differs from pure quantization because eviction failures look different from rounding errors, but the memory outcome is comparable.

Silent collapse: why throughput dashboards miss it

The sharpest operational finding in this comparison is not about accuracy at all. WitCert reports that SnapKV and KnormPress, two cache-compression methods, “collapse silently under the query-agnostic protocol,” and the paper states the implication directly: when compression fails, the system emits no signal.

That sentence should worry anyone who plans to validate a 2-bit rollout with latency and throughput checks. Those checks will pass. Compression that destroys the information a future query needs does not make tokens slower or the scheduler unhappy; it makes answers quietly wrong on exactly the queries that depend on the discarded or corrupted context. The failure surfaces to users as degraded retrieval quality, and to operators as nothing at all. Groundy’s FlashMemory coverage shows the same pattern for sparse attention: the technique cuts the physical KV footprint sharply, and whether that saving is safe to take still has to be settled by recall measurement, not by the serving stack.

The consequence for capacity planning is that a model that fits is not a model that serves. Doubling concurrent sequences per GPU with a 2-bit cache is a real saving only if the recall quality survives, and the evidence here says you cannot observe that from the serving stack. You have to measure it from the outside, with probes, before and after the configuration change.

So which bit width, for which workload?

The evidence supports a workload-typed answer, and it is worth stating as a bounded inference rather than a rule.

For accuracy-critical long-context workloads (document QA, codebase reasoning, anything where a needle buried deep in the context determines the answer), the defensible default is 4-bit, or eviction-plus-4-bit if you need deeper compression. ReST-KV’s measurement that direct 2-bit costs significant accuracy is the only head-to-head comparison between direct 2-bit and eviction-plus-4-bit in the papers cited here, and nothing cited here contradicts it. The 2-bit recall cost itself is corroborated inside KVQuant’s own paper: average accuracy falls from 56.40 for fp16 to 36.54 for its 2-bit configuration at an average of 2.33 bits (RULER table), tested on LLaMA-2-7B-32K.

For high-concurrency workloads where per-request context is moderate and outputs are tolerant (summaries, classification, short-horizon chat), 2-bit becomes a reasonable bet precisely because the recall surface is smaller. The failure mode WitCert documents matters less when there is less long-range retrieval to fail at.

For workloads dominated by many short sessions, the KV cache may not be your constraint at all; check the weights-and-batch math first.

Whatever you choose, treat rotation-based INT2 like OptR as an experiment, not a default. The method is well-motivated, the objective correction is principled, and its reported AIME25 gains are large, but they are author-reported from a single preprint with no independent replication, which is not a basis for a fleet-wide configuration change.

Recall probes to run before shipping any low-bit KV cache

The evidence converges on one operational requirement: probe recall directly, because the system will not tell you when it breaks. A minimal pre-ship suite, derived from what the failure modes actually look like:

1. Needle-in-haystack probes at your production context lengths, not the demo length. Place retrievable facts at varied depths, including the far end of the cache where compression error concentrates.
2. Long-context retrieval QA on your own traffic distribution. Public suites measure someone else’s workload; ReST-KV’s results are a reminder that aggregate accuracy can hide task-specific collapse.
3. A/B comparison against the FP16 or INT4 baseline on identical request logs, so the delta is attributable to the cache precision, not traffic drift.
4. A gating threshold with an automatic rollback. WitCert’s finding that failures are silent means the gate is the only detection layer you have; a canary at low traffic percentage with recall-probe pass criteria is the cheapest version of this.
5. Re-probe after every engine upgrade. Quantization kernels change; a configuration that passed last quarter is not evidence about this one.

None of these are exotic. All of them are the cost of taking the capacity win.

What these papers do not establish

Every quantitative claim in this article is author-reported from a single paper, with no independent replication as of 2026-09-24. KVQuant’s 1M/10M-context figures are for LLaMA-7B on specific hardware, not your model. ReST-KV’s accuracy findings come from a competing method’s evaluation. OptR’s AIME25 gains and 2.32 bits-per-element cost are one preprint’s numbers. WitCert’s serving numbers cited above come from one RTX 4090 configuration, though the paper separately replicates capacity and throughput on an 8×H200 system serving models from 7B to 70B (source).

Deployment is a separate gap. None of these methods arrived as a stock engine setting: KVQuant required custom CUDA kernels, and OptR runs in a custom SGLang-based INT2 pipeline. This article does not cover runtime flags, so check current engine documentation for your exact version before treating any of this as deployable.

The verdict the papers support: 2-bit KV-cache quantization is a real capacity lever with a measured recall tax and a silent failure mode. Gate it behind context-recall probes, default to 4-bit or eviction-plus-4-bit on accuracy-critical workloads, and treat rotation-based INT2 as a promising preprint until someone replicates it.

Frequently Asked Questions

What is the measured accuracy impact of direct 2-bit KV cache quantization?

Its evaluation reports that applying aggressive 2-bit KIVI or KVQuant directly to the full KV cache “results in significantly lower accuracy.”

How much memory does 2-bit KV cache quantization save compared to FP16?

keys and values stored at 2 bits instead of 16 is an 8× reduction in the per-token cache footprint, which translates roughly into 8× the concurrent sequences, 8× the context length, or some mixture of both, at fixed VRAM.