Why Offloaded LLM Inference Stalls: Weight Placement by Prefetch Deadline

If your offloaded LLM decode stalls even after quantizing, the most likely cause is not model size or raw bandwidth. It is that a layer’s weights arrive from a slow memory tier later than the layer’s compute needs them. A preprint posted in November 2026, SlackTier (arXiv:2611.13413), formalizes this as a placement rule: a table of weights can live on any tier whose read time fits inside the compute window of the layer that reads it. All of SlackTier’s numbers are author-reported and unreplicated, so treat them as a mechanism demonstration rather than a result to bank on. The mechanism itself, though, is corroborated by independent measurements from several other systems, and it gives you something durable: a two-number comparison per layer that tells you which weights must sit in fast memory and which can stream from RAM or NVMe.

That comparison, not a faster drive, is the first thing to reach for.

The stall in one number: weight arrival versus the layer’s compute window

During decode, each generated token must read the model’s weights once. When those weights do not fit in VRAM, runtimes stream them from host DRAM or NVMe as each layer executes. The stall condition is simple: if the fetch for layer N finishes after the compute preceding it, the GPU idles. If the fetch finishes inside the preceding compute window, the slow tier is invisible.

SlackTier measured how uneven those windows are. In one decode step of the workload it studied, “Layer 1 needs its rows after about 0.3 ms, and layer 14 after about 9.3 ms. An uncached NVMe read takes 2–4 ms, so it only hurts layer 1. An uncached RDMA read from another node’s memory takes about 0.2 ms” (SlackTier). That is a roughly 30x spread across layers in a single step. A tier that is “fast enough” for most layers can still starve the earliest ones, which is why a whole-model verdict like “NVMe is too slow for decode” is both true and useless.

Independent work shows the same arithmetic biting in different settings. TriMoE, an MoE offloading system, measured what happens when non-resident expert weights are fetched over PCIe: “Their low token count yields computation times too brief to overlap the significant I/O latency, forcing the GPU to stall. Consequently, overall GPU utilization drops below 8.6%” (TriMoE, arXiv:2603.01058). And on the CPU-only side, the cflow report measured single-token decode as bandwidth-bound: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each token must stream every active weight once (cflow, arXiv:2608.23841). Quantization shrinks the bytes per weight, which raises the ceiling, but it does not change whether a given layer’s fetch overlaps its compute.

A placement checklist you can run without the paper

The practical procedure reduces to four steps. Everything here is measurable on your own hardware; none of it requires trusting SlackTier’s specific numbers.

1. Measure each layer’s window. Profile the compute time that elapses, in one decode step, before each layer first touches its weights. Early layers tend to have short windows; later layers give you more slack. Per-layer timing from your runtime’s profiler or an Nsight/rocprof trace is enough.
2. Measure each tier’s read time under load. Not the spec sheet: the actual time to read the layer’s byte footprint from that tier while your real workload is running. MIST validated its KV-retrieval cluster’s latency predictions for NVMe SSDs and DDR4 against fio sequential reads across block sizes from 256 KB to 1 GB (MIST, arXiv:2504.09775); that style of per-tier, per-block-size measurement is what the comparison needs.
3. Place by deadline, not by hotness alone. A layer whose tier read time exceeds its window needs its weights in VRAM or a hot cache. A layer with a long window can stream from RAM or NVMe at no decode cost, provided the prefetch starts early enough.
4. Re-measure when anything changes. SlackTier itself states that “New tables, tiers and hardware need each layer’s window and read time under the target load, plus the tables’ miss curves” (SlackTier). Windows are a property of your model, batch size, and hardware together, not a transferable constant.

The cache-allocation corollary matters as much as tier choice. SlackTier reports that on its workload, layer 14’s table ran from NVMe with no cache and no measurable decode cost, “while layer 1 needs a fast tier or a share of the cache: moving a 1 GB cache from layer 14 to an even split cuts the overhead from 4.9% to 1.3%” (SlackTier). Again, author-reported and unreplicated. But the direction is consistent with the mechanism: cache spent on a layer with a 9.3 ms window buys nothing, because the slow tier already meets that deadline. Cache spent on a 0.3 ms layer is the difference between a stall and an overlap.

What your runtime actually does with offloaded weights

The placement decision is only real if your runtime gives you per-layer control and overlapping transfers. Documented behavior varies a lot.

DeepSpeed inference “pins the model weights either in DRAM (if large enough) or NVMe, and streams each layer into GPU memory for computation when needed,” as documented in the Turnip paper, arXiv:2405.16283. Per-layer streaming is exactly the regime where deadline reasoning applies: each layer’s fetch is a discrete event you can schedule and measure.

CPU-only runtimes are harsher. The cflow report’s current single-token scheduler “reads the entire file sequentially and the I/O and compute phases do not overlap. These are not experimental failures; they are experimentally confirmed structural limitations” (cflow). If your runtime mmap-reads the weight file sequentially with no overlap, placement refinements on the storage side cannot help; the bottleneck is the scheduler’s read pattern, and the levers are quantization, a smaller model, or a runtime with overlapping I/O.

Serving stacks make placement choices you may not have noticed. Per SlackTier’s related-work survey, SGLang and vLLM keep both Engram lookup tables in pinned host DRAM to free HBM for KV cache, and the deployments it surveyed, spanning SSD, per-device NVMe or DRAM, a remote host over RoCE, and peer HBM, “Each puts both tables on one tier” (SlackTier). For Engram-style tables, whole-component, single-tier placement is the default, and if you serve a model with such tables it is probably what you are running. The gap SlackTier identifies is real even if its fix is unproven.

When faster NVMe pays, and when ordering matters more

The second-order consequence of deadline reasoning is that it demotes bandwidth upgrades. A faster NVMe drive shortens read time on that tier, which only helps the layers whose windows the old drive missed. If your placement already meets every deadline, or if the missing layers are the ones no NVMe could serve, the upgrade buys nothing measurable.

Consider the arithmetic with SlackTier’s reported figures as an illustration, not a prescription: if an uncached NVMe read takes 2–4 ms and only layer 1’s 0.3 ms window misses it, doubling NVMe bandwidth still misses layer 1 while making no difference to layer 14. The money is better spent on a small amount of fast memory for the short-window layers. Conversely, if several mid-window layers sit just beyond the drive’s read time, a faster tier can pull them into compliance, but only a per-layer measurement tells you which situation you are in.

There is also a contention caveat. Unmanaged prefetch traffic can contend with demand traffic on the shared tier, but management improves prefetching rather than enabling it: a study of DRAM-cache prefetching for pooled memory measured a 7% IPC gain from prefetching alone, and a further 7–10% from managing demand-versus-prefetch bandwidth, either with weighted fair queuing at the memory node or prefetch-rate adaptation at the compute node (arXiv:2406.14778). The mitigations exist because prefetch requests cause queuing delays at the pooled memory node, so on a bandwidth-constrained tier, prefetch sharing the link with demand reads can erode the benefit it was meant to add.

Batching changes the deadline

Everything above assumes latency-bound decode, where one token’s weights must arrive before one layer’s compute ends. Throughput-oriented serving relaxes the deadline structurally. FlexGen, which runs OPT-175B on a single 16 GB GPU, states the principle directly: “In throughput-oriented scenarios, we can sacrifice latency by using a large batch size, and amortize the expensive I/O operations…” (FlexGen, arXiv:2303.06865). It reported 1 token/s generation at an effective batch size of 144, trading per-request latency to amortize the I/O cost across the batch. Larger batches lengthen each layer’s compute, giving transfers more room to overlap, but that mechanism is inference from the deadline arithmetic, not a claim the paper makes. FlexGen’s placement policy, found by solving a linear-programming problem and paired with 4-bit compression of weights and attention cache, optimizes for aggregate throughput, not per-token deadlines.

So “NVMe is too slow for decode” is a regime statement, not a law. If you are batching for throughput, bigger batches and compression may do more than any placement refinement. If you are serving interactive single-stream decode, deadlines bite and placement is the lever. Decide which regime you are actually in before optimizing for the other one.

Rule out the impostors first

Three documented failure modes produce the same symptom, a GPU waiting during decode, without any weight arriving late.

CPU orchestration. A multi-GPU serving study found that “multi-GPU performance often degrades not because GPUs are saturated, but because CPUs fail to keep them busy” through tokenization, runtime orchestration, and communication coordination (arXiv:2603.22774). A stalled GPU is evidence of a stall, not of its cause.

Capacity ceilings. FlashAccel’s arithmetic for Qwen3-32B on an 80 GB H100: after roughly 64 GB of weights, about 16 GB remains for KV cache, and at roughly 2 GB per 8K-context request the batch ceiling is 8 (FlashAccel, arXiv:2607.10186). If your throughput is capped, capacity may bind before bandwidth or placement ever get a vote. Relatedly, DualPace found host KV capacity acts as “an admission-control knob rather than a monotonic ITL knob”: adding capacity helps until the CPU execution tier saturates on AMX execution and memory bandwidth (DualPace, arXiv:2610.33501).

Misleading counters. EdgePace microbenchmarks found a 28 GB/s bandwidth threshold “detects only 45.26% of actual misses, while 93.84% of its warnings are false positives,” (EdgePace, arXiv:2610.37307) because monitors averaged over roughly 20 ms smooth out the bursts that cause deadline misses. Diagnosing “memory is slow” from an average-utilization dashboard can be wrong in both directions. Measure per-transfer latency under load instead.

What the SlackTier preprint measured, and what it did not

The honest scope: SlackTier’s end-to-end results cover Engram-style lookup tables in batched serving on 4xH200 nodes, with shared disks emulated. Its windows (0.3 ms and 9.3 ms), its cache result (4.9% to 1.3% overhead; SlackTier), and its survey of single-tier deployments are author-reported in a single preprint with no independent replication. It also reports reproducing the early-layer-tight behavior in a second model’s per-layer n-gram embedding, with LRU hit rates of 86–87% on agent traffic and 54–60% on chat (SlackTier), which suggests the pattern is not a single-model artifact but does not settle it. No system cited here benchmarks deadline-ordered placement for dense transformer weights in partial-offload runtimes, so extending the rule to that case is inference from the mechanism, not a measured result.

What the preprint contributes, regardless of whether its numbers hold up, is a crisp statement of a rule the other systems here already obey piecemeal: per-layer streaming runtimes, skew-aware MoE expert placement, and throughput-oriented LP placement are all special cases of matching data arrival to consumption deadlines. The zero-stall condition itself is not the preprint’s invention: SlackTier’s related-work section credits Ma et al. (2026a) with defining it, “lookups finish within the compute of the preceding layers,” for a CXL memory pool, and builds the tier-and-cache rule on top of that condition (SlackTier).

The verdict: a placement order of operations

Profile before buying. Rule out CPU-orchestration stalls and capacity ceilings first, since they mimic weight starvation and no amount of placement fixes them. Then measure each layer’s prefetch window and each tier’s read time under your real load, with per-transfer latency rather than averaged bandwidth. Place short-window layers’ weights in VRAM or a hot cache, and let long-window layers stream from RAM or NVMe; spend cache on the layers whose windows a slow tier cannot meet, which is usually not where hotness-based caching puts it. Check your regime: if you batch for throughput, amortization and compression may dominate placement entirely. And treat a faster drive as the last lever, not the first. It pays only for the layers whose deadlines the current ordering already misses, and the measurement is what tells you whether any exist.

Frequently Asked Questions

How do I determine which layers need fast memory for offloaded LLM inference?

1. Measure each layer’s window. Profile the compute time that elapses, in one decode step, before each layer first touches its weights. Early layers tend to have short windows; later layers give you more slack. Per-layer timing from your runtime’s profiler or an Nsight/rocprof trace is enough.
2. Measure each tier’s read time under load. Not the spec sheet: the actual time to read the layer’s byte footprint from that tier while your real workload is running.

Does batching affect the need for precise weight placement?

FlexGen, which runs OPT-175B on a single 16 GB GPU, states the principle directly: “In throughput-oriented scenarios, we can sacrifice latency by using a large batch size, and amortize the expensive I/O operations…” (FlexGen, arXiv:2303.06865). It reported 1 token/s generation at an effective batch size of 144, trading per-request latency to amortize the I/O cost across the batch. Larger batches lengthen each layer’s compute, giving transfers more room to overlap, but that mechanism is inference from the deadline arithmetic, not a claim the paper makes.

What should I check before optimizing weight placement?

Profile before buying. Rule out CPU-orchestration stalls and capacity ceilings first, since they mimic weight starvation and no amount of placement fixes them. Then measure each layer’s prefetch window and each tier’s read time under your real load, with per-transfer latency rather than averaged bandwidth.