Pure-CPU inference keeps local LLMs honest: it runs anywhere, asks for nothing, and tops out at whatever the CPU’s memory bandwidth will allow. The path most guides skip is splitting the model across the CPU and whatever GPU the machine happens to ship with, even an integrated one. A July 2026 paper, ATSInfer (arXiv:2607.10183), argues the split should happen at the granularity of individual tensors rather than whole layers, and reports up to 1.94× faster prefill and 3.29× faster decode on consumer hardware as a result. Whether that matters for your laptop depends on which phase of generation you are actually paying for.
Why layer-level offloading wastes the GPU
Most local inference stacks treat offloading as a coarse, all-or-nothing decision: a layer runs on the GPU or it runs on the CPU, and the boundary is set once, usually by how much VRAM is free. That works when the GPU can hold the whole model and breaks down exactly when consumer hardware shows its limits. As the ATSInfer authors put it, model weights “often exceed GPU memory capacity” on laptops and desktops, “making offloading inference necessary to extend effective model capacity with CPU memory” (arXiv:2607.10183).
The flaw they attack is granularity. Existing offloading systems “rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions” on consumer devices (arXiv:2607.10183). The claim is specific and worth taking literally: inside a single transformer layer, the attention computation, the projection weights, the biases, and the activation buffers do not cost the same to move, do not compute at the same rate, and do not benefit equally from being on one device or the other. Treating the layer as an atomic unit forces the scheduler to either over-commit VRAM to a cheap tensor or evict an expensive one to free space for a neighbour that barely needs it.
llama.cpp, the reference implementation most people reach for, is honest about the breadth of hardware it targets: “From your laptop to a cluster, llama.cpp runs on whatever you have,” with “hand-tuned kernels for every GPU and CPU” across Apple Silicon, Nvidia RTX, AMD Radeon, and Intel Arc (llama.app). What that generality does not give you is a per-tensor placement policy. The same binary that runs on an H100 runs on a laptop CPU, and the offload decisions on the laptop are made at the layer granularity the ATSInfer paper criticises. The interesting question is whether finer control actually pays off, or just adds coordination cost that swamps the gains.
This is the framing tension for the rest of the piece: hybrid is not free, and the paper’s contribution is a claim about where the breakeven point sits.
How tensor-granularity scheduling works in ATSInfer
ATSInfer does three things, and the order matters. First, it places tensors statically. Second, it transfers them dynamically based on observed load. Third, it coordinates CPU and GPU asynchronously so that data movement overlaps with compute instead of serialising in front of it. The paper describes this as combining “static tensor placement with load-aware dynamic transfer” plus “asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends” (arXiv:2607.10183).
Static placement is the cheap part. At model load, ATSInfer decides for each tensor whether it lives on the GPU, the CPU, or gets fetched on demand, using a profile of how expensive that tensor is to compute versus to move. Because the decision is per-tensor, a layer can be split: the projection weights might stay resident in VRAM while a rarely touched bias lives in system memory and ships over PCIe only when needed. Layer-level schedulers cannot express this because their unit of placement is the whole layer.
The dynamic transfer is the part that addresses the “adapts poorly to changing hardware load conditions” criticism. Consumer machines are not idle servers; the GPU is also driving a compositor, a browser, and whatever the user is doing. A placement that was optimal when the GPU was free becomes a stall when the GPU is busy. Load-aware transfer reacts to that, reprioritising what gets shipped over the bus based on current contention rather than a fixed schedule decided at startup.
Asynchronous coordination is the mechanism that lets the other two pay off. If transfers and compute are serialised, the GPU sits idle waiting for the next tensor to arrive over PCIe, and the win from finer placement evaporates into a pipeline bubble. Overlapping data movement with computation is the only way tensor granularity yields net throughput rather than just lower VRAM pressure. The paper reports that ATSInfer improves “GPU utilization and makes more effective use of PCIe bandwidth” relative to existing offloaders (arXiv:2607.10183), which is the observable signature of that overlap working.
The architecture is not exotic in any individual component. Static placement, dynamic prefetch, and async double-buffering are all standard techniques in the GPU programming literature. What ATSInfer claims is that nobody had stitched them together at tensor granularity for the consumer offloading case, and that the stitching, plus the per-tensor cost model, is where the gains come from.
Where the throughput gains land: prefill versus decode
The headline numbers are phase-dependent, and that dependency is the single most useful thing in the paper for anyone deciding whether to care. ATSInfer “improves prefill throughput by up to 1.94× and decode throughput by up to 3.29×” on representative consumer platforms, evaluated across “both dense and MoE models” (arXiv:2607.10183).
The asymmetry is not accidental, and it tracks how each phase stresses the memory bus. Prefill processes a long prompt in one big batched pass: the compute intensity is relatively high because many tokens share the same weight fetch, so the bottleneck leans toward FLOPs and the gains from smarter placement are real but bounded. Decode generates one token at a time, which means every weight has to be fetched for every token, and the workload becomes almost purely memory-bandwidth-bound. That is exactly the regime where borrowing the GPU’s bandwidth for the tensors it handles best pays the largest dividend, and where the 3.29× figure shows up.
The practical implication is that the workload profile determines whether hybrid scheduling is worth the complexity. A summarisation pipeline that, say, prefills a 4,000-token document and decodes a 200-token summary is prefill-heavy; the speedup lives near the lower bound. A chat assistant or an agent loop that prefills a short prompt and then decodes thousands of tokens of tool calls and reasoning is decode-heavy; that is where the upper-bound number becomes plausible. The paper does not publish a continuous curve between the two extremes, so anyone projecting these gains onto their own workload is interpolating.
There is also a second-order consequence worth naming. If decode throughput really can roughly triple on consumer hardware, the economics of long-running local agents change. Agents that burn tokens in a loop, the kind that keep a model decoding for minutes at a time, are the workloads where a 3× decode improvement compounds into a usable-versus-unusable experience. Prefill-bound batch jobs, by contrast, see a smaller and arguably less interesting win.
Does it help on integrated and entry-level GPUs?
This is the section where the paper’s silence is the story. The whole premise is that consumer devices, many of which ship without an Nvidia discrete GPU, can still contribute their integrated or entry-level GPU to inference. llama.cpp already supports this breadth of hardware, listing Apple Silicon, AMD Radeon, and Intel Arc alongside the usual Nvidia parts (llama.app). But integrated GPUs are not interchangeable, and the ATSInfer abstract does not break out results per vendor or per GPU class.
What the abstract does claim, generically, is that ATSInfer “makes more effective use of PCIe bandwidth” and “increases GPU utilization” (arXiv:2607.10183). That is the right thing to measure for hybrid scheduling, because the entire approach is a bet on the bus: if you can keep PCIe busy with useful transfers while both devices compute, you win; if the bus stalls or the integrated GPU is too weak to absorb the work, you have added coordination overhead for nothing.
The honest reading is that integrated GPUs vary enough that a single throughput multiplier cannot transfer cleanly between them. An Intel Arc iGPU, an AMD Radeon APU, and Apple Silicon all expose a unified or semi-unified memory model that changes the cost of “transfer” in the first place. On Apple Silicon, CPU and GPU share a unified memory, so the PCIe framing barely applies and the relevant bottleneck is memory-bandwidth contention rather than bus transfer (arXiv:2508.08531). The paper’s framing is closest to the discrete-entry-level-GPU case where there is a real bus to schedule against. Anyone running on an Apple unified-memory machine should not assume the 3.29× decode number applies, because the underlying transfer cost model is different.
The conservative interpretation: tensor-granularity scheduling helps most on devices where there is a discrete or semi-discrete GPU attached over a real interconnect, and where that GPU has enough compute to be worth feeding. On a pure integrated part with shared memory, the gains are likely smaller and more sensitive to memory-bandwidth contention from the rest of the system. None of this is contradicted by the paper, but none of it is confirmed by the abstract either, and the gap is exactly the kind of thing that separates a vendor demo from a deployable result.
What MoE models like Llama 4 do to static placement
Mixture-of-experts models complicate the picture because they route tokens to a subset of experts per layer, which means the set of tensors actually needed at any moment is data-dependent. Llama 4 is the canonical example: Scout is a 109B-parameter MoE with 17B active, and Maverick is a 400B-parameter MoE with the same 17B active (Ollama’s Llama 4 page). The active-parameter count is small, but the resident footprint is the full 109B or 400B, which is precisely the regime where offloading is unavoidable and where placement policy matters most.
The tension is between ATSInfer’s static placement and MoE’s dynamic routing. Static placement assumes the cost of fetching a tensor is stable, which holds for dense models where every layer’s weights are touched on every token. In an MoE, an expert that is not routed to on a given token does not need to be resident, but you cannot know in advance which experts a given input will select. Expert-level schedulers, the coarse baseline the paper criticises, handle this by evicting and prefetching whole experts based on predicted routing. Tensor-granularity scheduling can go finer, but the prediction problem does not disappear; it just gets more granular.
The paper does claim wins on MoE models specifically, grouping them with dense models in the throughput results (arXiv:2607.10183). That is encouraging but underspecified. The load-aware dynamic transfer component is presumably doing real work here, because static placement alone would misplace experts whenever routing deviates from the profile. Whether the gains on MoE match the gains on dense models, or whether they degrade as routing entropy rises, is not answered by the abstract. For practitioners running Scout or Maverick locally, that is the open question, and it is the one most likely to determine whether tensor-granularity scheduling survives contact with the models people actually want to run.
When should you run hybrid instead of pure CPU?
The decision is not “is hybrid faster,” it is “is hybrid faster for the thing I am running, on the hardware I have.” Three questions resolve most of it.
First, what is your phase mix? If your workload is decode-heavy, meaning short prompts and long generations, the decode-bound regime is where hybrid scheduling’s largest gains are reported, and the case for involving the GPU is strongest. If you are prefilling long documents and decoding little, the upside is capped near the lower throughput figure and pure CPU may not be giving up much. Agents and chat loops are the workloads most likely to benefit.
Second, what GPU do you actually have? A discrete entry-level GPU or a semi-discrete part with a real interconnect is the regime ATSInfer is designed for. A pure integrated GPU on a unified-memory architecture is a different cost model, and the gains there are unverified by the abstract. If the GPU is weak enough that feeding it costs more than it produces, hybrid scheduling adds overhead without return.
Third, what is your latency budget for added complexity? Hybrid scheduling introduces operator-placement overhead and coordination cost. The paper’s claim is that asynchronous overlap recovers more than it costs, but that is a claim measured on representative platforms, not a guarantee on yours. If you are building a tool that must behave predictably across many unknown machines, pure-CPU inference via llama.cpp has the property of degrading gracefully and uniformly, which is its own kind of value.
The honest summary is that hybrid wins when you have a real GPU, a decode-heavy workload, and tolerance for a more complex stack. Pure CPU wins when you want a single binary that runs identically everywhere, when your GPU is integrated and bandwidth-contended, or when your workload is prefill-dominated and the decode gains do not apply to you.
The asterisks: PCIe, placement overhead, and hardware variance
The strongest caveat is also the most boring one: the paper does not publish exact hardware configurations in the abstract. The phrase “representative consumer platforms” covers a wide range, and the throughput multipliers are reported as upper bounds. Anyone treating 3.29× as a floor is misreading the result. The watch-items are concrete.
PCIE generation and width matter. The entire approach is a bet on keeping the interconnect busy with useful transfers, and the difference between PCIe 3.0 x16, PCIe 4.0 x8, and an integrated GPU’s fabric is large enough to change whether overlap produces net throughput or net stalls. The paper’s “more effective use of PCIe bandwidth” claim (arXiv:2607.10183) is the right thing to assert, but the absolute throughput you achieve depends on which bus you have, and laptops ship with wildly different interconnect budgets.
Placement overhead is real and the paper does not hide it. Static placement requires a profiling pass, and dynamic transfer adds a runtime scheduler that has to observe load and reprioritise without itself becoming the bottleneck. On a system where the CPU is already contended by other work, the coordination cost can eat into the gains, which is the same “adapts poorly to changing hardware load conditions” failure mode the paper ascribes to the baselines it beats.
Hardware variance across vendors is the gap most likely to bite a deployer. Intel Arc, AMD Radeon, and Apple Silicon have different memory models, different driver maturity for async compute and copy engines, and different effective bandwidths. llama.cpp supports all of them (llama.app), but supporting a device and extracting peak hybrid throughput from it are not the same task. The paper’s results, whatever the exact hardware, are a point estimate; transferring them to a different GPU family requires re-measurement.
Finally, the MoE question is unresolved by the abstract. Static tensor placement is most natural for dense models where every weight is touched every token. MoE routing introduces data-dependent access patterns that strain the static assumption, and while the load-aware transfer component is presumably meant to absorb that, the paper does not break out MoE results separately from dense results. For the models that increasingly dominate the open-weights landscape, Llama 4 included (Ollama), that is the result practitioners most need and the one the abstract least provides.
The bottom line, stated plainly rather than announced: tensor-granularity hybrid scheduling is a genuine architectural refinement over layer-level offloading, and the throughput ceilings are high enough to matter for decode-heavy local workloads. It is also a result measured on unspecified consumer hardware, reported as upper bounds, and unverified for the integrated-GPU and MoE cases where most local inference actually happens. Treat it as a strong signal that the offloading granularity question is worth reopening, not as a deployment-ready guarantee.
Frequently Asked Questions
Do MoE models benefit less from static tensor placement when routing is highly dynamic?
Yes. Static tensor placement assumes stable fetch costs, which holds for dense models where every layer touches every token. In MoE models with high routing entropy, expert activation patterns become data-dependent enough that static placement can misallocate tensors whenever routing deviates from the profile. The ATSInfer paper reports wins on MoE models but does not break out whether gains degrade as routing unpredictability increases, which is precisely what practitioners running Scout or Maverick need to know.
How do gains differ between Intel Arc, AMD Radeon, and Apple Silicon integrated GPUs?
The paper does not publish per-vendor results. Apple Silicon’s unified memory model changes the transfer cost equation entirely, because CPU and GPU share the same memory pool and the PCIe framing barely applies. Intel Arc and AMD Radeon APUs sit somewhere between discrete GPUs and unified memory. The brief explicitly warns that the 3.29× decode figure is an upper bound measured on unspecified platforms, and that integrated GPU performance varies enough that transferring the result between vendors requires re-measurement.
Does llama.cpp’s single-binary approach sacrifice the tensor-granularity wins?
Partially. llama.cpp’s design priority is the same binary running identically across Apple Silicon, Nvidia RTX, AMD Radeon, and Intel Arc. That generality requires layer-granularity offload decisions, which is the coarse baseline ATSInfer critiques. The tradeoff is explicit: llama.cpp degrades gracefully and uniformly across unknown machines, while tensor-granularity scheduling requires per-hardware profiling and adds coordination cost that may not repay on weaker GPUs.
What happens to placement overhead when the CPU is already contended by other work?
The coordination cost eats directly into the gains. ATSInfer’s dynamic transfer scheduler must observe load and reprioritise without itself becoming the bottleneck. On a system where the CPU is already saturated by other processes, the scheduler overhead can swamp the throughput wins from smarter tensor placement. This is the same failure mode the paper ascribes to existing layer-level systems, but now it applies to ATSInfer’s own runtime component.
Do the 1.94× and 3.29× figures apply to quantized models?
The paper does not break out quantization separately in the abstract. llama.cpp supports quantization extensively, and most local inference runs 4-bit or 8-bit quantized weights to fit models into limited memory. Whether tensor-granularity scheduling’s relative gains hold when the base model is already compressed is an open question. Quantization changes both the compute-to-memory ratio and the per-tensor transfer cost, which are the two variables ATSInfer’s placement policy optimizes against.
