Memory, not compute, decides who can profitably serve LLMs the moment inference becomes bandwidth-bound and HBM stays scarce. A July 2026 arXiv preprint from Satoshi Matsuoka argues that is the default state through 2030, and that the announced data-center buildout only pays back inside a narrow corridor requiring roughly 2x annual token-demand growth for four years alongside sticky premium pricing.
The preprint frames the restructuring around four forces: the DRAM/HBM price surge, frontier-capable open-weight models such as GLM-5.2, rapid inference-efficiency gains (near-Shannon-limit KV-cache compression, lightweight local runtimes), and the entry of Meta and xAI into compute resale on fleets bought before memory repriced. The fourth force is the quiet one. Incumbents who ordered silicon before early 2025 hold depreciating fleets at legacy memory costs; late entrants pay current prices for the same bandwidth. That timing gap, not model quality, is what the rest of the analysis turns on.
Why does the paper say memory, not compute, decides who can serve LLMs?
Large-model inference is bandwidth-bound on modern accelerators: the cost to serve a token is set by how fast weights and KV state move across HBM, not by peak FLOPs. Once that is true, the chip is waiting on memory, and whoever holds the cheapest memory bandwidth holds the cost floor. The paper’s claim is that this condition now holds persistently, which reframes where competitive advantage sits. Compute ownership was the moat when chips were the scarce factor; bandwidth ownership is the moat when HBM is.
Wikipedia’s entry on High Bandwidth Memory records the backdrop. HBM has seen an “unprecedented demand increase,” and general DRAM prices in early 2026 “experienced compounded increases, some exceeding 200%, since early 2025,” driven by AI demand, with HBM “crowding out commodity DRAM capacity.” Micron pegs the conversion ratio at 3-to-1 between HBM and DDR5 wafer capacity, so every HBM ramp directly compresses commodity memory supply. Scarcity here is structural rather than a quarter-long squeeze; SK Hynix, Samsung, and Micron remain the three manufacturers that matter, with TSMC producing the base die and lined up as foundry for several HBM companies in 2026.
The second-order move is that memory scarcity transfers the moat from model labs toward memory suppliers, because Samsung, SK Hynix, and Micron sit closer to the cost floor than any lab does. The lever a serving provider can actually pull is allocation, not architecture.
What does pricing inference in dollars-per-petabyte reveal?
Pricing inference in dollars per petabyte of bandwidth delivered ($/PB) separates model quality from serving cost, because $/PB is model-agnostic for bandwidth-bound decode. The practical effect is that two providers can run equally capable models and face wildly different costs, since their $/PB depends on when they bought their HBM, not on the model in the loop.
This is where the entrant-incumbent gap appears. The depreciation conveyor delivers newly amortized fleets to incumbents faster than hardware prices normalize, so the cost gap never closes: 3.2x in 2026, narrowing to 1.9x in 2027, then re-widening to 3-4x by 2029-2030. A new entrant buying today is not catching up; they are buying into a cost curve incumbents have already half paid down. The 2027 narrowing is not convergence but a brief window before repriced fleets roll back onto incumbents’ books and the gap widens again.
The claim worth pressing is that this erodes the serving-cost moat closed labs currently lean on. If open-weights competitors can run a frontier-capable model but can never match the incumbent’s $/PB, then openness shifts the quality question without touching the economics question. Solvency, the paper concludes, now depends on monetized bandwidth demand, premium stickiness, and vintage ownership.
What do the five scenarios say about open versus closed models?
The five scenarios split the modal outcome between two opposing forces: incumbents retaining the cost floor, or premium pricing collapsing under them.
| Scenario | Probability | Implication for who serves profitably |
|---|---|---|
| Rotating Landlord Oligopoly | 25% | Pre-repricing fleets retain the cost floor; open models exist but pay the memory tax |
| Commoditization Crash | 25% | Premium pricing collapses; only the cheapest-vintage operators survive |
| Jevons Absorption | 20% | Per-token cost falls, but demand expands enough to keep memory bound |
| System-Layer Re-differentiation | 18% | Advantage moves up the stack to orchestration, caching, and routing, not raw weights |
| Geopolitical Bifurcation | 12% | Separate cost curves form; China’s domestic-HBM path decouples |
Rotating Landlord Oligopoly and Commoditization Crash tie at 25%, and they pull in opposite directions, which is the point: the most likely outcome is genuinely contested, and the difference between the two is whether premium pricing holds. Jevons Absorption (20%) is the efficiency paradox; per-token cost falls, but demand expands enough to keep memory bound. System-Layer Re-differentiation (18%) pushes advantage up the stack toward orchestration, caching, and routing rather than raw weights.
Geopolitical Bifurcation carries the lowest weight at 12%, but it is where China’s LineShine LX2, domestic HBM on a standard ISA, decouples its cost curve from the memory crisis. A low-probability tail that rewrites the supply map deserves more attention than its weight suggests.
On the training side, costs split into two tiers rather than a single frontier ladder. A luxury tier reaches $18-38B per frontier run by 2030, while a mass tier achieves previous-frontier parity through RL and distillation, with costs falling toward $5M. If the mass tier holds, the open-model question stops being whether open weights can match frontier quality, and becomes whether anyone can afford to serve them at all.
Why is 2027 the only vintage of capacity that survives?
Only the 2027 vintage of capacity is robust across pricing regimes; 2026 and 2028-29 are each fatally exposed to a single regime. Capacity built in the wrong year is a bet on one price path, and most years are the wrong year. The abstract names 2026 and 2028-29 as each fatally exposed to one pricing regime without mapping which vintage to which regime; that detail lives in the paper’s breakeven tables. The natural reading is that 2026 capacity, bought near peak repricing, needs premium pricing to clear, while 2028-29 sizing depends on continued demand growth, but that is interpretation rather than a quoted result.
The buildout-solvency corridor itself requires roughly 2x annual token-demand growth for four years with sticky premium pricing. The paper then undercuts its own optimism with a measurement critique: public token trackers overstate monetizable demand, and all pre-Q2-2026 projections predate the industry’s shift from token maximization to token minimization. Providers incentivized to burn tokens are not the same as customers willing to pay for them.
Could HBM supply or on-device inference break the open-model case?
The memory-as-binding-constraint thesis rests on HBM staying tight through 2030, and three counterfactuals could break it.
First, HBM supply could loosen. SK Hynix, Samsung, and Micron are all expanding, and higher HBM4 stack heights could ease the conversion-ratio pressure. The paper does not assume this away; its weights put Commoditization Crash, the supply-loosens case, at 25%, tied for the top, which is a tacit admission that loosening is the single most likely failure mode for the tight-memory thesis.
Second, on-device DRAM inference bypasses cloud HBM entirely. If lightweight local runtimes shift substantial load onto consumer DRAM, the cloud bandwidth bill shrinks and the $/PB framing loses some of its bite. This is the counterfactual most threatening to the cloud-incumbent story, because it removes the customer from the incumbents’ cost floor rather than competing on it. Companion empirical work on whether mixture-of-experts actually helps inference on consumer and edge hardware (arXiv:2606.21428) is directly relevant to how much load can realistically move on-device.
Third, a greenfield custom-silicon entrant could remove the merchant GPU margin. The paper models this and finds it removes the margin but not the memory premium: the central outcome is 25% success, 34% mediocre, 41% loss, improvable via staged go/no-go gates. Designing your own chip does not let you design your own HBM supply.
How should practitioners route and build-versus-buy in 2026 and 2027?
For teams routing between open and closed models in 2026 and 2027, the solvency threshold says to evaluate providers on vintage ownership and memory position, not just capability. The question to ask a serving provider is not which model do you run, but when did you buy the HBM behind it. A provider on 2027-vintage or pre-repricing hardware has a structural cost advantage that surfaces either as lower prices or as wider margins to weather a pricing-regime shift.
Build-versus-buy tilts toward buy for most teams, because the entrant cost gap of 3.2x in 2026 makes self-hosting economically punishing unless you already own depreciated hardware or have a path to domestic HBM. The custom-silicon route is a 41%-loss bet without staged gates; treat it as a portfolio option, not a plan.
The open-weights question sits downstream of all this. Open models can match quality; whether they can be served profitably is a memory-supply question, and that answer belongs to whoever bought bandwidth before early 2025.
Frequently Asked Questions
Does the memory-cost argument apply to training as well as inference?
It is mostly an inference thesis. Training remains dominated by FLOPs and interconnect, so its bottleneck is different. The paper still projects a split training market by 2030: a luxury tier of $18-38 billion per frontier run and a mass tier that reaches previous-frontier parity for roughly $5 million through RL and distillation.
Why is $/PB better than cost-per-token for comparing providers?
Cost-per-token bundles model quality, context length, and pricing into one opaque figure. $/PB strips away the model identity and shows the underlying bandwidth price, so two equally capable providers can quote very different costs because their HBM was bought at different times. A lower per-token sticker price can even hide a weaker memory position, which is what determines solvency once decode becomes bandwidth-bound.
What should buyers ask a serving provider beyond model names?
Ask when the HBM behind the fleet was purchased and what share of capacity is pre-repricing versus 2027 vintage. Then ask which pricing regimes that vintage can survive: 2027 is robust, while 2026 and 2028-29 are each exposed to a single regime. If a provider cannot explain its position on the demand-growth corridor, roughly 2x annual token growth for four years with sticky premium pricing, its cost advantage is a guess.
Where could on-device inference break the cloud HBM story?
If lightweight runtimes move enough inference onto consumer DRAM, cloud providers lose the volume needed to amortize their HBM investments and the $/PB framing weakens. The companion study on mixture-of-experts on consumer and edge hardware, arXiv:2606.21428, suggests those gains are uneven and task dependent, so the shift is not automatic. Consumer DRAM capacity is also being crowded out by the same HBM-to-DDR5 conversion ratio that is tightening server memory.
Why does the geopolitical-bifurcation scenario matter despite its 12% weight?
It is the only scenario that rewrites the physical supply map rather than redistributing margin among incumbent memory owners. China’s LineShine LX2 uses domestic HBM on a standard ISA, so its cost curve decouples from the SK Hynix, Samsung, and Micron bottleneck. If that domestic supply scales, Chinese providers could undercut the global bandwidth floor without waiting for HBM4 capacity expansion.
