On a 24GB RTX 3090, a dense 32B model is a 4-bit proposition: only Q4-class GGUF quantizations fit beside a usable KV cache, bandwidth arithmetic puts decode in the low-to-mid 30s of tokens per second at 8k context, and 32k context does not fit with an FP16 cache at any quant that loads. The caveat belongs in the first paragraph, not a footnote: none of the sources reviewed contains a measured tokens-per-second figure for any DeepSeek-class 32B checkpoint on an RTX 3090, so every throughput number below is labeled as derived arithmetic or cut.
What “DeepSeek 32B” means in August 2026
The phrase “DeepSeek 32B” no longer points at anything in DeepSeek’s current catalog, so the first job of any benchmark is naming the exact checkpoint. In the Hub snapshot fetched 2026-08-20, the Hugging Face trending list shows DeepSeek-V4-Pro-0813 at 1.7 trillion parameters1 (“updated 6 days ago”) and DeepSeek-V4-Flash-0731 at 304B1 (“updated 19 days ago”). Neither is 32B, neither is dense in the sense that matters here, and neither loads on a gaming card. The flagship line has left the size class the question is about.
The search term most plausibly refers to a 32B dense distill in the lineage of DeepSeek-R1-Distill-Qwen-32B, the Qwen2.5-32B-based release that put DeepSeek’s name on a checkpoint a consumer could actually run [unverified]. If that is the file on your disk, the arithmetic below applies directly, and it applies equally to any dense 27-32B model with grouped-query attention.
The current model in that size band is the 27-28B tier: Qwen/Qwen3.8-27B at 28B parameters, with unsloth’s community GGUF quantization at 4.32 million downloads, refreshed roughly eight hours before that snapshot. The Hub now exposes GGUF as a library filter and vLLM, llama.cpp, LM Studio, and Ollama as application filters across 3,006,968 model entries, which tells you the ecosystem treats local quantized serving as a mainstream path rather than a hobbyist corner. What the directory does not tell you is what any of those files do on your GPU.
Three million entries, and not one throughput number.
Why decode speed on an RTX 3090 is arithmetic before it is a benchmark
Decode at batch=1 is memory-bandwidth-bound: each generated token requires reading effectively the entire model from VRAM once, so the speed ceiling is effective bandwidth divided by bytes read per token. The compute hardware is mostly idle during this. Generating one token at a time turns weight matrices into matrix-vector operations with terrible arithmetic intensity; the GPU spends the step moving bytes, not multiplying numbers. Prompt processing is the opposite regime, compute-bound and comparatively fast, which is why a long prompt stalls briefly and generation is what you watch.
The inputs to the model are assumptions, not fetched facts: no source in this set contains the RTX 3090’s specification, so take the card as a 24 GB board, assume no decode kernel reaches nominal memory bandwidth, and assume a well-tuned loop achieves 70-85% of it, giving 650-800 GB/s effective. The efficiency fraction is the one thing kernel releases actually move, and the band itself is an assumption, not a measurement. The ceiling does not move:
derived ceiling (tok/s) ≈ effective bandwidth (GB/s) ÷ bytes read per token (GB)
A 20 GB Q4 file predicts 32-39 tok/s before a single benchmark runs. That prediction is falsifiable and cheap to check, which is the point of doing it first.
Independent measurement supports the premise that generation dominates and generation is bandwidth. arXiv:2608.19535, submitted 2026-08-20 and one day old at observation, profiled RAG inference on an NVIDIA Jetson AGX Thor using Llama and Qwen generators: generation accounted for roughly 90% of per-query latency and 91% of GPU energy2 for 7B-8B models. Different hardware class, same physics.
llama.app sells exactly this framing from the vendor side: one binary that runs “from your laptop to a cluster” with “same models, same hand-tuned kernels for every GPU and CPU,” the RTX 3090 named alongside H100, B200, RTX 4090, and RTX 5090. Note what the claim is and is not. Hand-tuned kernels raise the efficiency fraction toward the top of that 70-85% band; they do not repeal the division. A 2020 card holding a place in a list that otherwise reads like a data-center order form is a statement about how far GDDR6X bandwidth still goes.
Which quantizations actually fit in 24GB?
Only the 4-bit class fits: Q4_K_M and its neighbors load with room for a KV cache, Q5_K_M arrives with room for nothing, and Q6_K, Q8_0, and FP8 do not load at all. Weights-only sizes for a dense ~32B checkpoint, derived from the GGUF formats’ per-weight bitrates and therefore approximate [unverified]:
- IQ4_XS: about 17 GB
- Q4_K_S: about 19 GB
- Q4_K_M: about 20 GB (the “Q4” label averages 4-bit and 6-bit tensors, which is why Q4 files differ in size)
- Q5_K_M: about 23 GB
- Q6_K: about 26-27 GB
- Q8_0: about 34-35 GB
- FP8: about 33 GB plus overhead
From 24GB subtract roughly 1.5-2 GB for the CUDA context, compute buffers, and activations (derived), leaving about 22 GB of usable budget for weights plus KV cache. Run the ledger and the decision makes itself. IQ4_XS leaves roughly 5 GB for cache. Q4_K_M leaves 2-3 GB, which is exactly one 8k FP16 KV cache, as the next section shows. Q5_K_M at 23 GB is over budget before a single token of context; Q6_K and Q8_0 are over before the download finishes.
Derived decode ceilings at short context, using 650-800 GB/s effective: Q4_K_M reads about 20.5 GB per token for 32-39 tok/s; IQ4_XS reads about 18 GB for 36-44; a hypothetical in-VRAM Q5_K_M would manage 28-34 and a Q8_0 19-23. Read that list again with the fit table in mind: the quants that would decode faster cannot load, and the quant that loads decodes in the thirties. The card, not your preference, picks the operating point.
FP8 deserves its own sentence because the question keeps being asked. At about 33 GB plus overhead it does not load, and that ends the analysis: FP8 artifacts like Qwen3.8-27B-FP8 and NVFP4 variants like unsloth’s Qwen3.8-27B-NVFP4 are live listings in the Hub’s quant ecosystem, just not for this card. A quant your card cannot load is not a deployment plan.
One failure mode compounds all of this. Spilling layers to system RAM is not graceful degradation, it is a cliff. PCIe 4.0 x16 moves about 32 GB/s [unverified], roughly 3% of the card’s bandwidth, so a fully offloaded 20 GB model reads at approximately 1.6 tok/s derived, and every partially offloaded layer pays the same ratio on its share of the weights. “Doesn’t quite fit” is a tuning problem. “Doesn’t fit” is a different machine.
How much context fits beside the weights?
At Q4_K_M, 8k context fits with roughly 2 GiB to spare; 32k context does not fit at FP16, fits at 4-bit KV cache only, and that at the edge. The KV cache is the second memory resident, and unlike the weights it grows with usage.
Grouped-query attention is why this is survivable. With 8 KV heads against 40 query heads, the cache is one-fifth the size of a full multi-head design; an MHA 32B would store 1.25 MiB per token and demand 40 GiB at 32k, more than two 3090s hold once the roughly 20 GB of weights is counted alongside it. When you shop for a model in this class, the KV-head count matters almost as much as the parameter count.
The ledger at Q4_K_M, including the roughly 2 GB of fixed overhead: 8k at FP16 KV totals about 24 GB, in, with zero slack. 32k at FP16 totals about 30 GB, over by six. 32k with an 8-bit KV cache totals about 26 GB, still over. 32k with a 4-bit KV cache totals about 24 GB, in, at the edge. The alternative route is IQ4_XS weights (17 GB) plus 8-bit KV at 32k (4 GiB) plus overhead, about 23 GB: it fits, and you paid quality twice, once on weights and once on cache, to buy the same window.
Context costs speed as well as fit, because cached entries are read at every step. At 32k with 4-bit KV, per-token reads reach roughly 22 GB, putting the derived ceiling at 30-36 tok/s; the linear read model also grows optimistic at depth because attention kernels lose efficiency as the window fills. The honest summary: context hits the fit constraint before it hits the speed constraint.
The folklore that longer context is monotonically, linearly worse is also not what 2026 evidence shows. arXiv:2608.19203 (submitted 2026-06-10) reports that in 4096-token experiments3, several per-head local-context-allocation variants achieved lower validation loss than pure full attention, and fixed local budgets were competitive with adaptive hierarchy: context behaves like a tunable per-head resource, not a fixed bill. The same edge-RAG measurement found compression tradeoffs non-monotonic, with mild compression missing most of the available energy savings, aggressive compression hurting quality, and the middle band cutting GPU energy up to 53.2% (SoC energy up to 48.2%)2 at negligible quality loss. Both papers are 7B-8B-scale results on edge hardware at short contexts; their transfer to a 32B on a 3090 is analytical, not demonstrated. The operational conclusion survives anyway. Context is a budget you allocate through retrieval discipline, KV quantization, and summarization, and a 24GB card forces that discipline at settings where an 80GB card lets you procrastinate.
Is batch=1 llama.cpp or batched vLLM the right tool for one card?
For one interactive user, llama.cpp at batch=1 is the right runtime; vLLM or TGI earn their keep under concurrency, and on a single 24GB card the KV budget, not the GPU, decides how much concurrency exists. Batching amortizes the weight read, which is the entire economics of serving: with N concurrent sequences, aggregate throughput approaches bandwidth divided by (weights ÷ N plus per-sequence KV reads). The weights stop being the bottleneck as N grows.
VRAM will not let N grow. After Q4_K_M weights and fixed overhead you hold roughly 2 GiB free, which is one 8k sequence at FP16 KV or two at 8-bit KV (1 GiB each). Run the N=2 arithmetic: each step reads about 20 GB of weights once plus 2 GB of combined cache, so aggregate lands at the same derived 30-36 tok/s, split as 15-18 per user. You halved per-seat speed to double total throughput and cannot reach N=4 at 8k without 4-bit cache. At this model size the weights dominate every read, batching pays only when N is large, and the card forbids large N. A dense 32B on a 3090 is a single-seat device, and the serving-stack question mostly dissolves.
If you do serve, the ecosystem has already split along this line: Hugging Face maintains TGI as its optimized serving toolkit, and the Hub’s app filters route vLLM, llama.cpp, LM Studio, and Ollama as peer deployment paths. Continuous batching and prefix reuse across a shared system prompt are the two levers that make multi-tenant serving efficient; both save KV work, neither repeals the per-sequence cost. There is also a format tax to check early: llama.cpp’s native artifact is GGUF, vLLM’s first-class quantization paths are its own formats, so verify your exact file loads in your exact runtime before planning around it.
On the local side, llama.app documents a zero-config serving path: llama serve, plus the pi-llama plugin for the Pi coding agent, no API keys, with “files stay on your machine, requests never leave it.” That guarantee is worth more in 2026 than it was two years ago, for reasons the next section gets into.
Does an idle 3090 beat paying per token?
Electricity is the cheap part: an RTX 3090 generating at 35 tok/s costs roughly $0.42 per million tokens in power, and what you actually pay in is 4-bit quality, a context ceiling around 8k, and your own time. The arithmetic, with every assumption on the table: board power around 350W under sustained decode [unverified], electricity at an assumed $0.15/kWh, so about $0.053 per hour; at 35 tok/s that is roughly 126,000 tokens per hour, or approximately $0.42 per million generated tokens, derived, electricity only, with no hardware amortization, cooling, or idle draw counted. Provider per-token rates for 32B-class models are not in the fetched sources and vary by vendor regardless, so compare against your actual invoice rather than a number this article does not have.
The managed alternative is Hugging Face’s Inference Providers: 45,000+ models4 from multiple providers behind a single unified API, marketed with no service fees, meaning you pay the underlying provider’s token rates and never size a KV cache. That last clause is the real product. Managed serving deletes every constraint this article has derived: the quant ladder, the context ledger, the concurrency math. Local serving deletes different things: per-token marginal cost beyond electricity, data egress, and dependency on someone else’s uptime.
The security ledger runs in both directions, and Wikipedia’s record of Hugging Face supplies one entry: the platform was hijacked by hackers in early 20265. The cached record truncates mid-sentence there, so what was distributed and how far it reached sits outside this article’s source set. One incident is enough for the lesson. A managed hub is attack surface you do not control, and the GGUF you pull from it is a supply-chain artifact you execute on your own silicon. Pin your checkpoints, hash your files, and treat “self-hosted” as removing the runtime dependency, not the download dependency.
What should you run, and why you must re-benchmark it?
Run Q4_K_M at 8k context through llama.cpp for one user, expect the low-to-mid 30s of tokens per second pending your own measurement, and move to Inference Providers the moment you need concurrency, 32k context, or unquantized quality. That is the only configuration the arithmetic supports on this card, and IQ4_XS is the variant to pick when you want KV headroom more than the last few points of Q4 quality.
The limitation, stated as flatly as the verdict: the fetched evidence contains zero measured tokens-per-second figures for this pairing. llama.app lists the RTX 3090 as supported and supplies no numbers. Hub listings show formats and download counts, never throughput. The nearest measurement anywhere in the source set is a 7B-8B generator on a Jetson AGX Thor. Every figure in this article is bandwidth arithmetic with an explicit efficiency assumption, and that assumption is precisely the part that moves between llama.cpp kernel releases. These numbers were derived on 2026-08-21 against a checkpoint lineup (V4-Pro-0813, V4-Flash-0731)1 and a quant ecosystem (the Qwen3.8-27B family) that will look different within a quarter. Date-stamp your own runs and name your binary.
One boundary should not be crossed in either direction: this arithmetic is for dense models, and it does not transfer to mixture-of-experts. Per Ollama’s llama4 page, Scout is 109B total with 17B active parameters, Maverick 400B with the same 17B active. Decode reads only active weights, so a 17B-active MoE reads roughly half the bytes per token of a dense 32B and would decode faster per byte moved, but total parameters decide what loads, and 109B does not sit in 24GB even at 2-bit, derived. The frontier has left the single-card tier entirely. What remains on it is dense, 4-bit, and yours to benchmark.
Frequently Asked Questions
Why does the bandwidth arithmetic fail for MoE models like Llama 4 Scout?
The dense 32B model reads all weights per token, but MoE architectures like Llama 4 Scout (109B total, 17B active) only read the active subset. This means the bytes-per-token cost is roughly half that of a dense 32B, invalidating the direct bandwidth division used for dense models. However, the total parameter count still dictates VRAM fit, so the 109B total size prevents loading on a 24GB card even at 2-bit quantization.
What specific metadata is required to validate a tokens-per-second claim?
A valid benchmark must report the model repository hash, weight and KV quantization levels, context window fill percentage, batch size, runtime commit hash, and power limit. Omitting the runtime commit is critical because llama.cpp kernel updates frequently shift the efficiency fraction between 70% and 85% of nominal bandwidth, rendering ‘latest’ version claims useless for reproducibility.
How does the 2026 Hugging Face security incident affect self-hosting decisions?
The early 2026 platform hijack and the July 2026 autonomous agent breach demonstrate that managed hubs are uncontrolled attack surfaces. Self-hosting removes runtime dependency on external uptime but does not eliminate supply-chain risk, as the GGUF files themselves are artifacts executed on local silicon. Practitioners must pin checkpoints and hash files to mitigate this residual risk.
Why is 32k context with FP16 KV cache impossible on an RTX 3090?
At 32k tokens, an FP16 KV cache requires 8 GiB. Combined with the ~20 GB of Q4_K_M weights and ~2 GB of fixed overhead, the total demand reaches 30 GB, exceeding the 24 GB VRAM limit by 6 GB. To fit 32k context, the KV cache must be quantized to 4-bit, which reduces the cache to 4 GiB but introduces quality degradation in both weights and cache.
What is the operational difference between llama.cpp and vLLM for single-user serving?
llama.cpp is optimized for batch=1 interactive decode, where weight reads dominate and batching offers no benefit. vLLM excels in concurrent serving by amortizing weight reads across multiple sequences, but on a 24GB card, the KV cache budget limits concurrency to N=2 at 8k context. For a single user, vLLM adds complexity without throughput gains, as the card cannot support the high N required to justify its batching overhead.
