Running DeepSeek R1 locally: Build a memory budget

A local R1 memory plan needs the chosen weight artifact, the distill’s own configuration, and the intended context length. The 32B example below combines published file sizes with configuration-derived cache payloads. Throughput needs a separate runtime and workload measurement.

Running DeepSeek R1 locally: Build a memory budget

DeepSeek R1 is not a single model. It is a family spanning a 1.5B distilled checkpoint up to the full 671B Mixture-of-Experts (MoE) model trained with large-scale reinforcement learning. (DeepSeek AI. “DeepSeek-R1.” GitHub, January 2025) For the training background, see the DeepSeek V3 and R1 training discussion.

The distilled models (1.5B, 7B, 8B, 14B, 32B, 70B) were produced by fine-tuning Qwen2.5 and Llama 3 series checkpoints on 800K reasoning samples generated by the full R1 model. They are dense transformers, not MoE, which makes them easier to quantize and deploy on single GPUs. The 671B full model activates 37B parameters per token but requires storing all 671B, with a 128K context window. (DeepSeek AI. “DeepSeek-R1.” Hugging Face) That distinction shapes every hardware decision below. The original release card also identifies the distills’ upstream Qwen and Llama models; review those exact base-model terms when considering redistribution (Hugging Face).

DeepSeek released R1-0528 in May 2025 as an update to the full model (model card). The release reports a longer average reasoning trace on the cited evaluation (12K to 23K tokens on AIME), lifting AIME 2024 Pass@1 from 79.8% to 91.4% and AIME 2025 from 70.0% to 87.5%. The update also ships a new distilled variant, R1-0528-Qwen3-8B, trained from Qwen3 8B Base on traces from the updated full model. At 86.0% Pass@1 on AIME 2024 it beats the original R1-Distill-Qwen-7B’s 55.5% by 30 points and edges Qwen3-235B-A22B’s 85.7%, per the 0528 model card and the January R1 evaluation. The worked memory budget below uses the original 32B distill and its own configuration.

For another model design, see the related DeepSeek-V4 FlashMemory discussion.

Hardware Tiers: What Actually Runs What

Single-Card Consumer GPUs

Parameter-count arithmetic gives a starting estimate. A deployment budget also needs the selected artifact’s size, context cache and runtime buffers. Smaller distills, larger dense checkpoints and the full MoE have different allocations.

For workstation budgets, the RTX PRO 6000 Blackwell packs 96 GB of GDDR7 at 1,792 GB/s on one card, per NVIDIA’s specifications. Whether a chosen 70B artifact and its intended context fit still depends on actual allocations. It cannot hold the tested full-model artifacts listed in the cited study: the smallest variant in China Unicom’s comparison is Unsloth’s dynamic 2-bit UD-Q2_K_XL at 212 GB (arXiv, Table 1).

Budget an exact 32B artifact and its context

The current Bartowski artifact table lists the original R1-Distill-Qwen-32B Q4_K_M file at 19.85 GB, Q3_K_M at 15.94 GB, and Q8_0 at 34.82 GB. These are approximate on-disk sizes in the publisher’s table, not exact device allocations. Converted from decimal GB, they are about 18.49, 14.85, and 32.43 GiB. A 24 GiB device therefore has an arithmetic payload margin with Q4_K_M or Q3_K_M, while Q8_0 exceeds that capacity before context and runtime buffers. Check the runtime allocation log to see whether those payload estimates fit alongside its buffers. (Bartowski R1-Distill-Qwen-32B artifacts)

The exact distill configuration supplies the missing cache constants: 64 layers, eight KV heads, hidden size 5,120 and 40 attention heads, giving head dimension 128. For a conventional two-byte key/value cache, 2 × 64 × 8 × 128 × 2 gives 262,144 bytes per cached token, or 0.25 MiB. That is 2 GiB at 8,192 tokens and 8 GiB at 32,768 tokens, before runtime overhead. These are explicitly calculated cache payloads; quantized caches, allocation blocks and implementation details can change observed memory use. (Exact distill configuration; distill card)

For a simple dense batch-one estimate, 1,000 GB/s divided by a hypothetical 19.85 GB weight read is about 50.4 tokens per second before other traffic and overhead. This is an ideal arithmetic estimate, not a benchmark of that file on your card. It does not transfer directly to MoE selection, batching, cache quantization or speculative decoding.

For Apple runtime choices, see MLX vs llama.cpp on Apple Silicon.

Quantization: What the Measurements Say

Quantization reduces weight precision, changing memory needs and potentially task accuracy. For local LLM deployment, llama.cpp supports GGUF artifacts, 1.5-bit through 8-bit integer quantization and CPU+GPU hybrid inference. (ggml-org/llama.cpp)

For the 32B distill, China Unicom’s study supplies measured effects rather than rules of thumb (arXiv, Table 4):

The report’s zero average drop for one 32B Q4_K_M experiment is not a guarantee of unchanged answers. Its Table 4 reports uncertainty for several task scores and compares against the authors’ BF16 evaluation. Treat the result as a reason to test that precision on the intended task, rather than a universal default. The study’s artifact, calibration and evaluation settings are not automatically identical to another publisher’s GGUF. (arXiv, Table 4)

For the 671B, the same study compares five builds, including Unsloth’s dynamic 2-bit UD-Q2_K_XL (arXiv, Tables 1 and 2):

Within this experiment, the dynamic configurations performed well against the tested static alternatives. UD-Q2_K_XL at 212 GB drops 0.94% while static Q3_K_M at 298 GB drops 1.80%; China Unicom’s own DQ3_K_M lands within 0.34% of FP8 at 281 GB. The listed artifacts contain hundreds of gigabytes of weights; the reported runtime memory figures belong to the paper’s eight-GPU, 32K-context setup.

Reported Throughput Across Configurations

Dated generation (decode) figures from LMSYS’s October 2025 review, with their stated precision and batch settings:

The Spark rows are dated observations reported by LMSYS. The source specifies batch eight for the 14B run and does not state a batch for the 70B row. Keep those settings with the figures, and distinguish prefill, aggregate output and one user’s token spacing.

Practical Deployment: Ollama vs. llama.cpp

Ollama appears in LMSYS’s Spark benchmarks of quantized models, alongside SGLang (LMSYS). A runtime name alone does not fix the checkpoint or precision. Keep the resolved artifact and runtime configuration with any result you compare.

llama.cpp offers control over context length, batch size, and threading. Its current quick start reaches Hugging Face directly, so a GGUF can be pulled at invocation time rather than downloaded separately (ggml-org/llama.cpp). An invocation using a local 32B file and an 8,192-token context can look like this; the GPU-layer setting still depends on the device allocation:

llama-cli \
  -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  -t 8 \
  -p "Your prompt here"

The layer setting requests GPU placement; the memory budget determines how much can fit. For browser inference, see WebAssembly AI: Running Models.

Common Pitfalls

Buying capacity instead of checking latency. Capacity determines whether allocations fit; throughput also depends on the workload and runtime. NVIDIA’s DGX Spark (GB10 Grace-Blackwell, 128 GB unified LPDDR5X at 273 GB/s) makes the point with numbers: it ran GPT-OSS 20B at 49.7 t/s decode where an RTX PRO 6000 Blackwell reached 215 t/s and an RTX 5090 reached 205 t/s, roughly four times faster on the same model, per LMSYS. The RTX Spark unified-memory tradeoff works the numbers in full. Treat that dated comparison as a workload result, not an experiment isolating memory bandwidth. Check capacity and measured latency together.

Ignoring the reasoning-token tax. The R1-0528 release reports average reasoning lengths around 23K tokens per question on its cited AIME evaluation, compared with 12K for the original release, per the model card. At an assumed 20 tokens/second, 23,000 generated tokens alone would take roughly 19 minutes. That arithmetic illustrates why output length belongs in a latency budget; it is not a measured R1 generation rate.

Using the base model’s configuration for a distill. DeepSeek notes that it slightly changes distill configurations and tokenizers and asks users to use its settings (distill card). Read the deployed distill’s own config.json for the layers, KV heads and head dimension. Its published constants supply the 0.25 MiB-per-token two-byte cache calculation above; no inference from the base model is needed.

Mismatched chat templates and settings. DeepSeek’s usage notes recommend temperature 0.5–0.7 (0.6 suggested), no system prompt, and enforcing a <think> prefix, because the original R1 can otherwise skip its reasoning pattern (DeepSeek-R1, Hugging Face). Check the resolved runtime template against the exact checkpoint; a template mismatch is one possible source of changed output, rather than proof that every reported regression is a template bug. R1-0528 supplies its own usage recommendations; check that version’s template and prompts instead of assuming the January rules apply unchanged (R1-0528, Hugging Face). Its release record is the relevant reference when comparing May behavior with the original checkpoints (model card).

What the Distilled Models Actually Score

DeepSeek’s January 2025 evaluation remains the reference for the original distills, with the May 2025 card supplying the Qwen3-8B row (Hugging Face; 0528 card):

The 32B distill beats o1-mini on four of the five published columns: AIME 2024 (72.6 vs 63.6), MATH-500 (94.3 vs 90.0), GPQA-Diamond (62.1 vs 60.0) and LiveCodeBench (57.2 vs 53.8). Competitive programming is the exception; o1-mini’s 1820 CodeForces rating beats the distill’s 1691. DeepSeek’s own summary is worded accordingly, that the 32B “outperforms OpenAI-o1-mini across various benchmarks,” not all of them (model card). The R1-0528-Qwen3-8B row changed the calculus for budget hardware: 86.0% on AIME 2024 from an 8B model, ahead of Qwen3-235B-A22B’s 85.7% on that benchmark, per the May 2025 model card. Its LiveCodeBench figure uses the later 2408–2505 problem window, so it is not directly comparable to the original distills’ column.

The 671B Full Model: Who Actually Needs It?

The cited AIME 2024 rows favor the full model: 91.4% on AIME 2024 for R1-0528 per the 0528 card, against 72.6% for the 32B distill per the January R1 evaluation. But the hardware gap is enormous. The paper’s full-model resource table requires hundreds of gigabytes for its tested builds and context settings. This is a separate hardware class from a dense distill on one consumer card, and neither the active-parameter count nor an average benchmark score can choose the deployment for you. (China Unicom) The vendor reports that the Llama-70B distill scores 94.5% on MATH-500 and the 32B scores 94.3% (DeepSeek-R1), but a hardware purchase still requires an artifact, context and latency plan.

For broader serving choices, see GLM-5.2’s home-deployment path and the related local LLM deployment discussion.

Use the runtime’s separate prompt-processing and token-generation benchmarks, recording its revision, prompt and output lengths, threads, GPU placement and repeated runs. The current benchmark tool exposes those controls and structured output. (llama.cpp benchmark method)