Running DeepSeek R1 locally is possible on a single consumer GPU—but only if you’re realistic about which variant you’re targeting. The full 671B model demands hundreds of gigabytes of memory. The distilled 14B fits comfortably on an RTX 4090 at 50+ tokens per second. The gap between “technically runs” and “usably fast” is where most guides fail practitioners.

The Model Landscape: 671B vs. Distilled Variants

DeepSeek R1 isn’t a single model. It’s a family spanning from a 1.5B distilled checkpoint to the full 671B Mixture-of-Experts (MoE) base that DeepSeek trained using reinforcement learning from scratch.¹ The economics behind this achievement are remarkable—DeepSeek V3/R1: How Chinese Engineers Matched GPT-4 for $6 Million details how they trained these models for a fraction of typical frontier AI costs.

The distilled models (1.5B, 7B, 8B, 14B, 32B, 70B) were produced by fine-tuning Qwen2.5 and Llama 3 series checkpoints on reasoning traces generated by the full R1 model. They are dense transformer networks, not MoE, which makes them easier to quantize and deploy on single GPUs. The 671B full model uses 37B active parameters per forward pass across its 256 expert layers—but you still need to store all 671B parameters in memory.

This distinction shapes every hardware decision that follows.

Hardware Tiers: What Actually Runs What

Consumer GPUs (Single Card)

The RTX 4090 with 24 GB VRAM is the current practical ceiling for single-card consumer deployment. At Q4_K_M quantization, it handles:

• 14B distilled: ~58 tokens/s through Ollama³
• 32B distilled: ~30–35 tokens/s at Q4_K_M⁴
• 70B distilled: Not recommended—requires heavily quantized format with limited context, and performance drops sharply

The RTX 5090 (32 GB VRAM) expands the ceiling modestly, allowing the 70B model to run at 4-bit quantization with a meaningful context window without offloading.

For the 14B and smaller models, mid-range cards work well:

• RTX 3060 (12 GB): Comfortable with 7B at Q8 or 14B at Q4
• RTX 3090 / 4080 (24 GB): Covers up to 32B at Q4_K_M

Apple Silicon

Apple’s unified memory architecture eliminates the GPU/CPU memory split that complicates NVIDIA workflows. This matters enormously for large quantized models.

The M4 Max MacBook Pro with 48 GB runs the 70B 4-bit quantized variant at approximately 100 tokens per second.⁵ That’s faster than most NVIDIA consumer setups for the same model, at a price point under $5,000. Apple’s unified memory advantage becomes even more pronounced with larger context windows—The Million-Token Context Window: What Can You Actually Do? explores how massive context capabilities are changing what’s possible with local AI deployment. The M3 Ultra Mac Studio (192 GB unified memory) runs the full 671B at 4-bit quantization and delivers approximately 17–18 tokens per second⁶—slow for coding assistance, but genuinely functional for batch tasks.

Multi-GPU and Server Setups

Running the full 671B at usable speed requires enterprise-grade hardware. A pair of NVIDIA H100 80 GB GPUs achieves around 14 tokens/second for single-user inference with Unsloth’s 1.58-bit dynamic quantization, and burst throughput of 140 tokens/second with batching.⁷

For CPU-only inference using llama.cpp on a dual AMD EPYC server with 384 GB DDR5, the full 671B IQ4_XS variant achieves 5–8 tokens/second.⁸ Usable for overnight batch jobs, not for interactive use.

Quantization Explained: The GGUF Format

Quantization reduces model weight precision from 16-bit floats (BF16) to lower bit-widths, trading some accuracy for dramatically lower memory and faster inference. The GGUF format, used by llama.cpp and Ollama, is the standard for local deployment.

Standard GGUF Levels

Format	Bits/Weight	Approx. Size (14B)	Quality vs BF16
Q8_0	8.0	~15 GB	~99%
Q6_K	6.6	~12 GB	~98%
Q4_K_M	4.8	~9 GB	~97%
Q3_K_M	3.9	~7.3 GB	~94%
Q2_K	3.35	~5.5 GB	~89%

Q4_K_M is the practical sweet spot. Research from Red Hat and independent benchmarking shows accuracy differences between BF16 and Q4_K_M consistently under 1% on STEM-oriented benchmarks for the distilled Qwen-32B variant.⁹ Across broader application benchmarks, 4-bit quantization introduces an average 3.52% performance drop.¹⁰

Q8 delivers near-lossless accuracy but runs 20–30% slower on the same hardware. For most practitioners, Q4_K_M gives you 97% of the quality at significantly better throughput. Beyond quantization, there are architectural approaches to improving inference speed—explore Two Different Tricks for Fast LLM Inference: Speeding Up AI Responses for techniques like speculative decoding and KV cache optimization that complement quantization strategies.

Unsloth Dynamic Quantization for the 671B

For the full model, Unsloth’s dynamic quantization approach is the most practical option below enterprise hardware. Rather than applying uniform precision across all layers, it quantizes the MoE expert layers to 1.58 bits while keeping attention and other critical layers at 4–6 bits.¹¹

Four tiers are available:

Variant	Size	Accuracy (Flappy Bird Test)	Min VRAM (fast)
1.58-bit	131 GB	69.2%	160 GB
1.73-bit	158 GB	~75%	160 GB+
2.22-bit	183 GB	91.7%	192 GB
2.51-bit	212 GB	~94%	256 GB

With offloading enabled, the 1.58-bit version runs on a 24 GB GPU (like the RTX 4090) by spilling layers to system RAM—but inference speed drops below 5 tokens/second. This is the “technically runs” category, not the “usably fast” one.

Real Throughput Numbers Across Configurations

At time of writing, these are representative token-per-second figures for generation (not prefill):

Hardware	Model	Quant	Tokens/s	Tool
RTX 4090 (24 GB)	R1-Distill-14B	Q4_K_M	~58	Ollama
RTX 4090 (24 GB)	R1-Distill-32B	Q4_K_M	~30–35	Ollama
RTX 3090 (24 GB)	R1-Distill-14B	Q4_K_M	~35–40	Ollama
M4 Max 48 GB	R1-Distill-70B	Q4	~100	llama.cpp
M3 Ultra 192 GB	R1 671B	Q4	~17–18	llama.cpp
2× H100 80 GB	R1 671B	1.58-bit dyn	~14 (1 user)	llama.cpp
Dual EPYC + 384 GB	R1 671B	IQ4_XS	~5–8	llama.cpp

“Usable” for a coding assistant is generally accepted as 30+ tokens/second. Below that, the latency noticeably disrupts interactive flow. When evaluating AI tools for development workflows, features like Claude’s web search capabilities show how cloud-based models are evolving beyond pure text generation to provide real-time research assistance.

Practical Deployment: Ollama vs. llama.cpp

Ollama provides the lowest-friction path. One command installs and runs a model:

Install Ollama, then:

ollama run deepseek-r1

For the 32B variant:

ollama run deepseek-r1

Ollama handles quantization, model download, and server management automatically. It defaults to Q4_K_M for most model sizes. The trade-off is that it doesn’t expose all llama.cpp tuning parameters, and GPU utilization may not be fully maximized compared to a hand-tuned llama.cpp invocation.

llama.cpp offers more control over context length, batch size, and threading. For the 32B model on an RTX 4090:

./llama-cli
-m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf
-ngl 99
-c 8192
—threads 8
-p “Your prompt here”

The -ngl 99 flag offloads all layers to GPU. Setting it lower begins CPU offloading, which tanks throughput.

LM Studio provides a GUI wrapper over llama.cpp for users who want a desktop experience without terminal management. For those interested in alternative deployment approaches, running models in the browser through WebAssembly AI: Running Models offers an interesting middle ground between cloud APIs and full local deployment.

Benchmarks and Quality: What the Distilled Models Actually Score

DeepSeek’s distilled models punch well above comparable-sized models from other families. On AIME 2024 (a strong math reasoning benchmark):

Model	AIME 2024 Pass@1	MATH-500
R1-Distill-Qwen-7B	55.5%	~83%
R1-Distill-Qwen-14B	~69%	~92%
R1-Distill-Qwen-32B	72.6%	94.3%
R1-Distill-Llama-70B	70.0%	94.5%
OpenAI o1-mini (reference)	~50–63%	~91%

The 32B distilled model outperforms OpenAI o1-mini on multiple benchmarks while running locally on a single RTX 4090.² For practitioners evaluating whether local R1 is worth the setup cost, this is the relevant comparison. When comparing reasoning capabilities across models, Gemini 3.1 Pro: Google’s New Reasoning Model Explained offers an interesting contrast in how different approaches handle complex problem-solving tasks.

The 671B Full Model: Who Actually Needs It?

The full 671B model delivers marginally better results than the 70B distilled in most reasoning tasks, but the hardware gap is enormous. Unless you have:

• An M3 Ultra or M4 Ultra Mac Studio (192 GB+ unified memory), or
• A multi-GPU server with 160 GB+ VRAM, or
• A CPU cluster with 400+ GB RAM and tolerance for 5–8 tokens/second

…the 70B distilled model is the rational choice. The Llama-70B distilled variant achieves 94.5% on MATH-500. That’s not a meaningful quality deficit for the vast majority of use cases.

Frequently Asked Questions

Q: What’s the minimum hardware to run DeepSeek R1 at all? A: The 1.5B distilled variant runs on CPU alone with 8 GB of system RAM, though performance will be under 5 tokens/second. For anything useful, plan for at least an 8 GB VRAM GPU to run the 7B model at Q4.

Q: Does Q4 quantization meaningfully hurt DeepSeek R1’s reasoning quality? A: For the distilled Qwen-32B and Llama-70B variants, research shows Q4_K_M introduces under 1% accuracy difference on STEM benchmarks versus BF16. On broader tasks, the average degradation is around 3.5%. For most practical use cases, this is negligible.

Q: Can I run the full 671B model on a single RTX 4090? A: Technically yes, using Unsloth’s 1.58-bit dynamic quantization with CPU offloading. Practically, expect under 5 tokens/second—too slow for interactive use. The R1-Distill-32B on the same card is a vastly better experience.

Q: Is Apple Silicon competitive with NVIDIA for local DeepSeek inference? A: For single-user interactive use, Apple Silicon is highly competitive, especially at the 70B level. The unified memory architecture removes the traditional GPU memory bottleneck that constrains NVIDIA cards. The M4 Max at 100 tokens/second for the 70B model beats what most NVIDIA consumer cards can do for the same model size. The advantage comes from unified memory, which eliminates the GPU VRAM bottleneck for large quantized models.

Q: How much system RAM do I need alongside my GPU? A: For models that fully fit in VRAM (7B–32B on a 24 GB card), 32 GB of system RAM is sufficient. If you’re offloading layers to CPU RAM, you’ll need enough to hold the overflow—plan for at least 64 GB for 70B offloading scenarios, and 256 GB+ for any attempt at the full 671B.

Footnotes

1. DeepSeek AI. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” GitHub, January 2025. https://github.com/deepseek-ai/DeepSeek-R1 ↩

2. DeepWiki. “Distilled Models — deepseek-ai/DeepSeek-R1.” https://deepwiki.com/deepseek-ai/DeepSeek-R1/2.3-distilled-models ↩ ↩²

3. DatabaseMart. “Deepseek-R1 14B on Ollama Performance Tests: Best GPU Recommendations.” https://www.databasemart.com/blog/deepseek-r1-14b-gpu-hosting ↩

4. SitePoint. “Running DeepSeek R1 on Consumer GPUs: RTX 4090 vs M3 Max.” https://www.sitepoint.com/deepseek-r1-consumer-gpus-rtx4090-m3max/ ↩

5. DEV Community. “Benchmarking DeepSeek R1 on a Developer’s MacBook.” https://dev.to/ocodista/deepseek-r1-7bs-performance-on-a-developers-macbook-3mg2 ↩

6. MacRumors. “Mac Studio With M3 Ultra Runs Massive DeepSeek R1 AI Model Locally.” March 17, 2025. https://www.macrumors.com/2025/03/17/apples-m3-ultra-runs-deepseek-r1-efficiently/ ↩

7. Unsloth AI. “Run DeepSeek-R1 Dynamic 1.58-bit.” https://unsloth.ai/blog/deepseekr1-dynamic ↩

8. GitHub llama.cpp Discussions. “Inference LLM Deepseek-v3_671B on CPU only.” https://github.com/ggml-org/llama.cpp/discussions/11765 ↩

9. Red Hat Developer. “Deployment-ready reasoning with quantized DeepSeek-R1 models.” March 3, 2025. https://developers.redhat.com/articles/2025/03/03/deployment-ready-reasoning-quantized-deepseek-r1-models ↩

10. arXiv. “Quantitative Analysis of Performance Drop in DeepSeek Model Quantization.” 2025. https://arxiv.org/html/2505.02390v1 ↩

11. Daniel Han (@danielhanchen) on X. January 2025. https://x.com/danielhanchen/status/1883901952922448162 ↩