groundy
Models & Research

DeepSeek V3 in 200 Lines of Jax: What It Leaves Out

A 190-line JAX implementation of DeepSeek V3 teaches MLA and MoE routing mechanics but lacks reference weight comparisons, making it unsuitable for production serving or scale

Published 5 references
A skeptical translucent green resin dinosaur opens its torso to reveal branching channels and a yellow path. A separate tangled resin mass rests beside it on warm ivory, casting a hard shadow.
On this page10 sections

Use the 190-line JAX implementation to learn DeepSeek V3’s attention and routing mechanisms, not to estimate what serving DeepSeek V3 requires. The repository’s README self-reports implementations of Multi-head Latent Attention (MLA), DeepSeekMoE, auxiliary-loss-free balancing and MTP, while explicitly reducing the model to 36.2M parameters, six layers and a 256-token context. Those are the author’s reported figures, not independent verification.

The practical consequence is that a compact forward pass can be a useful reading guide without being a useful serving plan. It can show how latent caching and expert selection fit together, but it does not answer the weight-memory, cache-management, batching or kernel-scheduling questions that production inference raises. The repository does document some correctness: it ships its own unit tests and reports training results. What it does not contain is any comparison against reference DeepSeek-V3 weights, so fidelity to the reference model is unverified. That means exactly what it says, not that the code has been shown to be wrong.

What the repository actually implements

The README’s mechanism list is more informative than the line count. Three parts do most of the architectural teaching.

MLA changes what attention stores. In this implementation, the cache holds one small latent representation per token instead of full keys and values. The author reports that this reduces the cache from 480 to 144 numbers per token per layer for the repository’s configured dimensions. That makes the central MLA tradeoff visible: attention can reduce cache storage by retaining a compressed representation rather than every full key and value vector. The figures describe the mini implementation, not DeepSeek V3’s production attention ranks.

The query side also uses low-rank compression in DeepSeek’s broader MLA design: a DeepSeek-V4 sparse attention report, citing the DeepSeek-V3 codebase, gives the query bottleneck q_lora_rank a default of 1536. That number should not be substituted for the mini version’s dimensions, but it helps explain why latent attention is an architectural family of projections and bottlenecks, rather than a single cache-size trick.

DeepSeekMoE separates selection from compute. The JAX version has one shared expert that always runs, plus 16 routed experts with top-4 selection per token, according to its README. The selection is mathematical, not computational: every expert runs on every token and the unpicked ones are zeroed, with no sparse dispatch or expert parallelism, so the mini model pays dense expert compute while reproducing the routing math exactly. The lesson still carries into the full-size model: total parameters and per-token activated parameters are different quantities.

The DeepSeek-V3 technical report describes the reference model as having 671B total parameters, with 37B activated for each token. Sparse routing reduces the work along a token’s active path; it does not make the remaining model weights disappear from the storage and memory problem.

The repository also names auxiliary-loss-free balancing and MTP. Those names establish intended mechanism coverage, and here there is something behind them: the README lists 39 tests covering causal masking, cached versus full forward, routing, balancing, MTP targets, NaN handling and bf16 master weights, and reports a balancing ablation in which validation loss at step 2000 was 3.921 with the bias and 3.923 without. Those are author-written tests and author-reported numbers. None of them compares outputs against reference DeepSeek-V3 weights, so fidelity to the reference model is a separate and unanswered question. A passing test suite is a starting point for that work, not a substitute for it.

Compression is not the same as omission

The repository is unusually clear about its reduced scale. Its disclosed configuration, set beside the reference report, looks like this:

Property190-line JAX implementationDeepSeek-V3 reference
Total parameters36.2M671B
Activated parameters per token13.6M37B
Layers661
Context256 tokens128K tokens

The mini figures, including the 13.6M active-parameter count, come from the repository author, as does the 61-layer and 128K-token comparison; the 671B and 37B reference figures are reported in the DeepSeek-V3 technical report. They are configurations, not performance results.

Scale is not the only simplification the repository discloses. Its own omissions list includes MLA without weight absorption, where the latent is decompressed explicitly, which is easier to read and slower at inference; FP8 training, DualPipe, node-limited routing, YaRN context extension and any post-training, none of which are included; and the dense MoE compute described above, where “active parameters” describes the routing math rather than the compute actually performed. Repository README

This scale reduction is not inherently a flaw. Six layers can expose the relationship between attention, routing and residual computation without forcing a reader through a production model’s full configuration. A 256-token context can likewise demonstrate cache mechanics without requiring a long-context memory system. For learning, that is often the right tradeoff.

What the table cannot support is a quality, speed or fidelity conclusion. A small model trained from scratch is not a compressed copy of DeepSeek V3’s learned behavior. The README says the implementation was pretrained from scratch on a MacBook Air using public-domain detective fiction, reports validation loss falling from 5.02 at step 250 to 3.51, and describes the result as “an architecture demo, not a strong language model”; all of it is author self-report. It does not claim to load reference weights or reproduce reference outputs, which remain open verification questions.

The distinction matters when choosing a substrate:

  • For understanding how MLA and MoE routing fit into a forward pass, reduced scale removes distracting infrastructure.
  • For testing a small architectural modification, the repository may be a useful sketch, provided the relevant mechanism is first checked for correctness.
  • For estimating production quality, memory use, throughput or long-context behavior, its disclosed configuration does not provide the needed evidence.

The KV cache is only one memory problem

MLA’s 480-to-144 cache reduction is a concrete and useful teaching example. It can also create a misleading impression if “smaller cache” is read as “small serving problem.”

First, the mini implementation’s dimensions cannot simply be extrapolated to a 61-layer, 128K-context deployment. Its cache figures describe its own configuration, and the repository does not document a complete production cache calculation. Batch size, weight storage and auxiliary inference memory also need separate accounting.

Second, model weights remain a substantial problem even when only part of an MoE activates for each token. An author-reported quantization analysis found that 4-bit quantization reduced the memory cost of DeepSeek-R1’s model weights from 670GB to about 370GB. Crucially, that paper’s figure excludes the KV cache and auxiliary inference memory.

Those numbers are for DeepSeek-R1 weights, not a sizing estimate for DeepSeek-V3. Their value here is narrower: even after quantization, a large model’s weights alone can require hundreds of gigabytes before the cache and other inference allocations are counted. A compact forward-pass implementation does not address how those weights are quantized, placed, moved or made available to the experts selected for each token.

The cited numbers therefore support a serving-memory warning, not a measured comparison between the two codebases. No documented comparison runs the same workload on the mini implementation and a production stack, so the size of the serving gap has not been measured.

The serving layer that a forward pass does not describe

Production attention can involve selection and scheduling machinery that has no counterpart in a short educational forward pass. A recent example comes from DeepSeek-V3.2, rather than the original V3.

In author-reported work on DeepSeek Sparse Attention, a lightning indexer precedes every attention layer, scores the whole cache and retains the k=2048 highest-scoring tokens for that layer. The paper describes the indexer scan as a serial dependency of the attention step it feeds. Its proposed Multi-layer Indexer schedule makes a layer’s indexer and attention independent so they can run concurrently. According to the same paper, this requires no retraining or new weights and runs inside SGLang’s decode CUDA graph with production kernels unchanged.

This is not evidence that the 190-line V3 repository should have implemented DeepSeek-V3.2’s sparse attention. It should not be criticized for omitting a feature from a different model version. The example instead shows where an implementation stops telling the production story: even when the mathematical attention operation is understood, token selection, dependencies between stages, decode scheduling and kernel integration remain engineering problems.

The same boundary applies to quantization, KV-cache management, continuous batching, expert parallelism and memory layout. The repository does not document production support for these serving functions. That is an absence of documented scope, not a reviewed inventory of every line of code. A real inference project needs to evaluate them separately rather than assume they follow from a correct forward pass.

Before building on any model-in-N-lines repository

The reusable question is not “Can this architecture really fit in N lines?” It is “Which claims has this repository actually established?” A practical checklist:

  1. Identify the reference precisely.
    Record the model version, configuration and paper being followed. Keep later mechanisms separate: DeepSeek-V3.2’s sparse attention, for example, should not be attributed to the original V3.

  2. Write down the scale reductions.
    Compare total parameters, activated parameters, layers, context length and attention dimensions. Distinguish configured dimensions in the mini version from production ranks and cache sizes.

  3. Separate mechanism coverage from correctness.
    A list containing MLA, MoE or MTP shows intended scope, and author-written unit tests show the code runs as intended. Fidelity is a third question: look for output comparisons against reference weights, with stated tolerances. Where those are absent, treat fidelity as unresolved.

  4. Establish the weight relationship.
    Determine whether the code trains from scratch, loads pretrained weights or supports both. A from-scratch training demonstration does not establish compatibility with reference weights or reproduce their behavior.

  5. Account for memory in separate categories.
    Ask about weights, KV cache and auxiliary inference memory independently. Record the quantization format, context and batch assumptions behind any figure. A cache reduction alone is not a deployment memory estimate.

  6. Locate the serving plan.
    Check for cache management, batched decode, quantization, expert placement and scheduling. If the repository covers only forward or training mechanics, choose the serving stack as a separate decision rather than expecting the minimal code to grow into one unnoticed.

  7. Measure the decision you actually need to make.
    For a learning project, clarity and testability may be enough. For a prototype, test the modified mechanism. For inference, evaluate output quality, memory, latency and throughput on the intended workload and hardware. None of those results can be inferred from line count.

Where this repository fits

I would use this repository as a companion to the DeepSeek-V3 report: read the latent-cache path, follow the shared and routed expert selection, then inspect the balancing and MTP code before accepting their fidelity. It is also reasonable substrate for a small, mechanism-level experiment if the work can tolerate the effort of writing correctness checks first.

I would not use it as the foundation of a serving prototype on the assumption that the hard parts are already represented. Its own README draws a clear scale boundary, and its tests and training results stop short of any comparison against reference DeepSeek-V3 weights. For real inference, choose and evaluate the serving layer independently. The 190 lines can teach the architecture; they cannot validate a production deployment.

Frequently Asked Questions

What are the parameter and layer counts for the 190-line JAX implementation compared to the DeepSeek-V3 reference?

Property190-line JAX implementationDeepSeek-V3 reference
Total parameters36.2M671B
Activated parameters per token13.6M37B
Layers661
Context256 tokens128K tokens

How does the MLA implementation in the repository reduce cache storage per token?

In this implementation, the cache holds one small latent representation per token instead of full keys and values. The author reports that this reduces the cache from 480 to 144 numbers per token per layer for the repository’s configured dimensions.

Does the repository compare its outputs against reference DeepSeek-V3 weights?

What it does not contain is any comparison against reference DeepSeek-V3 weights, so fidelity to the reference model is unverified. That means exactly what it says, not that the code has been shown to be wrong.

References

Follow the links in the article for context. The supporting material is collected here for further reading.

  1. deepseek-v3-jaxgithub.comAccessed
  2. DeepSeek-V4 sparse attention reportarxiv.orgAccessed
  3. DeepSeek-V3 technical reportarxiv.orgAccessed
  4. author-reported quantization analysisarxiv.orgAccessed
  5. DeepSeek Sparse Attentionarxiv.orgAccessed

Join the discussion

Share a useful perspective or ask a question about this article.

Discussion guidelinesComments privacy