Serving GLM-5.3 on Two Nvidia DGX Stations: What vLLM Multi-Node Requires

If you are weighing a second DGX Station to serve GLM-5.3-class models with vLLM, the deciding question is not whether two nodes can do it, NVIDIA’s developer blog reports that they can, but whether the measured 464 GB checkpoint genuinely requires both stations’ GPU memory, and whether the operational cost of a second node (fabric setup, NCCL tuning, failure handling) beats a tighter quantization or a hosted API. Every throughput figure in this story is author-reported by NVIDIA, the company selling the hardware, and no third-party reproduction accompanies the post. Treat the numbers as a ceiling to reproduce yourself, not a specification.

What NVIDIA measured, and how to read its numbers

The NVIDIA developer post, titled “When One GPU Isn’t Enough,” demonstrates serving GLM-5.3 across two DGX Stations with vLLM, sending requests, and measuring behavior under concurrent load. The model under test is a quantized build: “The NVIDIA GLM-5.3 NVFP4 model, a quantized version of Z.ai’s mixture-of-experts (MoE) model, has 753 billion parameters, with 40 billion activated per token.”

The configuration is stated, not implied. The model is split by tensor parallelism: “Tensor parallelism (TP=2) splits the model’s tensors across the two GB300 GPUs.” The launch flags set a context ceiling of 98,304 tokens (input plus output), cap batched tokens at 8,192, and allow 16 or 32 concurrent sequences, and the benchmark numbers come from SPEED-Bench Coding prompts with prefix caching enabled, four MTP draft tokens per step, and an FP8 KV cache.

The headline number, 607.8 tok/s, is NVIDIA’s own report: “Overall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, where mean request latency is 453.5 s.” Both figures come from the same Table 2 concurrency sweep; the Hacker News submission title simply picked the peak. So 607.8 tok/s is not a configuration mystery. It is the best point on a curve that degrades at higher concurrency, and it remains an author-reported vendor measurement on a vendor-selected benchmark.

What the post establishes is that two-node vLLM serving works for this model at the stated settings. What it cannot establish is how the same setup behaves on your prompts, at your concurrency, with your latency budget.

The memory math: a 464 GB checkpoint vs one station’s 252GB of HBM3e

The trigger for going multi-node, per the post’s own framing, is single-node memory limits. The per-station envelope is fixed by the hardware: MSI’s DGX Station spec page describes “a 748GB coherent memory pool combines 252GB of high-speed HBM3e GPU memory and 496GB of LPDDR5X system memory through the C2C interconnect.”

The checkpoint size is measured, not inferred: NVIDIA reports “At about 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, with CPU weight offload disabled.” Once loaded, “The startup log reports about 218 GiB of weights per GPU, roughly half the checkpoint,” and the post notes “MIXED_PRECISION is expected: NVFP4 quantization targets MoE-block linear weights and activations, not every tensor.”

Two things follow. First, 464 GB is larger than naive 4-bit arithmetic on 753 billion parameters would suggest, because not every tensor is quantized to NVFP4. Do not size your own deployment from parameter counts alone. Second, the demo deliberately forecloses the hypothetical that the coherent memory pool might bail out a single station: CPU weight offload is disabled, so the 496GB of LPDDR5X is not a weight overflow path in this setup. A tighter quantization could still shrink the footprint toward one station’s HBM3e, but NVIDIA’s post includes no single-node quantized baseline, so the quantize-harder-versus-add-a-node comparison is untested. You will have to run it yourself.

Throughput, concurrency, and the C32 latency tail

NVIDIA’s author-reported numbers describe a familiar serving tradeoff. Per-user throughput collapses as you add concurrent requests: “Higher concurrency also reduces mean per-user throughput, from 115.3 tok/s at C1 to 13.0 tok/s at C32,” per the post’s Table 2 sweep. Meanwhile aggregate throughput peaks at 607.8 tok/s at C16 and falls to 385.0 tok/s at C32, with a mean request latency of 453.5 seconds.

Read that latency figure in context. Benchmark requests “use the model’s default reasoning effort (max) and an 8,192-token output cap,” so the 453.5-second mean at C32 reflects long max-effort reasoning generations queuing behind each other, not an unexplained tail. For an interactive coding or chat workload, that tail is the product experience, and 13.0 tok/s per user is slow by hosted-API standards. The two-node setup buys you capacity (the model runs at all, at usable aggregate throughput under load), not per-user responsiveness. If your use case is a small team of interactive users, low-concurrency behavior matters more than the C16 peak, and the sweep’s C1 endpoint is 115.3 tok/s per user.

The comparison table below assembles the three real options side by side.

Node two’s real price: fabric, NCCL, and cluster operations

The second station is not just a capital line item. Splitting a 753B-parameter MoE across nodes moves the bottleneck from VRAM to the network fabric, and the fabric is where self-hosting attempts tend to stall.

The hardware side is documented: the station’s ConnectX-8 SuperNIC delivers “up to 800Gb/s of low-latency, high-bandwidth connectivity”. That number invites a misreading. Eight hundred gigabits per second sounds like it should translate directly into serving throughput, but the MoE communication pattern says otherwise. The NCCL EP paper, which evaluates expert-parallel communication end-to-end with vLLM on an H100-based multi-node cluster, reports that during decode, per-GPU batch sizes are small (1–128 tokens) and latency, not bandwidth utilization, is the primary concern. Expert-parallel dispatch and combine operations are many small, latency-sensitive messages. A wider pipe helps less than a shorter round trip.

The same paper carries a second caution: “NCCL EP trails DeepEP by 7–10% in throughput and 7–9% in ITL/TPOT across all scales,” with the gap attributed partly to combine overhead and optimization still ongoing. In other words, the communication API layer that multi-node MoE serving depends on is itself still being unified and tuned. That is a preprint-level, author-reported result on H100 hardware, not a DGX Station measurement, but it tells you where the engineering risk sits: not in the model weights, and not in raw NIC bandwidth, but in the collective-communication stack you will be operating.

Operating that stack is the unglamorous cost of node two. NCCL topology discovery, RDMA configuration across two chassis, and deciding what happens when one station dies mid-request are new problems a single-node deployment does not have. NVIDIA’s post does not cover node-failure behavior for this setup, so plan for it as an unknown rather than assuming vLLM handles it gracefully.

Break-even: second station vs hosted API

The adversarial economics come from the HN thread, and they deserve to be quoted in full because every assumption is load-bearing: “Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity.”

This is one anonymous community estimate at assumed prices, so treat it as a stress test, not a verdict. Its structure is the useful part. Payback scales inversely with utilization: if you are not saturating the stations with billable or productivity-replacing tokens around the clock, seven years stretches toward never. And it excludes electricity, cooling, and the engineer-hours the NCCL/fabric operations described above will consume. Re-run the arithmetic with your real token volume, your actual quote for the hardware, and a realistic utilization curve before it informs a purchase order. A separate thread comment also notes that cheaper GPU alternatives over PCIe will not match this interconnect: “You’re not going to get the same token throughout with the pro 6000 over pcie. You’re paying for speed and bandwidth.” The premium is the fabric, which is consistent with where the technical risk sits.

Deployment checklist: what to verify before rack-and-stack

Before committing to the two-node path, work through these in order:

1. Memory fit. Confirm your quantized build’s actual weight footprint plus KV cache against one station’s 252GB of HBM3e. If it fits on one station, stop here; the two-node case evaporates.
2. The quantize-harder alternative. Test whether a tighter single-node quantization meets your quality bar. NVIDIA’s post includes no such baseline; it is cheap to test relative to a second station.
3. Fabric prerequisites. Verify ConnectX-8 RDMA connectivity between stations and NCCL topology detection before benchmarking anything; NVIDIA’s tutorial routes this through its own two-station connection playbook.
4. Reproduce the vendor numbers. NVIDIA’s figures are author-reported, measured on SPEED-Bench Coding prompts at max reasoning effort with an 8,192-token output cap. Run your own concurrency sweep from C1 upward on your prompts and watch per-user throughput and the latency tail, not the aggregate peak.
5. Latency regime. Remember decode is latency-bound at small per-GPU batches; if your workload is interactive at low concurrency, the fabric’s headline bandwidth is not the metric that governs your experience.
6. Failure handling. Determine what a single-station outage does to in-flight requests. The post does not cover it; test it.
7. Break-even. Recompute the ~7-year community ROI with your utilization and electricity costs before signing.

The verdict, and what NVIDIA’s post does not tell you

Go two-node only when one station’s 252GB of HBM3e provably cannot hold the quantized weights you need. If a tighter quantization fits and survives your quality evaluation, it avoids the fabric, NCCL, and failure-handling burden entirely. If neither fits and your token volume is modest, the hosted API wins on the community’s own arithmetic until utilization justifies the capex.

What remains open is narrower than the headline suggests. The 607.8 and 385.0 tok/s figures in NVIDIA’s sweep are two points on one author-reported curve, not competing claims, and the configuration behind them is documented. But they are single-vendor measurements of a vendor-selected benchmark, and no third-party reproduction accompanies the post. Licensing is pinned but not reproduced: deployment requires “Acceptance of the NVIDIA Open Model Agreement and the original GLM-5.3 license terms, listed on the model card,” so read that card before planning. And there is no single-node quantized baseline to close the quantize-versus-scale comparison. The two-node path is real and documented; whether it is yours depends on numbers only you can measure.

Frequently Asked Questions

What is the reported throughput at maximum concurrency?

Overall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, where mean request latency is 453.5 s.

What is the estimated ROI for self-hosting versus a hosted API?

Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity.