A single research team has published what appears to be the first detailed account of full-parameter post-training for trillion-parameter MoE models running entirely on non-NVIDIA hardware. The paper, SLAI T-Rex, reports 34.22% MFU on an Ascend SuperPOD for DeepSeek-V4 family fine-tuning, a 2.93x improvement over the open-source baseline. Every number in it is self-reported, measured on one cluster, and independently reproduced by nobody. That verification status is the story, not a footnote to it.
What does SLAI T-Rex actually claim, and what does “full-parameter” cover?
The paper claims an end-to-end optimization practice for full-parameter post-training of trillion-parameter-scale MoE models on Huawei’s Ascend NPU SuperPOD, using the DeepSeek-V4 family as the target workload, with both continued pretraining (CPT) and supervised fine-tuning (SFT) in scope. The paper is arXiv 2607.20145, 73 pages, 22 figures, 20 tables, classed cs.CL with a cs.AI cross-listing, and the arXiv cite line already shows v3.
“Full-parameter” here has a specific and narrow meaning that the headline invites readers to overread. It means no frozen layers, no adapters, no LoRA: every parameter of the trillion-scale MoE is updated during CPT and SFT. That is genuinely the expensive regime. LoRA fine-tuning of a large model on consumer or workstation hardware is routine; full-parameter post-training of a trillion-parameter MoE is a cluster-scale systems problem, and that is why the paper exists.
What it does not mean is pretraining. Nothing in the abstract claims the DeepSeek-V4 family was pretrained on Ascend. The run starts from existing pretrained weights and post-trains them. The CUDA moat, to the extent this paper narrows it, narrows at the fine-tuning layer only. Pretraining a frontier MoE from scratch on Ascend hardware remains untested in this record, and nobody should read the title as evidence otherwise.
The workload detail matters for a second reason. On top of the infrastructure, the authors build a CPT and SFT pipeline for Operations Research tasks using DeepSeek-V4-Flash, generating 10K SFT samples across four task categories and three problem representations, mixing collected domain resources with solver-verified synthetic optimization documents, according to the abstract. So the paper is simultaneously a systems paper and an applied OR model paper, and the two contributions need to be evaluated separately, because they have very different failure modes.
How strong are the headline numbers, 34.22% MFU and 2.93x?
The 34.22% MFU figure and the 2.93x speedup over the open-source baseline recipe are the authors’ own measurements on their own cluster, and no independent reproduction exists anywhere in the public record as of 2026-08-20. Both numbers come from the paper’s abstract and nowhere else.
MFU, model FLOPs utilization, measures achieved throughput against the theoretical peak FLOPs of the hardware the run executed on. A 34.22% MFU on an Ascend SuperPOD says the training loop used about a third of that cluster’s theoretical compute. It does not say how that cluster’s peak compares to an H100 or B200 cluster’s peak, and it does not say what the resulting tokens-per-second costs in dollars or watts. For context that engineers will already have: full-parameter training MFUs in the mid-30s are respectable but not exceptional for trillion-parameter MoE work on any hardware, and NVIDIA-cluster results in a similar band exist in the published literature. The interesting claim is that this number was reached on Ascend at all, given how much of the published MFU literature assumes CUDA.
The 2.93x figure is a comparison against the “open-source baseline recipe,” per the abstract. Baselines against open-source recipes deserve scrutiny, because open-source training recipes are frequently unoptimized on any given hardware: they ship for correctness and portability, not peak throughput. A 2.93x win over a naive port tells you the team did real systems work. It does not tell you how far the result sits from the achievable ceiling on the same silicon. Both things can be true, and the paper gives no way to separate them from the abstract alone.
How does an Ascend stack map onto a PyTorch/CUDA post-training pipeline?
The honest answer, from the verified record, is that the abstract does not say, and any article claiming a component-level walkthrough of this specific run is extrapolating beyond the source. The arXiv abstract names no Ascend software components. CANN, torch-npu, and MindSpeed are not mentioned in the fetched text, so the mapping below describes the general Ascend-to-CUDA correspondence that any migration would traverse, not confirmed details of this run. Treat it as the shape of the work, not a verified account of theirs.
The paper does name its framework at the systems level: a hierarchical optimization spanning three layers, model-level parallelism, computation-communication orchestration, and low-level kernel execution, per the abstract. That three-layer decomposition maps cleanly onto what a CUDA-side post-training stack already does:
| Stack layer | CUDA-side equivalent | What the paper claims it fixed |
|---|---|---|
| Model-level parallelism | Megatron-LM / FSDP / DeepSpeed sharding (TP, PP, EP, DP) | Severe memory pressure at trillion-parameter MoE scale |
| Computation-communication orchestration | NCCL collectives, overlap schedulers, gradient bucketing | Non-overlapped communication overhead |
| Kernel execution | cuDNN, FlashAttention, custom Triton/CUDA kernels | Inefficient kernel execution |
Those three bottlenecks, memory pressure, non-overlapped communication, and kernel inefficiency, are named verbatim in the abstract, and they will be familiar to anyone who has ported a large training job between hardware generations. They are the same three problems that dominate every “we moved training to new silicon” paper since the TPU v2 era. The correspondence is not a coincidence. Training systems have converged on one architecture of problems; what changes per vendor is which layer is worst and how much of the fix is documented.
For a team evaluating migration, the practical translation is this. Model-level parallelism strategy (how you shard experts, where sequence parallelism goes) is mostly portable thinking; the configs are not. Communication orchestration is where vendor collective libraries diverge most from NCCL behavior and where silent throughput regressions hide. Kernel execution is where the porting budget actually goes, because MoE routing and grouped-GEMM kernels are the pieces most likely to have hand-tuned CUDA implementations with no equivalent maturity elsewhere. The paper’s 73-page body presumably details which kernels they rewrote; that detail lives in the full PDF, not the abstract, and this article does not invent it.
Should you trust the 71.81% Pass@1 claim against GPT-5.4-Mini?
Treat the model-quality benchmark as a self-graded result: the specialized OR model’s 71.81% average zero-shot Pass@1, per the arXiv abstract, beating GPT-5.4-Mini by 3.98 points and base DeepSeek-V4-Flash by 11.27 points, comes from an evaluation designed, constructed, and run entirely by the paper’s own authors. That is the abstract’s own framing of the numbers, and the setup deserves the skepticism.
The evaluation data is solver-verified synthetic OR documents plus collected domain resources, 10K SFT samples across four task categories and three representations. Self-built benchmarks have a well-known failure mode: the training distribution and the test distribution are shaped by the same design decisions, so the gap over a general model measures fit to the benchmark author’s task ontology as much as it measures capability. An 11.27-point lift over the base model on your own OR benchmark is evidence the CPT/SFT run did something; it is weak evidence about how the model would score on an OR benchmark the authors did not write.
None of this makes the result useless. Solver verification is a real guardrail: OR problems admit exact checking, which removes the label-noise problem that plagues most synthetic SFT data. If the solver says the answer is optimal, the label is right, and 10K verified samples is a defensible corpus size for a narrow domain. The claim to file away is narrower than the headline: full-parameter post-training on Ascend produced a model that improved substantially on an author-constructed OR evaluation. Whether it beats GPT-5.4-Mini on OR problems generally is an open question that requires a third-party benchmark nobody has run yet.
There is also a provenance point worth stating plainly. The hardware vendor’s incentives and the evaluating team’s incentives are aligned in a way that should affect how you weight the number. Per Wikipedia’s Huawei entry, Huawei’s largest customer is the Chinese government. A benchmark showing Ascend handles trillion-parameter training workloads is a strategically valuable document for the vendor regardless of its technical accuracy. That does not make it false; it makes independent replication more than a formality.
Why is 34.22% MFU not a cost number?
MFU measures how hard the chips were pushed relative to their own theoretical peak; it contains zero information about dollars or watts, and the paper reports no tokens-per-second-per-dollar or per-watt comparison against NVIDIA hardware at equivalent scale. This is the single most common misreading of training-efficiency papers, and this one invites it more than most.
The procurement question a team actually faces is: what does a finished post-training run cost on Ascend versus on an NVIDIA cluster? That decomposes into at least four terms the MFU figure does not touch:
- Hardware acquisition cost per unit of peak FLOPs. Ascend pricing is not publicly comparable to H100/B200 street prices in any dataset cited here, and export-control friction distorts both sides of the comparison depending on your jurisdiction.
- Achieved throughput per chip. MFU times peak FLOPs gives this, but you need both sides’ peaks, which the abstract does not provide for a matched NVIDIA configuration.
- Engineering migration cost. Kernel porting, collective-library tuning, and the debug cycles of a stack with almost no published operational lore. This is a real, recurring cost, paid in senior-engineer months.
- Operational risk premium. One team on Earth has published a successful full-parameter run. Support depth, documentation quality, and the hiring market for Ascend-experienced training engineers are all thinner than the CUDA equivalents.
The paper demonstrates that term two is non-trivial on Ascend, which was not obvious before. It says nothing usable about terms one, three, or four. A migration decision made on the MFU number alone would be a decision made on roughly a quarter of the cost model.
This cuts both ways, and the honest reading requires saying so. If your constraint is GPU supply rather than GPU cost, if you literally cannot buy enough NVIDIA hardware regardless of budget, then the cost comparison is secondary and the feasibility result is the whole story. That describes a large fraction of the Chinese domestic market, which is presumably the audience the run was built for. It describes almost nobody buying through US cloud providers.
Who can actually buy and support Ascend hardware?
For teams outside China, the procurement question gets part of its answer before benchmarks enter the picture. Huawei is a Shenzhen-headquartered Chinese company whose largest customer, per Wikipedia, is the Chinese government, and buying its silicon from inside the US and most allied jurisdictions carries export-control friction that no MFU figure offsets. Ascend SuperPOD procurement is a question for your legal counsel before it is a question for your infrastructure team, full stop.
The legal envelope matters beyond the purchase order. Support contracts, firmware updates, spare parts, and vendor engineering engagement all run through Huawei, a company that, per Wikipedia, reported CN¥862.1 billion (US$118.1 billion) in 2024 revenue, has 208,000 employees, and operates in more than 170 countries. The company’s scale means the hardware ecosystem is not going away; access to it is bifurcated along geopolitical lines and is unlikely to unify soon.
The timing sharpens this. Huawei’s corporate site schedules HUAWEI CONNECT 2026, themed “Advancing the Agentic World,” for September 17-19, 2026 in Shanghai, and the Chinese-language homepage confirms the same dates. That event is roughly four weeks out as of this writing. If a broader Ascend training roadmap exists, new SuperPOD configurations, published pricing, a supported software stack with an English documentation commitment, that is where it would surface, and it would materially change the procurement math this article can currently only bound from one side. Teams with a genuine interest should treat the event as the next verification checkpoint rather than treating this paper as the final word.
When should you pilot non-NVIDIA fine-tuning, and what would verification look like?
Pilot Ascend (or any non-NVIDIA accelerator) for fine-tuning if GPU supply is a binding constraint for your organization, you can legally procure the hardware, and you can staff a kernel-level debugging effort; otherwise hold, and re-evaluate when the MFU claim is converted into hardware-normalized cost or an independent party reproduces the run. That is the decision the evidence supports, and it is deliberately narrower than the headline.
The verification checklist, in order of evidentiary value:
- Independent reproduction. A different team, on a different Ascend cluster, publishing a comparable MFU for full-parameter MoE post-training. Nothing in the fetched record indicates this exists. The only technical coverage surfaced in the research for this article was the arXiv page itself; the absence of third-party analysis is itself a data point about how early this is.
- Cost normalization. Tokens-per-second-per-dollar or per-watt against a stated NVIDIA configuration at comparable scale. This converts the paper’s systems result into a procurement input. The abstract does not provide it; the full PDF or a follow-up might.
- Stack disclosure. Published, versioned software (the CANN/torch-npu/MindSpeed equivalents actually used, with the kernel modifications upstreamed or released) so the migration cost is estimable rather than speculative. The abstract names none of these components.
- Third-party model-quality evaluation. The 71.81% Pass@1 OR result rerun on a benchmark the authors did not construct. Solver-verified tasks make this unusually cheap to do; someone should.
Until items one and two land, the correct posture for a production fine-tuning pipeline on CUDA is to stay put and instrument the exit: keep your training stack’s parallelism layer vendor-neutral enough that a port is a quarter of work, not a year. The CUDA moat at the fine-tuning layer is demonstrably narrower now, on one cluster, by one team’s account. Narrower is not closed, and a moat report written by the bridge salesman deserves a second surveyor.
One last boundary the paper itself draws and the coverage will blur: this is a post-training result. Pretraining a trillion-parameter MoE from scratch on Ascend remains unclaimed in this record. If that run happens and publishes, the story changes category. HUAWEI CONNECT 2026, September 17-19 in Shanghai per Huawei, is the earliest plausible venue for such a claim, and the correct price on it today is “unannounced.”
Frequently Asked Questions
Does the 34.22% MFU figure imply cost parity with NVIDIA H100 clusters?
No, because MFU measures utilization against the specific hardware’s theoretical peak, not absolute cost. The paper provides no tokens-per-dollar or per-watt comparison, so the 34.22% figure cannot be converted into a procurement decision without independent cost normalization data.
What specific software components are confirmed in the SLAI T-Rex stack?
The abstract names no specific software components, leaving the exact stack composition unverified. While the paper describes a three-layer optimization framework, it does not explicitly cite CANN, torch-npu, or MindSpeed, so any claim about specific library versions is extrapolation.
How does the 71.81% Pass@1 result compare to independent benchmarks?
The result is self-graded on a solver-verified synthetic dataset constructed by the authors, which introduces a bias toward their specific task ontology. No third-party evaluation exists to confirm whether the model outperforms GPT-5.4-Mini on external, non-author-constructed Operations Research benchmarks.
What is the primary legal barrier for US-based teams procuring Ascend hardware?
US export controls and FCC bans restrict the sale and import of Huawei equipment, creating a legal envelope that prevents most US and allied jurisdiction teams from purchasing the hardware regardless of technical performance. Procurement is a legal counsel issue before it is an infrastructure one.
What event might provide the next verification checkpoint for Ascend training capabilities?
HUAWEI CONNECT 2026, scheduled for September 17-19 in Shanghai, is the earliest plausible venue for a broader Ascend training roadmap or independent reproduction data. Teams should treat this event as the next critical checkpoint for verifying the paper’s claims.
