Compare DeepSeek, Qwen and GLM leaderboards by artifact and task

Compare DeepSeek, Qwen and GLM leaderboards by artifact and task

A leaderboard rank is only comparable when the models, revision, precision, prompt protocol, and benchmark version match.

There is no durable overall rank for the DeepSeek, Qwen, and GLM families. A leaderboard ranks submitted artifacts under its own scope and evaluation setup; it does not settle which family will be best for every deployment. Hugging Face describes its Open LLM Leaderboard as a way to compare open pretrained models within categories, and its general guidance says comparisons should keep model size, precision, and category aligned. Hugging Face’s leaderboard introduction is a better starting point than a launch chart or a composite number.

That matters especially when a model name masks differences in checkpoint, post-training, quantization, and serving configuration. The Open LLM Leaderboard’s own FAQ says models can appear more than once for different commits or precision settings and distinguishes pretrained, continued-pretrained, fine-tuned, chat, and merged models. Read its FAQ before treating adjacent rows as interchangeable. A normalized score is also not a raw-score copy: the board’s normalization documentation explains that it adjusts for task difficulty and random-guess baselines.

Start with the deployable artifact

The current official materials show why a family-level rank is too coarse. DeepSeek’s official DeepSeek-V3 repository describes a 671B-total-parameter MoE model with 37B activated parameters and a 128K context length, along with several deployment paths. Qwen’s official Qwen3-235B-A22B-Instruct-2507 model card identifies a 235B-total/22B-active model, a native 262,144-token context, and a documented extension path to roughly one million tokens. The checkpoint’s license file supplies its Apache-2.0 terms. Z.ai’s GLM-4.5 repository describes the GLM-4.5 family, later 4.6/4.7 entries, software support, and distinct hardware configurations.

The table is a selection constraint, not a ranking. The models are large MoE systems with different active sizes, context configurations, and deployment paths. A score can inform a shortlist only after the artifact and operating conditions are fixed.

Use a leaderboard for the question it was built to answer

The right board depends on the decision. A general open-model board can help narrow candidates for English language, knowledge, math, or reasoning tasks; it cannot substitute for a coding-agent evaluation, a multilingual safety assessment, or a latency measurement on a team’s hardware. An arena can expose user preference on the prompts it receives, but not necessarily tool use or a private codebase. A vendor model card is vital for identity, license, supported runtime, and disclosed evaluation settings, but it is still vendor-reported performance.

LiveBench illustrates why benchmark version belongs beside every score. Its official repository says it releases new questions monthly, draws on recently released materials, and evaluates 18 tasks across six categories with objective answers. That design addresses a particular contamination concern. It does not make a LiveBench score interchangeable with a normalized Open LLM Leaderboard score, a model-card table, or an arena rank.

Before quoting a number, record these fields alongside it:

1. The exact model identifier and revision or commit.
2. The precision, quantization, context length, prompt template, decoding settings, and tool setup.
3. The benchmark name, release/version date, metric, and whether the result is independently rerun or vendor-reported.
4. The task that makes the score useful: for example, code repair, multilingual extraction, long-context retrieval, or structured tool calling.
5. The deployment constraints that can overturn a score: license, VRAM, throughput, latency, privacy, and cost.

This is a practical inference from the methodology and model cards, not a claim that a particular benchmark is useless. The point is to stop a composite rank from answering questions it was never designed to answer.

A small evaluation protocol beats a frozen top-three list

For a real shortlist, choose two or three artifacts that fit the legal and hardware envelope, then run the same bounded test for each. Use a fixed repository or corpus, fixed context budget, documented prompts, the same checker or human rubric, and a capped spend or time budget. Report pass rate, review burden, latency, and cost separately. If tool use is central, include the actual tool loop rather than extrapolating from a text-only benchmark.

Use published results as a screening signal, not an entitlement to production performance. A model that is high on an English pretraining board may be the wrong choice for a multilingual deployment; a long-context setting may change memory and latency; and an FP8 run may not be comparable with a BF16 result. Treat a score as a property of a reproducible run, not a permanent attribute of a brand.

Turn a shortlist into a deployment decision

Use a board to screen artifacts that fit the deployment constraints, then compare a small shortlist on the same task and acceptance test. The procurement question should be which artifact clears the workload’s requirements, with vendor terms and runtime configuration checked for that specific release.

Decision rule

Do not ask which family is “best” in the abstract. Ask which exact artifact, under which precision and harness, clears the task’s quality floor inside its license, hardware, latency, privacy, and cost constraints. Use one reproducible leaderboard to screen candidates, then validate the shortlist on the work the deployment actually needs to do.