How to Choose Which LLM Weights to Prune: Gradient vs Magnitude

If you run an open-weight LLM on a single 24GB-class GPU and quantization alone still does not fit the model, the pruning-criterion question answers itself faster than most comparisons do. The gradient-informed criterion with the strongest reported evidence, GBLM-Pruner, needs 62GB of memory just to score a 7B model and 96GB for a 13B model, measured on an A100. That is two to four times your entire card, spent before a single weight is removed. Magnitude and Wanda, two of the three pruning defaults shipped in vLLM, cost almost nothing to compute by comparison: Wanda’s measured 7B prune step runs 257 milliseconds against the gradient pass’s 71.3 seconds, and Magnitude needs no calibration data at all. SparseGPT is the exception inside the family: its layer-wise prune measured about 257 seconds on a 7B model in the GBLM-Pruner paper’s own comparison, slower in wall-clock terms than the gradient pass it sits next to, though it does not need a gradient pass. So on your hardware the choice is still largely forced, and the real decisions are the ones after it: how much sparsity to target, what accuracy you will actually lose, and why you should not expect the pruned model to run faster, or to occupy less memory on a mainstream runtime.

This article maps each criterion family to what it delivers in deployment, using four primary sources. One of them, the QK-Wanda preprint, is an unrefereed v1, so treat its quantitative results as author-reported claims rather than replicated findings. The hardware-deciding numbers come from Llama-family models on data-center GPUs, which matters and is addressed at the end.

What pruning actually saves on one GPU

Start with the bottleneck, because pruning is often sold as a speed technique and on mainstream inference stacks it is not one. Unstructured pruning removes individual weights and leaves zeros scattered through the matrices. The nonzero count drops, but on a dense runtime nothing else does: the matrix shapes stay as they were, a dense GPU kernel multiplies zeros just as diligently as real values, and those zeros occupy full bytes in storage. The QK-Wanda preprint states the limitation directly: “Setting weights to zero does not by itself reduce dense-kernel runtime or storage; we do not benchmark inference speedups or hardware-supported N:M sparsity.” A pruned checkpoint served through a dense stack occupies the same VRAM as before a single weight was zeroed. Memory shrinks only when the weights are held in a compressed sparse format by a runtime built to load it, and that is the same sparse-kernel ecosystem that gates speedup.

The evidence on this point is direct. A 2026 paper on unified static-dynamic pruning (SPDP), published in the Proceedings of the VLDB Endowment (PVLDB 19(11), 2026) per its own reference header, reports that general-purpose sparse kernels often fail to accelerate moderate-sparsity workloads, which it places at 30–60% sparsity, the regime typical in LLM pruning. That range is exactly where most no-retraining LLM pruning operates. So the common intuition that 50% sparsity buys something like 2x inference speed does not survive contact with general-purpose kernels. The same paper ties any footprint reduction to compression rather than to zeroing, defining its compression ratio so that “A higher CR indicates stronger compression, reducing memory footprint and data transfer overhead”; realizing that compression takes the Flash-LLM, SpInfer and SPDP class of sparse-aware storage formats and kernels, which the paper itself positions as specialized infrastructure rather than general-purpose stacks. Structured approaches (removing whole attention heads, rows, or channels) can accelerate because they shrink the matrices themselves, but they typically cost more accuracy per parameter removed. Semi-structured 2:4 sparsity is the exception on speed: GBLM-Pruner’s measurements report that 2:4 sparsity improves inference (1.24× overall; 1.54–1.63× per layer on an A6000) and reduces storage, though at a noticeable perplexity cost relative to unstructured pruning, and the SPDP paper associates 2:4’s accelerability with NVIDIA’s sparse tensor cores.

Treat pruning as a conditional memory lever. If your problem is “the quantized model plus KV cache does not fit,” pruning helps only if your serving path holds the pruned weights in a compressed sparse format; zeros stored densely change nothing. If your problem is “tokens per second,” the evidence here offers no support for unstructured pruning as the fix.

The production baseline: magnitude, SparseGPT, Wanda

The practical starting point is what the tooling already ships. According to a security analysis of LLM pruning, “vLLM provides three default pruning algorithms: Magnitude pruning (Han et al., 2015), SparseGPT (Frantar & Alistarh, 2023), and Wanda (Sun et al., 2023).” One caveat on provenance: that claim comes from a research paper, not vLLM’s own documentation, so check it against the release you actually run before relying on it.

The three defaults represent a progression in what the criterion looks at:

• Magnitude pruning scores each weight by its absolute value and removes the smallest. It is the cheapest possible criterion, requiring no data and no forward pass, and it is also the crudest: small weights can still matter when the activations flowing through them are large.
• Wanda multiplies weight magnitude by the norm of the input activations observed on a small calibration set, so a modest weight feeding a large activation survives. This is the activation-informed correction to pure magnitude, at near-zero extra cost.
• SparseGPT goes further, using second-order information (an approximate Hessian from calibration data) to remove weights while adjusting survivors to compensate, one layer at a time.

All three are one-shot methods, but their costs are not equal. The GBLM-Pruner paper’s timing comparison puts the 7B prune step at 257 milliseconds for Wanda and about 257 seconds for SparseGPT on an A100, so the family’s compute budget spans three orders of magnitude. None of the four papers cited here documents the memory footprint of these methods on consumer hardware, so whether SparseGPT fits comfortably on a 24GB card is inference from its layer-wise design, not a measured fact. The same security paper notes that 50% is a common target sparsity in prior work on unstructured pruning without retraining, which gives you the de facto operating point of this whole family.

What gradient-informed criteria add, and the 62GB catch

The gradient side of the title’s comparison has a concrete champion. GBLM-Pruner is presented by its authors as “the first training-free gradient-based pruning method for LLMs that maintains task-agnostic zero-shot generalization across models up to 70B parameters.” The intuition is sound: instead of scoring a weight by its size or its activations, score it by how much the loss would move if the weight vanished, which is what gradients approximate. That is a more direct measure of importance than magnitude, and the reported zero-shot results across model sizes are the strongest claim in favor of gradient criteria.

The catch is the cost of asking the question. The same paper reports: “Gradients require 71.3s/62GB (LLaMA1-7B) and 102.1s/96GB (LLaMA1-13B) on an A100.” Two things follow. First, “training-free” does not mean free; the criterion itself needs a full gradient pass over calibration data at near-training memory footprint. Second, and decisive for the 24GB operator, 62GB exceeds 24GB. That is arithmetic, not a benchmark I ran on consumer hardware, but the gap is large enough that workarounds (offloading, chunking the gradient computation) would dominate the cost of the whole exercise.

If you have access to an A100-class machine for the one-time scoring pass, the calculus changes: you pay the gradient cost once, ship the pruned weights, and the runtime footprint is what matters afterward. On a single consumer card, gradient-informed scoring prices itself out before the accuracy comparison even starts.

Criterion error does not predict the model you get

Suppose you do have the compute to compare criteria on their own terms. A natural method is to measure each criterion’s pruning error, how much it perturbs the weights it touches, and pick the one that perturbs least. A preprint on coupling queries and keys for unstructured pruning tested exactly that logic for attention QK matrices and found it fails. The headline sentence of its limitations section is worth quoting in full:

Lower full QK error does not guarantee lower PPL or higher task accuracy.

At 80% QK sparsity, the same section reports that all four Llama 3 checkpoints have higher PPL than Wanda, and three have lower QK error.

Read that carefully. Across four Llama 3 checkpoints at 80% sparsity, per the QK-coupling preprint, three pruned variants scored better on the criterion’s own error metric and worse on perplexity than plain Wanda. The proxy and the outcome decoupled. The paper’s own contribution, a Wanda-coupled variant called QK-Wanda, adds only 1.3% pruning time over Wanda on an A100 and 3.1% on an H200 across four Llama 3 sizes, so criterion refinement within the activation family is cheap. But the headline finding undercuts the idea that you can shop for criteria by proxy metric alone.

Two practical consequences. First, validate per checkpoint: prune, then measure perplexity (and any task you care about) on the pruned artifact, because the criterion’s internal score will not tell you. Second, treat 80% unstructured sparsity as a regime where accuracy behavior gets erratic, per the QK results, and stay near the 50% convention unless you have a measured reason to push further. This is a reported claim from one unrefereed preprint on one model family, but the direction of the warning is consistent with what the SparseGPT and Wanda line of work established: aggressive no-retraining sparsity has cliffs, and they arrive per model, not per formula.

Pricing the recovery step

The criterion decision does not end at pruning time. If accuracy drops past what your application tolerates, the recovery move is a healing fine-tune or distillation pass on the pruned model, and that cost belongs in the comparison against quantization.

None of the four sources here prices a healing fine-tune in GPU-hours, so I will not invent a number. What the evidence does establish is the structure of the tradeoff. Quantization (which you have presumably already applied) is cheap, uniform, and well-supported in consumer runtimes. Pruning on top adds a one-time criterion cost (near zero for Wanda, prohibitive for gradients on your card), a validation pass per checkpoint, and a contingent fine-tune if the perplexity check fails. For structured pruning, some recovery budget is the norm rather than the exception; for unstructured pruning at the 50% target, the no-retraining line of work exists precisely because retraining is usually not in budget.

My judgment, conditional on the evidence above: if you are choosing between a more aggressive quantizer and pruning plus a healing tune, the quantizer usually wins on total cost unless you have a specific reason, such as a structured-pruning path your runtime accelerates natively, to prefer removed weights over smaller ones. Pruning earns its place when memory is the binding constraint after quantization, when your serving stack can hold the pruned weights in compressed sparse form so the savings are actually realized, and when you can validate the result yourself.

Pruned downloads as an attack surface

One risk axis has nothing to do with accuracy. The vLLM-defaults security paper demonstrates that an adversary can craft a model that behaves benignly as distributed but activates malicious behavior once a user prunes it with the default Magnitude, SparseGPT, or Wanda algorithms. Pruning is the trigger.

For an operator who downloads open-weight checkpoints, this inverts the usual trust calculation. A model that passes your evaluation as-shipped may not be the model you end up serving, because your own compression step is part of the attacker’s plan. The mitigation is procedural rather than technical: evaluate the artifact you actually deploy, after pruning, not the download before it; and treat pre-pruned weights from unofficial re-uploads with the same suspicion you would give any modified binary. The paper shows the attack works against the default criteria specifically, which is one more reason the “just use the vLLM default” path deserves a verification step rather than blind trust.

The decision, condensed

The verdict: on a single 24GB GPU, magnitude and Wanda-family criteria are the practical default, not because they are more accurate, but because the gradient alternative cannot run on the hardware. Target the 50% sparsity convention, expect no speedup on general-purpose stacks (hardware-supported 2:4 sparsity is the exception) and memory savings only if your runtime stores pruned weights in compressed sparse form, verify perplexity on each pruned checkpoint because criterion error demonstrably mispredicts it, and hold a fine-tune budget in reserve. And evaluate the post-pruning artifact, because pruning itself can be the thing that changes what the model does.

Where this evidence stops

The limits here are real and worth stating plainly. The hardware-deciding anchors are Llama-only: LLaMA1-7B and 13B for the gradient memory costs, four Llama 3 sizes for the QK pruning-time comparison, all measured on A100 and H200 data-center GPUs. Cross-family accuracy evidence does exist: the GBLM-Pruner paper evaluates Qwen2.5-7B, Qwen3 8B and 14B, and DeepSeek-R1-Distill-Qwen 7B and 14B against Wanda and SparseGPT, and the QK preprint includes Qwen2.5-72B. But those results are author-reported and unreplicated, so whether criterion rankings and sparsity cliffs transfer across model families remains an open question, not a safe assumption. None of the four papers cited here measures criterion outcomes on 24GB consumer hardware, and none applies a pruning criterion to an already-quantized model, which is the stack this audience actually runs; the SPDP paper’s quantization check uses the opposite order, pruning before 4-bit quantization. The QK-Wanda results come from an unrefereed preprint, and timing numbers from any of these papers will date as sparse kernels and runtimes improve. If sparse-kernel support in mainstream inference engines matures, the memory-versus-speed framing of this article is the first thing to revisit.

Frequently Asked Questions

How much memory does GBLM-Pruner need to score a 7B model?

The gradient-informed criterion with the strongest reported evidence, GBLM-Pruner, needs 62GB of memory just to score a 7B model and 96GB for a 13B model, measured on an A100.

Does unstructured pruning speed up inference on standard runtimes?

Unstructured pruning removes individual weights and leaves zeros scattered through the matrices. The nonzero count drops, but on a dense runtime nothing else does: the matrix shapes stay as they were, a dense GPU kernel multiplies zeros just as diligently as real values, and those zeros occupy full bytes in storage.

What is the common target sparsity for unstructured pruning without retraining?

The same security paper notes that 50% is a common target sparsity in prior work on unstructured pruning without retraining, which gives you the de facto operating point of this whole family.