When to Merge LLM Checkpoints: Diagnosing Task Vector Interference

When a merged checkpoint underperforms, the useful first move is not to reach for a different merge recipe. It is to figure out which of three documented interference mechanisms you hit. The merging literature names all three: tasks competing for model capacity until averaging destroys task-specific knowledge (arXiv:2503.05320), parameter updates whose signs disagree across tasks and cancel out, and redundant or noisy components in one task vector corrupting valuable components in another (arXiv:2510.14697). Each has a matching intervention, and each failure looks different in your evals.

The practical consequence: before shipping any merge, audit both skills. Run the eval that tests the capability you wanted to add and the eval that covers the base model’s existing strengths. A merge that “works” on the new skill but quietly degrades the base skill is the failure this whole taxonomy exists to catch, because model-level averaging is documented to lose task-specific knowledge (arXiv:2503.05320).

One honesty note up front, because it colors everything below. Almost every quantitative claim in this literature is author-reported, with no independent reproduction in the sources cited here, and several of the supporting experiments were run on image classification and dense prediction rather than production LLM checkpoints. Some of the work is peer-reviewed: the PAVE paper carries an EMNLP 2026 acceptance note, and its bibliography records TSV-M in CVPR proceedings (arXiv:2510.14697). Peer review is still not reproduction. The newest method discussed here, CUSD, is a single preprint observed 2026-11-05 with no independent reproduction. I will flag each number accordingly rather than let them harden into facts.

Triage the symptom first

Your audit will land in one of three places:

• Base skill lost, new skill gained. The merge worked but overwrote something. This points toward capacity competition or a few large, conflicting updates.
• New skill missing, base intact. The task vector’s contribution was drowned out, often by redundant or noisy magnitude in the other vector, or by sign disagreement canceling the updates that mattered.
• Both degraded. The tasks are fighting over shared directions. This is the hardest case and the one the newest research targets directly.

Mapping symptoms to causes this way is my inference from the mechanisms documented below, not a protocol any paper has validated end to end. Treat it as a diagnostic hypothesis generator: each cause suggests a cheap test before you commit to a fix.

Cause 1: your task vectors disagree in sign

Fine-tuning two tasks from the same base produces two delta vectors. Where those deltas point in opposite directions on the same parameter, naive averaging splits the difference and satisfies neither task. This is the interference mechanism TIES-Merging was built for. The description in the TSV-M paper is precise about the rule: TIES “addresses redundancies in model parameters by first selecting the top-k most significant parameter changes and then constructing a sign vector based on the majority sign across all models,” then merges “disjointly, meaning the average is not computed when a parameter is zero or when parameters disagree in sign” (arXiv:2412.00081).

That description doubles as a diagnostic you can run yourself. Compute both task vectors, compare their signs parameter by parameter, and measure how much disagreement concentrates in the top-magnitude entries. Heavy disagreement in the large-magnitude tail is the signature of this failure mode. If you see it, a sign-aware merge (TIES-style trimming) is the matched intervention. If sign disagreement is sparse, the problem lies elsewhere and sign trimming will add little.

Cause 2: magnitude and redundancy imbalance

Not every parameter shift during fine-tuning carries task capability. The PAVE purification paper puts it plainly: “plain task vectors often contain substantial redundancy, as not all parameter shifts during fine-tuning contribute to task-specific capabilities,” and “the redundant or noisy components may interfere with the valuable components from other task vectors during merging” (arXiv:2510.14697).

Two published interventions attack this from different angles. DARE randomly resets redundant parameters to their pre-trained values and rescales the survivors proportionally (arXiv:2412.00081). PAVE is more surgical: it decomposes the task-vector subspace and keeps only the largest r singular values and vectors, on the observation that “the first several components account for the majority of the learned task-relevant knowledge” (arXiv:2510.14697).

The tell for this failure mode is asymmetry. If one fine-tune ran long or with a high learning rate, its task vector tends to be large and noisy, and it will swamp a smaller, cleaner vector from the other task. The new skill goes missing or the base skill erodes depending on which vector is bloated. A quick check: compare the norms of the two task vectors and inspect their singular value spectra. A vector with a long, flat tail of small singular values is carrying noise you can prune.

There is also a cheap fix worth trying before any structural surgery. In parameter-efficient checkpoint merging, loss-weighted averaging “often yields the best results, delivering up to 5% higher task accuracy than the baseline uniform merge” (arXiv:2504.18580), and even surpasses the final individual checkpoint’s performance, the authors report. That figure is author-reported, and its setting is narrow: the paper merges checkpoints from within a single PEFT training run, not independently fine-tuned task models. Applying the same weighting across two task vectors is my inference. The intervention still costs almost nothing: weight each checkpoint by its validation loss instead of averaging uniformly. If re-weighting alone closes the gap, you had a magnitude imbalance, not a deep conflict.

Compression is not just storage savings

One more result belongs here because it changes what “fixing interference” means. The TSV-M authors report that “while interference reduction contributes most to performance improvements, the optimal results are achieved in combination with compression” (arXiv:2412.00081). In other words, a diagnosis that only removes interference may under-deliver; compressing the task vectors at the same time is where their reported peak came from (normalized accuracy up to 97%, author-reported in arXiv:2412.00081, as a storage-efficient alternative to ensembles and multi-task learning). Separately, task-vector quantization is reported to maintain or improve merging performance while using only 8% of full-precision checkpoint memory (arXiv:2503.06921), though the 8% figure comes from the image classification and dense prediction experiments; the paper also evaluates NLP tasks with RoBERTa-base. If you are storing many task vectors for repeated merging, the compression axis is a real decision, not an afterthought.

Cause 3: both skills degrade, and the fight is over shared directions

The newest entry in this literature reframes interference as a question of which directions in weight space serve both tasks. CUSD (Common-Unique Subspace Decomposition) is “a data-free pre-merging method that separates task vectors into common and unique directional components, merges the common components with a downstream arithmetic operator, and retains unique residuals separately” (arXiv:2611.04108). It computes a cross-task dispersion score for each direction in a shared basis: directions where energy is comparable across tasks get labeled common, directions dominated by one task get labeled unique.

The logic is that averaging everything indiscriminately is exactly what destroys both skills. Common directions can be merged with a standard operator, while the unique residuals are retained separately and controlled by a scalar coefficient rather than averaged away.

The authors report improvements in 42 of 44 model-operator combinations across 11 base models, four model families, and four arithmetic-based merging operators, with gains up to 22.5% over corresponding baselines (arXiv:2611.04108). Every word of that is author-reported from a preprint with no independent reproduction as of 2026-11-05, and the cited sources do not establish that the tested model families are LLMs. What CUSD changes even if the numbers shrink on review is conceptual: it shifts merge work from picking a blind recipe to explicitly accounting for which weights must serve both tasks. If your audit shows both skills degraded and sign conflict is modest, this is the mechanism to investigate, and re-benchmarking both skills on your own checkpoints is the only responsible way to adopt it.

The decision: merge, route, or fine-tune

Here is the full mapping, with the caveat that the symptom-to-cause rows are my synthesis of the cited mechanisms, not a validated protocol:

The last row deserves emphasis. Routing requests to two specialized checkpoints costs you double serving memory and a router, and it forfeits the elegance of one artifact. But if shared-capacity interference persists after sign trimming, purification, and direction separation, the capacity-competition mechanism says the weights genuinely cannot serve both tasks well. Continuing to re-merge blind at that point is not persistence; it is refusing to read the diagnosis. Further fine-tuning the merged checkpoint on mixed data is the middle path, but it abandons the whole premise of training-free merging, so budget it as such.

Where interference lives, and where this evidence does not

Two findings bound where all of this applies. First, the TSV-M paper observed that “task interference decreases in deeper layers” in its ViT experiments (arXiv:2412.00081). If your own probes show interference concentrated in early layers, that is consistent; if you find it deep in the stack, expect a harder merge.

Second, and more important for LLM work: the evidence is unevenly anchored across modalities, and the split is worth knowing precisely. The core baselines are not vision-only. TIES-Merging and DARE have author-reported LLM evaluations, on LLaMA-2-7B, Mistral-7B, and LLaMA-2-13B in one paper (arXiv:2406.11385) and on LLaMA2 in another (arXiv:2503.05320); Task Arithmetic and Ties-Merging are evaluated on LLaMA-2-7B math and code checkpoints in a third (arXiv:2510.14697). What the cited sources do anchor to vision are specific results relied on above: TSV-M’s layer-depth finding, TVQ’s 8% memory figure, and CUSD’s headline numbers, whose model families the sources do not name. Among the methods cited here, the only one that declares itself specifically designed for LLMs, an NTK-linearized task-arithmetic variant, carries its own boundary condition: “since our method is specifically designed for LLMs and relies on the NTK linearization, for small size models, our method may not perform well” (arXiv:2406.11385). The direction of that caveat matters: the method purpose-built for LLMs is the one flagged as unreliable at small scale, while the general-purpose baselines carry LLM evidence. Those LLM numbers are still author-reported, not independently reproduced. There is no scale at which this taxonomy has been independently verified end to end. Diagnose with it; do not trust it blindly.

A related gap: the cited sources do not cover mergekit configuration beyond the toolkit paper itself. The core methods above are what mergekit implements, per that paper, which describes linear averaging, Task Arithmetic, TIES and DARE among its supported approaches (arXiv:2403.13257); it predates PAVE and CUSD, so coverage of those two is not established. Specific flags, defaults, and naming remain outside the cited sources, so verify them against current mergekit docs rather than assuming a config name maps to the method you read about here.

A pre-deployment checklist

Before a merged checkpoint ships, I would want answers to these, in order:

1. Both evals run. New capability and base skill, on the merged checkpoint and on both parents. This is non-negotiable; a single-skill eval cannot detect the failure modes documented here.
2. Symptom classified. Lost base, missing new, or both degraded. This selects the row of the table.
3. Signs and spectra inspected. Parameter-level sign agreement and singular value decay of each task vector. Cheap to compute, directly diagnostic of causes 1 and 2.
4. Cheap fixes tried first. Loss-weighted averaging before any structural pre-processing. The author-reported 5% gain (arXiv:2504.18580) is the best cost-to-benefit among the interventions documented here.
5. Structural fix matched to cause. Sign trimming for sign conflict, pruning or purification for redundancy, direction separation for shared-capacity loss.
6. Compression considered alongside, not after, given the reported interaction between interference reduction and compression (arXiv:2412.00081).
7. Numbers discounted appropriately. Any headline percentage from a single preprint, especially CUSD’s 42-of-44 and 22.5% figures (arXiv:2611.04108), is a hypothesis about your checkpoints, not a measurement of them.
8. Routing priced. Know the serving cost of two checkpoints before you need it as the fallback.

The verdict

Merge when your two-skill audit shows a clean result or one that a matched intervention fixes cheaply. The mechanisms are real and named in the literature: sign conflict, magnitude redundancy, and capacity competition. What has changed recently is the third one. CUSD’s common-versus-unique split reframes the hardest failure as an accounting problem: which directions serve both tasks, and where to intervene. But its gains are author-reported and unreproduced, its model families are not confirmed as LLMs in the available evidence, and the broader literature leans on vision benchmarks. So the durable takeaway is not any one method. It is the audit habit: measure both skills, classify the failure, match the fix, and route to separate checkpoints when the weights genuinely refuse to serve two masters.

Frequently Asked Questions

What is the first step to take when a merged checkpoint underperforms?

When a merged checkpoint underperforms, the useful first move is not to reach for a different merge recipe. It is to figure out which of three documented interference mechanisms you hit.

How can you diagnose if task vectors disagree in sign?

Compute both task vectors, compare their signs parameter by parameter, and measure how much disagreement concentrates in the top-magnitude entries. Heavy disagreement in the large-magnitude tail is the signature of this failure mode.

What is a cheap fix to try before structural surgery for magnitude imbalance?

In parameter-efficient checkpoint merging, loss-weighted averaging “often yields the best results, delivering up to 5% higher task accuracy than the baseline uniform merge” (arXiv:2504.18580), and even surpasses the final individual checkpoint’s performance, the authors report.