Transformer Math

You have already spent the GPU budget. Three separate fine-tunes exist on disk: one specialized for code, one for math reasoning, one for medical Q&A. None of them is best at all three. Retraining a single model on a combined dataset would take days and could degrade each specialty.

Model merging is the alternative: arithmetic on weight checkpoints. Add the task-specific deltas— the difference between a fine-tuned model and its base — directly to the base model's weights. No forward pass. No gradient. Zero GPU-hours. The merged model often outperforms any individual fine-tune on held-out benchmarks.

This is not an ensemble (which multiplies inference cost). The merged checkpoint is a single model with the same parameter count as the base. Serving cost is identical.

The problem.Two fine-tunes that start from the same pre-trained checkpoint are not independent. The pre-training landscape has already carved out a basin of weights that know about language. Fine-tuning keeps the model inside that basin — it moves the weights to a new region that also knows the target task, but it doesn't escape the original basin. Both fine-tunes live in the same valley.

Linear mode connectivity (Frankle et al., 2020) is the empirical finding that when two models share the same initialization, there exists a low-loss path between their weight vectors. You can walk from one to the other along a straight line in weight space without crossing a high-loss barrier. This is what makes interpolation work — you stay in the valley the whole time.

Task vectors. Ilharco et al. (2022) framed this as arithmetic on task vectors: . Each task vector encodes the directional change from base to fine-tune. Adding two task vectors to the base adds both sets of task-specific changes. The result is a model that has been pulled in both directions simultaneously. Empirically, for tasks that do not conflict, the pulls reinforce.

1. SLERP

Spherical Linear Interpolation treats parameter tensors as vectors on a high-dimensional unit sphere. For two weight vectors and separated by angle :

At you get ; at you get . The sine weighting keeps for all when .

2. TIES-Merging (Trim · Elect Sign · Disjoint Merge)

Given fine-tunes, compute task vectors for each. Then:

1. Trim: . Zeroes out the (100-k)% lowest-magnitude changes.
2. Elect Sign: . Majority-vote sign per parameter position.
3. Disjoint Merge: , where . Only models that agree with the elected sign are averaged at each position.

3. DARE

Drop And REscale sparsifies each task vector before merging:

With , 90% of deltas are zeroed. The survivors are scaled up by 10× so their expectation matches the original delta. DARE is typically composed with TIES after the drop: DARE-TIES.

4. Model Soups (Weight Averaging)

Average the weights of multiple checkpoints fine-tuned from the same base (different hyperparameters or epochs). The “greedy soup” variant adds a checkpoint only if the running average improves validation accuracy:

Model soups exploit flat loss basins: interpolating between nearby minima in the same basin reaches regions with lower curvature and better generalization than any single minimum.

# pip install mergekit
import yaml
import subprocess

# Define a TIES merge of two Mistral-7B fine-tunes
merge_config = {
    "merge_method": "ties",
    "base_model": "mistralai/Mistral-7B-v0.1",
    "models": [
        {
            "model": "your-org/mistral-7b-math",
            "parameters": {"density": 0.5, "weight": 1.0},
        },
        {
            "model": "your-org/mistral-7b-code",
            "parameters": {"density": 0.5, "weight": 1.0},
        },
    ],
    "parameters": {
        "normalize": True,
        "int8_mask": True,
    },
    "dtype": "bfloat16",
}

with open("/tmp/merge_config.yml", "w") as f:
    yaml.dump(merge_config, f)

subprocess.run([
    "mergekit-yaml",
    "/tmp/merge_config.yml",
    "/tmp/merged-model",
    "--cuda",          # GPU acceleration
    "--low-cpu-memory",  # stream layers to avoid OOM
], check=True)

# Load and test the merged model
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("/tmp/merged-model")
model = AutoModelForCausalLM.from_pretrained(
    "/tmp/merged-model", torch_dtype="auto"
)
print("Merge successful. Parameter count:", sum(p.numel() for p in model.parameters()))