Model Merging

Model merging combines the weights of two or more trained models into a single model, usually by averaging or adding their parameters, so that the result keeps abilities from each without running several networks or retraining on all of their data. It works when the models share an architecture and, for most methods, a common pretrained starting point: fine-tuning moves each copy only a short distance in weight space, so their model weights still line up parameter for parameter. Merging is widely used to build open large language models (LLMs) and to combine LoRA adapters or Stable Diffusion checkpoints, and Meta's Llama 3 post-training averaged models obtained with different data or hyperparameters at each stage.

How Model Merging Works#

Merging methods differ in what they combine and how they resolve conflicts between models:

• Model soups: Average the weights of models fine-tuned from the same base with different hyperparameters. A greedy soup of fine-tuned ViT-G models reached 90.94% top-1 accuracy on ImageNet, a state-of-the-art result at the time.
• Stochastic weight averaging: Averages checkpoints saved at different points of one training run, which tends to settle in a wider, flatter region of the loss surface.
• Task arithmetic: Subtracts the base weights from each fine-tuned model to get a task vector, then adds scaled task vectors to the base to combine skills, or subtracts one to remove a behavior.
• TIES-Merging: Reduces interference between task vectors in three steps: it resets parameters that changed only slightly during fine-tuning, elects a sign for each parameter, and merges only the values that agree with that sign.
• DARE: Randomly drops most of each task vector's values and rescales the remainder before merging, which removes redundant changes while keeping each task's ability.

Averaging along one training run is also part of ordinary training. Ultralytics YOLO26 training keeps an exponential moving average (EMA) of the model's weights alongside the weights being optimized, with a decay that rises toward 0.9999 as training proceeds, and the last.pt and best.pt checkpoints it saves hold this averaged model.

Task-vector methods share one structure: every fine-tuned model is expressed as its difference from the base, the differences are cleaned and combined, and the result is added back to the base:

Open-source tools such as mergekit implement these recipes for LLMs, along with spherical interpolation (SLERP) between two models.

Why a Shared Starting Point Matters#

The hidden units of a neural network can be reordered without changing what the network computes, so two networks trained from different random initializations rarely store the same feature in the same position, and averaging them mixes unrelated weights. Models fine-tuned from one base keep the base's arrangement and stay in the same region of low loss, so their average still works. This example trains a small PyTorch classifier on handwritten digits, fine-tunes two copies on different halves of the remaining training data, and compares their merge with the merge of two models trained from separate initializations:

import copy

import torch
from sklearn.datasets import load_digits
from torch import nn

torch.manual_seed(0)
X, y = load_digits(return_X_y=True)
order = torch.randperm(len(X))
X, y = torch.tensor(X / 16, dtype=torch.float32)[order], torch.tensor(y)[order]


def mlp():
    """Return a small classifier for 8x8 digit images."""
    return nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))


def train(model, rows, steps):
    """Train a model with SGD on the given rows and return it."""
    optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
    for _ in range(steps):
        optimizer.zero_grad()
        nn.functional.cross_entropy(model(X[rows]), y[rows]).backward()
        optimizer.step()
    return model


base = train(mlp(), slice(0, 300), 400)  # shared pretrained starting point
halves = (slice(300, 900), slice(900, 1500))  # two different fine-tuning datasets
for name, (a, b) in [
    ("fine-tuned from one base", [train(copy.deepcopy(base), rows, 100) for rows in halves]),
    ("trained from separate inits", [train(mlp(), rows, 400) for rows in halves]),
]:
    # Merge: average every weight tensor of the two models
    merged = copy.deepcopy(a)
    merged.load_state_dict({k: (a.state_dict()[k] + b.state_dict()[k]) / 2 for k in a.state_dict()})
    accuracy = [(m(X[1500:]).argmax(1) == y[1500:]).float().mean().item() for m in (a, b, merged)]
    print(f"{name}: {accuracy[0]:.3f} and {accuracy[1]:.3f}, merged {accuracy[2]:.3f}")

On the 297 held-out digits, the two models fine-tuned from one base score 0.956 and 0.946, and their average scores 0.970, higher than either, because it combines what each learned from its half of the data. The two models trained from separate initializations score 0.946 and 0.956, and their average falls to 0.808. Git Re-Basin merges independently trained models by first permuting one model's hidden units to match the other's.

Model Merging vs. Related Terms#

• Model ensemble: Keeps every model and combines their outputs, so memory use and inference latency grow with the number of models. A merged model has the size and speed of one.
• Knowledge distillation: Trains a new student model to imitate one or more teachers, which needs training data and compute. Merging is a direct arithmetic operation on existing weights.
• Multi-task training: Trains one model on all tasks' data together, which usually sets the accuracy that merging tries to reach without access to that data.
• Federated learning: Averages the weights of models trained on separate devices in every round. Each round starts all clients from the same global model, the shared starting point that makes the average work.