Model Collapse

Model collapse is a degenerative process in which generative models trained on data produced by earlier models lose information about the original data distribution, generation after generation. Rare cases in the tails of the distribution disappear first, and after enough rounds the outputs converge on a narrow, repetitive version of the most common patterns. Shumailov et al. demonstrated the effect in a 2024 Nature paper for large language models (LLMs), variational autoencoders, and Gaussian mixture models. The risk grows as AI-generated text and images make up more of the web data that future models are trained on.

Why Collapse Happens#

Each generation fits a model to a finite sample and then produces the next generation's data by sampling from that fit. Shumailov et al. identify three errors that compound across generations: statistical approximation error, because a finite sample misses rare events; functional expressivity error, because the model cannot represent the true distribution exactly; and functional approximation error, because training procedures such as stochastic gradient descent are imperfect. Once a rare event is missing from one generation's data, the next model assigns it zero probability and it never returns. They distinguish early collapse, where the tails vanish, from late collapse, where the distribution narrows toward a few common outputs with little variance.

The simulation below treats a word-frequency table as the model. Each generation counts 10,000 words sampled from the previous model, either replacing the earlier data or adding to it:

import torch

torch.manual_seed(0)
vocab = 1000
human = 1 / torch.arange(1, vocab + 1)  # Zipf-like word frequencies: a few common words, a long tail
human = human / human.sum()
real = torch.multinomial(human, 10_000, replacement=True)  # human-written training text
print(f"Human data: {real.unique().numel()} distinct words")

for mode in ("replace", "accumulate"):
    data = real
    for generation in range(20):
        # "Train" a model by counting word frequencies, then sample the next generation's text from it
        model = torch.bincount(data, minlength=vocab) / len(data)
        samples = torch.multinomial(model, 10_000, replacement=True)
        data = samples if mode == "replace" else torch.cat([data, samples])
    print(f"{mode}: {data.unique().numel()} distinct words after 20 generations")

The human sample contains 916 of the 1,000 words. After 20 generations that each train only on the previous generation's output, 302 words remain, and the other 614, mostly from the long tail, are gone for good. When each generation's samples are added to the human data instead, all 916 survive, because the real data stays in every training set.

Evidence and Limits#

The replace-versus-accumulate distinction holds for real models. Gerstgrasser et al. (2024) pretrained sequences of language models and found that replacing real data with each generation's synthetic data led toward collapse, while accumulating synthetic data alongside the original real data avoided it, with matching results for diffusion models and variational autoencoders. For image generators, Alemohammad et al. found that without enough fresh real data in each generation of a self-consuming loop, quality or diversity progressively decreases, and that biasing samples toward quality, such as cherry-picking the best outputs, preserves quality at the expense of diversity. Deliberately generated and filtered synthetic data remains widely used, as in knowledge distillation from a stronger teacher model; collapse comes from indiscriminate, recursive reuse of a model's own unfiltered outputs.

Model Collapse vs. Related Concepts#

• Mode collapse: A generative adversarial network (GAN) generator learns to produce only a few kinds of output within a single training run, for example the same face repeatedly. Model collapse unfolds across successive models trained on each other's data.
• Catastrophic forgetting: A model loses an earlier task after training on a new one. In model collapse the task stays the same and the training data degrades.
• Overfitting: A model memorizes its training set and generalizes poorly. A collapsed model can fit its training data well, but that data no longer represents the real distribution.
• AI slop: Low-quality AI-generated content published online, one of the sources that can feed collapse when it is scraped into training sets.

Preventing Model Collapse#

Web-scale training closes the loop between models and the data that trains them:

The main defenses follow from the evidence: keep original human data in every training set and add synthetic data to it rather than replacing it, track data provenance so generated samples can be identified, using signals such as AI watermarking and C2PA content credentials, and filter or verify synthetic samples before training on them. Ultralytics Platform applies the accumulate pattern to vision datasets: its Generate similar images action creates 1 to 16 generated variations of an existing image, and the ones a user keeps are added as new, unlabeled images next to the real ones, ready for annotation.