Generative Adversarial Network (GAN)

A generative adversarial network (GAN) is a generative AI model built from two neural networks trained against each other: a generator turns random noise into samples, and a discriminator tries to tell those samples apart from real training data. Once training ends, the generator produces new images, audio, or tabular records in a single forward pass. Ian Goodfellow and colleagues at the Université de Montréal introduced GANs in the 2014 paper Generative Adversarial Nets, and GANs led image synthesis until diffusion models overtook them in 2021.

How GAN Training Works#

The two networks play a minimax game. The discriminator is a binary classifier that outputs the probability that its input is real, trained with a cross-entropy loss function to score real samples 1 and generated ones 0. The generator never sees real data and learns only from the discriminator's gradients. At the game's theoretical optimum, the generated distribution equals the data distribution and the discriminator outputs 0.5 for every input. Early in training, when fakes are easy to spot, the original generator objective gives almost no gradient, a form of the vanishing gradient problem, so Goodfellow's paper had the generator maximize the probability that the discriminator calls its samples real. The code below uses this non-saturating loss.

A one-dimensional GAN in PyTorch learns a Gaussian distribution from samples alone:

import torch
from torch import nn

torch.manual_seed(0)

# Generator maps 8-D noise to a number; discriminator scores a number as real (high) or fake (low)
generator = nn.Sequential(nn.Linear(8, 32), nn.ReLU(), nn.Linear(32, 1))
discriminator = nn.Sequential(nn.Linear(1, 32), nn.LeakyReLU(0.2), nn.Linear(32, 1))
opt_g = torch.optim.Adam(generator.parameters(), lr=1e-3)
opt_d = torch.optim.Adam(discriminator.parameters(), lr=1e-3)
bce = nn.BCEWithLogitsLoss()
real_label, fake_label = torch.ones(64, 1), torch.zeros(64, 1)

for step in range(3000):
    real = 4.0 + 1.25 * torch.randn(64, 1)  # training data: samples from N(4, 1.25^2)
    fake = generator(torch.randn(64, 8))

    # Discriminator step: label real samples 1 and generated samples 0
    loss_d = bce(discriminator(real), real_label) + bce(discriminator(fake.detach()), fake_label)
    opt_d.zero_grad()
    loss_d.backward()
    opt_d.step()

    # Generator step: push the discriminator to label generated samples 1
    loss_g = bce(discriminator(fake), real_label)
    opt_g.zero_grad()
    loss_g.backward()
    opt_g.step()

samples = generator(torch.randn(10000, 8)).detach()
print(f"Generated mean {samples.mean():.2f}, std {samples.std():.2f} (data: 4.00, 1.25)")
print(f"Discriminator's average P(real) on generated samples: {torch.sigmoid(discriminator(samples)).mean():.2f}")

After 3,000 alternating steps, the generated samples have a mean of 3.95 and a standard deviation of 1.20, against the data's 4.00 and 1.25, and the discriminator gives them an average probability of 0.51 of being real, close to the coin flip of the theoretical optimum.

Key GAN Variants#

• DCGAN: Alec Radford, Luke Metz, and Soumith Chintala's 2015 design made convolutional generators and discriminators train reliably, replacing pooling layers with strided convolutions and adding batch normalization.
• Conditional GANs and pix2pix: Feeding a class label or an input image to both networks lets the generator produce a requested output. Phillip Isola and colleagues' pix2pix (2017) learns paired translations such as label maps to street photos and edges to product photos.
• CycleGAN: Learns translation between two image collections without paired examples, such as horses to zebras or summer to winter landscapes, by requiring that a translated image map back to the original.
• Wasserstein GAN (WGAN): Martin Arjovsky and colleagues replaced the classifier with a critic that estimates the Wasserstein distance between real and generated data (2017), which keeps gradients useful even when the two distributions barely overlap.
• StyleGAN: NVIDIA's StyleGAN (2018) and StyleGAN2 (2019) feed a learned style vector into every generator layer. StyleGAN2 reached a Fréchet Inception Distance (FID) of 2.84 on 1024 x 1024 faces from the FFHQ dataset.

Applications#

• Super-resolution: SRGAN (2017), ESRGAN (2018), and Real-ESRGAN (2021) add an adversarial loss to upscaling networks, which restores sharp textures where a pixel-wise loss alone produces blur.
• Medical image augmentation: Maayan Frid-Adar and colleagues trained GANs on CT scans of 182 liver lesions and added the synthetic lesions to training, which raised a classifier's sensitivity from 78.6% to 85.7% and its specificity from 88.4% to 92.4%.
• Deepfakes: In a 2022 study by Sophie Nightingale and Hany Farid, participants identified StyleGAN2 faces as real or synthetic with 48% accuracy, slightly worse than chance, and rated the synthetic faces as more trustworthy.

GAN-generated images serve as synthetic data wherever real examples are scarce, as a layer of data augmentation on top of geometric transforms such as flips and rotations. They enter a YOLO26 training set like any other images: the train split in an Ultralytics detection dataset YAML accepts a list of directories, so synthetic images can sit in their own folder beside the real ones and be added or removed by editing one line.

GANs vs. Diffusion Models and VAEs#

Diffusion models overtook GANs on image quality after OpenAI's Prafulla Dhariwal and Alex Nichol reported in 2021 that a diffusion model scored a better FID on ImageNet than BigGAN-deep, the strongest GAN at the time. Diffusion training is a stable regression problem with no adversary, and it covers the variety of the data more fully. GAN training can oscillate without converging or fall into mode collapse, in which the generator settles on a few outputs that fool the discriminator and stops producing the rest. Variational autoencoders train stably as well, but their samples are blurrier. The GAN's lasting advantage is speed, since sampling takes one forward pass: GigaGAN, from Adobe Research, Carnegie Mellon University, and POSTECH, generated text-to-image results at 512 pixels in 0.13 seconds in 2023. Adversarial losses also survive inside diffusion systems. The autoencoder of Stable Diffusion is trained with a patch-based discriminator, and Stability AI's SDXL Turbo uses adversarial diffusion distillation, a form of knowledge distillation with a discriminator in the loss, to generate an image in a single step.