Neural Style Transfer

Neural style transfer (NST) is a computer vision technique that renders the content of one image, such as a photo of a street, in the visual style of another, such as a painting, by matching the internal features of a pretrained convolutional neural network (CNN). Content is captured by a deep layer's feature maps, which record what appears where. Style is captured by the correlations between feature channels, which record textures, colors, and brushstrokes but discard their arrangement. Leon Gatys, Alexander Ecker, and Matthias Bethge of the University of Tübingen introduced the method in August 2015 in A Neural Algorithm of Artistic Style, and the Prisma app brought it to smartphone users the following year.

How the Original Method Works#

Gatys's method uses VGG-19, a CNN trained on ImageNet, as a fixed feature extractor, and it changes the pixels of the output image instead of any network weights. Two loss functions drive the result:

• Content loss: The squared difference between the output's and the content photo's feature maps at one deep layer, conv4_2, which keeps objects and layout recognizable.
• Style loss: The squared difference between Gram matrices at five layers from conv1_1 to conv5_1. For a layer with C channels, the Gram matrix is a C x C table whose entry (i, j) measures how strongly channels i and j respond together, summed over every position in the image.

A weighted sum of the two losses is minimized with the L-BFGS optimization algorithm, and backpropagation carries the gradient through the frozen network to the pixels:

Because the Gram matrix sums over positions, it barely changes when the image's layout does. Swapping the quadrants of a photo shows this with torchvision's ResNet-18, a smaller ImageNet CNN than VGG-19, in PyTorch:

import torch
from skimage import data
from torchvision.models import ResNet18_Weights, resnet18

weights = ResNet18_Weights.DEFAULT
cnn = resnet18(weights=weights).eval()
early_layers = torch.nn.Sequential(cnn.conv1, cnn.bn1, cnn.relu, cnn.maxpool, cnn.layer1)
preprocess = weights.transforms()

photo = torch.tensor(data.astronaut()).permute(2, 0, 1)  # 3 x 512 x 512
rearranged = torch.roll(photo, shifts=(256, 256), dims=(1, 2))  # same pixels, quadrants swapped


def content_and_style(image):
    """Return an early CNN layer's feature map (content) and its Gram matrix (style)."""
    with torch.no_grad():
        features = early_layers(preprocess(image)[None])[0].flatten(1)  # channels x positions
    return features, features @ features.T / features.shape[1]


(content_a, style_a), (content_b, style_b) = content_and_style(photo), content_and_style(rearranged)
print(f"Content change: {(content_a - content_b).norm() / content_a.norm():.0%}")
print(f"Style change:   {(style_a - style_b).norm() / style_a.norm():.0%}")

Swapping the quadrants changes the feature map by 96% but the Gram matrix by only 7%, and the 7% that remains comes from the new seams and from the center crop, which keeps a different part of each image. This is why style loss copies a painting's brushwork and palette without copying where anything in the painting was.

Fast and Arbitrary Style Transfer#

Optimizing each image takes hundreds of iterations, so later work moved the optimization into training. Justin Johnson, Alexandre Alahi, and Li Fei-Fei of Stanford trained a feed-forward network per style with the same losses in 2016, producing results about three orders of magnitude faster, and applied the same perceptual losses to super-resolution. Dmitry Ulyanov and colleagues found that replacing batch normalization with instance normalization in those networks improved stylization quality. Xun Huang and Serge Belongie's adaptive instance normalization (AdaIN, 2017) handles any style without retraining: it shifts the mean and standard deviation of each content feature channel to match the style image's, then decodes, which runs in real time.

Style Transfer for Robust Vision Models#

Robert Geirhos and colleagues used AdaIN to show that ImageNet-trained CNNs rely more on texture than on shape: a ResNet-50 labels an image of a cat with the texture of elephant skin as an elephant, a form of shortcut learning. Their Stylized-ImageNet restyled every ImageNet image with a random painting from the 79,434-painting Painter by Numbers collection, and a ResNet-50 trained on it relied on shape instead. Using that shape-based network as the backbone of a Faster R-CNN object detection model raised mAP50 on Pascal VOC 2007 from 70.7 to 75.1 and on MS COCO from 52.3 to 55.2.

The same idea works as data augmentation for detection datasets, because style transfer keeps every object where it was. A stylized copy of a labeled image reuses the original's bounding boxes, so in an Ultralytics YOLO26 detection dataset it needs only a copy of the original label file under the new image name.

Neural Style Transfer vs. Related Terms#

• Generative adversarial networks (GANs): CycleGAN learns a mapping between whole image collections, such as photos and Monet paintings, instead of matching the statistics of one style image.
• Transfer learning: Fine-tunes pretrained weights for a new task. NST never updates the network's weights and uses it only to measure the losses.
• Diffusion models: Restyle images from a text prompt or a reference image by regenerating them, which can change shapes and details along with the style.