License YOLO
Back to Ultralytics Glossary

Visual Autoregressive Modeling (VAR)

Visual autoregressive modeling (VAR) generates images coarse to fine, predicting each next-resolution token map in parallel. Covers tokenizer and scaling.

Visual autoregressive modeling (VAR) is a generative AI method for images that predicts an image as a series of token maps of increasing resolution, from a single 1 x 1 token up to the full grid, instead of predicting tokens one at a time in reading order. Each step generates every token of the next scale in parallel, conditioned on all coarser scales, so a 256 x 256 image takes 10 steps. Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang of Peking University and ByteDance introduced VAR in April 2024 in Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, which won the NeurIPS 2024 Best Paper Award. On class-conditional ImageNet 256 x 256 generation, it improved an autoregressive baseline's Fréchet Inception Distance (FID) from 18.65 to 1.73 and its Inception Score from 80.4 to 350.2, with about 20 times faster inference.

Why Predict the Next Scale#

Earlier autoregressive image generators, such as the transformer in VQGAN and Google's Parti, convert an image into a grid of discrete tokens, flatten it row by row, and predict one token at a time, the way a GPT model writes text. That order fits images poorly. A token can only see the tokens above it and to its left, although image regions depend on each other in every direction, and a 16 x 16 grid takes 256 sequential steps. Predicting the next scale keeps the transformer's training objective and its causal ordering but changes the unit of prediction from one token to one whole resolution: tokens within a scale see each other, and every scale sees the complete image at all coarser resolutions.

The Multi-Scale Tokenizer#

VAR's tokenizer is an autoencoder whose latent is compressed with multi-scale vector quantization. Its encoder turns a 256 x 256 image into a 16 x 16 feature map, which is quantized at 10 scales from 1 x 1 to 16 x 16 with one shared codebook of 4,096 entries. Each scale quantizes only what the coarser scales left unexplained, the residual, so the coarse maps hold overall color and layout and the later maps add finer detail. The same residual scheme runs below on a downsampled photo with a random codebook:

import torch
import torch.nn.functional as F
from skimage import data

torch.manual_seed(0)
image = torch.tensor(data.astronaut(), dtype=torch.float32).permute(2, 0, 1)[None] / 255  # 1x3x512x512
features = F.interpolate(image, size=(16, 16), mode="area")  # stand-in for a 16x16 encoder feature map
codebook = torch.rand(4096, 3) * 2 - 1  # one shared codebook of 4,096 entries; residuals can be negative

# Multi-scale residual quantization: each scale encodes what the coarser scales left out
residual, reconstruction = features.clone(), torch.zeros_like(features)
for side in (1, 2, 3, 4, 5, 6, 8, 10, 13, 16):  # VAR's ten token-map sizes for 256 x 256 images
    small = F.interpolate(residual, size=(side, side), mode="area").permute(0, 2, 3, 1).reshape(-1, 3)
    tokens = torch.cdist(small, codebook).argmin(dim=1)  # side x side discrete token map
    quantized = codebook[tokens].reshape(1, side, side, 3).permute(0, 3, 1, 2)
    upsampled = F.interpolate(quantized, size=(16, 16), mode="bicubic")
    reconstruction, residual = reconstruction + upsampled, residual - upsampled
    print(f"{side:>2}x{side:<2} tokens -> reconstruction error {(features - reconstruction).pow(2).mean():.5f}")

The reconstruction error falls from 0.06267 after the single 1 x 1 token to 0.00212 after the 16 x 16 map, using 680 tokens across the ten scales. VAR trains its encoder, decoder, and codebook jointly. Detection models are built on the same kind of pyramid for recognition: Ultralytics YOLO26 predicts objects from feature pyramid levels at strides 8, 16, and 32, so a 640-pixel image yields 80 x 80, 40 x 40, and 20 x 20 maps, with large objects found on the coarse map and small ones on the fine map.

How VAR Generates an Image#

A GPT-style transformer generates the token maps in order, with a block-wise causal attention mask that lets each token attend to its own scale and every coarser scale:

The authors trained models of up to 2 billion parameters and found power-law scaling laws like those of large language models, with linear correlation coefficients near -0.998 between model size and test loss. The same model performs image inpainting, outpainting, and class-conditional editing zero-shot, with no extra training, by fixing the tokens outside the edited region to their ground-truth values while sampling the rest.

  • Infinity: ByteDance's December 2024 text-to-image successor replaces the 4,096-entry codebook with bitwise tokens from a tokenizer with an effectively unlimited vocabulary. It generates a 1024 x 1024 image in 0.8 seconds, 2.6 times faster than Stable Diffusion 3 Medium, and raised the GenEval score from SD3 Medium's 0.62 to 0.73.
  • Diffusion transformers (DiT): Denoise continuous latents over many steps. The VAR paper reports beating DiT on image quality, inference speed, data efficiency, and scalability.
  • Diffusion models: Refine every pixel or latent of the image at every step, starting from Gaussian noise.
  • Masked generative transformers: Google's MaskGIT predicts all masked tokens of a single-resolution grid in parallel and keeps the most confident ones over about 8 to 12 iterations.

Explore solutions

Computer vision in sports

Computer vision in sports

Discover how Ultralytics YOLO powers sports vision with player and ball tracking, performance analytics, coaching insights, and athlete rehabilitation.
Learn more
Computer vision for aerial imagery

Computer vision for aerial imagery

Turn drone and aerial imagery into real-time insights for agriculture, aquaculture, conservation, and geospatial analysis with Ultralytics YOLO.
Learn more
Computer vision in aerospace

Computer vision in aerospace

Apply computer vision in aerospace with Ultralytics YOLO for aircraft inspection, runway FOD detection, airport ground operations, and composite defect detection.
Learn more
Computer vision in security

Computer vision in security

Protect every site with Ultralytics YOLO models. Vision AI powers perimeter monitoring, threat detection, license-plate recognition, and workplace safety.
Learn more
Computer vision in robotics

Computer vision in robotics

Power smarter machines with Ultralytics YOLO models. Vision AI in robotics drives autonomous navigation, perception, object tracking, and real-time control.
Learn more
Computer vision in logistics

Computer vision in logistics

Streamline logistics with Ultralytics YOLO models. Vision AI enables package inspection, sorting, vehicle tracking, and real-time warehouse safety monitoring.
Learn more
Computer vision in retail

Computer vision in retail

Reimagine retail with Ultralytics YOLO models. Vision AI powers inventory tracking, shelf monitoring, queue management, and smarter customer insights.
Learn more
Computer vision in healthcare

Computer vision in healthcare

Build healthcare solutions with Ultralytics YOLO models. Vision AI in healthcare powers faster medical imaging, smarter diagnostics, and patient monitoring.
Learn more
Computer vision in manufacturing

Computer vision in manufacturing

Optimize manufacturing with Ultralytics YOLO models. Vision AI drives quality control, defect detection, PPE compliance, and assembly line automation.
Learn more
Computer vision in automotive

Computer vision in automotive

Apply computer vision in automotive with Ultralytics YOLO models. Vision AI elevates road safety, driver assistance, and vehicle automation for smarter roads.
Learn more
Computer vision in agriculture

Computer vision in agriculture

Bring vision AI to smart agriculture with Ultralytics YOLO models. Power crop monitoring, livestock tracking, and precision farming for higher, smarter yields.
Learn more

Let's build the future of AI together!

Begin your journey with the future of machine learning