Visual Autoregressive Modeling (VAR)
Visual autoregressive modeling (VAR) generates images coarse to fine, predicting each next-resolution token map in parallel. Covers tokenizer and scaling.
Visual autoregressive modeling (VAR) is a generative AI method for images that predicts an image as a series of token maps of increasing resolution, from a single 1 x 1 token up to the full grid, instead of predicting tokens one at a time in reading order. Each step generates every token of the next scale in parallel, conditioned on all coarser scales, so a 256 x 256 image takes 10 steps. Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang of Peking University and ByteDance introduced VAR in April 2024 in Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, which won the NeurIPS 2024 Best Paper Award. On class-conditional ImageNet 256 x 256 generation, it improved an autoregressive baseline's Fréchet Inception Distance (FID) from 18.65 to 1.73 and its Inception Score from 80.4 to 350.2, with about 20 times faster inference.
Why Predict the Next Scale#
Earlier autoregressive image generators, such as the transformer in VQGAN and Google's Parti, convert an image into a grid of discrete tokens, flatten it row by row, and predict one token at a time, the way a GPT model writes text. That order fits images poorly. A token can only see the tokens above it and to its left, although image regions depend on each other in every direction, and a 16 x 16 grid takes 256 sequential steps. Predicting the next scale keeps the transformer's training objective and its causal ordering but changes the unit of prediction from one token to one whole resolution: tokens within a scale see each other, and every scale sees the complete image at all coarser resolutions.
The Multi-Scale Tokenizer#
VAR's tokenizer is an autoencoder whose latent is compressed with multi-scale vector quantization. Its encoder turns a 256 x 256 image into a 16 x 16 feature map, which is quantized at 10 scales from 1 x 1 to 16 x 16 with one shared codebook of 4,096 entries. Each scale quantizes only what the coarser scales left unexplained, the residual, so the coarse maps hold overall color and layout and the later maps add finer detail. The same residual scheme runs below on a downsampled photo with a random codebook:
import torch
import torch.nn.functional as F
from skimage import data
torch.manual_seed(0)
image = torch.tensor(data.astronaut(), dtype=torch.float32).permute(2, 0, 1)[None] / 255 # 1x3x512x512
features = F.interpolate(image, size=(16, 16), mode="area") # stand-in for a 16x16 encoder feature map
codebook = torch.rand(4096, 3) * 2 - 1 # one shared codebook of 4,096 entries; residuals can be negative
# Multi-scale residual quantization: each scale encodes what the coarser scales left out
residual, reconstruction = features.clone(), torch.zeros_like(features)
for side in (1, 2, 3, 4, 5, 6, 8, 10, 13, 16): # VAR's ten token-map sizes for 256 x 256 images
small = F.interpolate(residual, size=(side, side), mode="area").permute(0, 2, 3, 1).reshape(-1, 3)
tokens = torch.cdist(small, codebook).argmin(dim=1) # side x side discrete token map
quantized = codebook[tokens].reshape(1, side, side, 3).permute(0, 3, 1, 2)
upsampled = F.interpolate(quantized, size=(16, 16), mode="bicubic")
reconstruction, residual = reconstruction + upsampled, residual - upsampled
print(f"{side:>2}x{side:<2} tokens -> reconstruction error {(features - reconstruction).pow(2).mean():.5f}")The reconstruction error falls from 0.06267 after the single 1 x 1 token to 0.00212 after the 16 x 16 map, using 680 tokens across the ten scales. VAR trains its encoder, decoder, and codebook jointly. Detection models are built on the same kind of pyramid for recognition: Ultralytics YOLO26 predicts objects from feature pyramid levels at strides 8, 16, and 32, so a 640-pixel image yields 80 x 80, 40 x 40, and 20 x 20 maps, with large objects found on the coarse map and small ones on the fine map.
How VAR Generates an Image#
A GPT-style transformer generates the token maps in order, with a block-wise causal attention mask that lets each token attend to its own scale and every coarser scale:
The authors trained models of up to 2 billion parameters and found power-law scaling laws like those of large language models, with linear correlation coefficients near -0.998 between model size and test loss. The same model performs image inpainting, outpainting, and class-conditional editing zero-shot, with no extra training, by fixing the tokens outside the edited region to their ground-truth values while sampling the rest.
VAR Successors and Related Terms#
- Infinity: ByteDance's December 2024 text-to-image successor replaces the 4,096-entry codebook with bitwise tokens from a tokenizer with an effectively unlimited vocabulary. It generates a 1024 x 1024 image in 0.8 seconds, 2.6 times faster than Stable Diffusion 3 Medium, and raised the GenEval score from SD3 Medium's 0.62 to 0.73.
- Diffusion transformers (DiT): Denoise continuous latents over many steps. The VAR paper reports beating DiT on image quality, inference speed, data efficiency, and scalability.
- Diffusion models: Refine every pixel or latent of the image at every step, starting from Gaussian noise.
- Masked generative transformers: Google's MaskGIT predicts all masked tokens of a single-resolution grid in parallel and keeps the most confident ones over about 8 to 12 iterations.










