Novel View Synthesis (NVS)

Novel view synthesis (NVS) is the task of generating an image of a scene from a camera position that was never photographed, given one or more input images. A model has to infer where surfaces are, how they look from the new angle, and what fills the regions that every input camera missed. NVS powers free-viewpoint video, virtual tours, 3D video calls, and simulators that replay recorded drives from new trajectories, and it is a standard test for 3D computer vision models, since held-out photos show directly whether a scene representation holds up from new angles.

How Novel View Synthesis Works#

The simplest approach reprojects pixels. With a depth value for every pixel, each point can be moved to where a new camera would see it, a technique called depth-image-based rendering. Points closer to the camera shift farther than distant ones, and surfaces hidden in the input but visible from the new position leave holes, called disocclusions, that must be filled. This example synthesizes the right view of a Middlebury stereo pair from the left image and its disparity, a form of inverse depth, then compares the result with the real right photo that the warp never saw:

import numpy as np
from skimage import data
from skimage.metrics import peak_signal_noise_ratio as psnr

# Left view, the real right view (held out), and per-pixel disparity (inverse depth) for the left view
left, right, disp = data.stereo_motorcycle()
ys, xs = np.nonzero(np.isfinite(disp))
xt = np.round(xs - disp[ys, xs]).astype(int)  # column where the right camera, 0.19 m away, sees each pixel

# Paint far to near so the nearest surface wins where several pixels land on one spot
order = np.argsort(disp[ys, xs])
ys, xs, xt = ys[order], xs[order], xt[order]
keep = xt >= 0  # drop pixels that leave the frame
novel = np.zeros_like(left)
novel[ys[keep], xt[keep]] = left[ys[keep], xs[keep]]

# Holes are spots no left pixel reached, such as surfaces the left camera never saw
filled = novel.any(axis=2)
print(f"Holes: {1 - filled.mean():.1%} of the novel view")
print(f"PSNR, synthesized view: {psnr(right[filled], novel[filled]):.1f} dB")
print(f"PSNR, left view copied: {psnr(right[filled], left[filled]):.1f} dB")

The synthesized view reaches a peak signal-to-noise ratio of 26.9 dB measured on the real photo, compared with 12.9 dB for simply reusing the left image, while 17.0% of the new view is left as holes. On a single photo, the disparity can come from monocular depth estimation: Ultralytics YOLO26 depth models predict each pixel's depth in meters, which converts to disparity for any chosen camera shift. The 3D photo method of Meng-Li Shih and colleagues at CVPR 2020 follows this route and fills the holes with learned image inpainting.

Dense captures support per-scene optimization. Given dozens to hundreds of photos with camera poses, usually from structure from motion, neural radiance fields, introduced by Ben Mildenhall and colleagues in 2020, fit a network that returns color and density at any 3D point and render new views by volume rendering along each camera ray. Gaussian splatting, introduced by Bernhard Kerbl and colleagues in 2023, replaces the network with millions of 3D Gaussians that rasterize in real time, at 30 frames per second or more at 1080p. Both are trained with differentiable rendering by comparing their renders with the input photos.

With one or a few inputs, generative models supply what the cameras never saw. Diffusion models conditioned on an input image and a relative camera pose, such as Zero-1-to-3 from Columbia University in 2023, generate plausible views of the far side of an object, and multi-view diffusion methods generate several consistent views and then fit a 3D representation to them.

Quality is measured on held-out photos with PSNR, the structural similarity index (SSIM), and LPIPS, a learned perceptual distance, on benchmarks such as the synthetic Blender scenes from the NeRF paper and the unbounded real scenes of Mip-NeRF 360.

Related Concepts#

Neural rendering is the broader family of graphics techniques built on neural networks, covering NVS along with relighting, avatars, and image editing. 3D reconstruction aims at measurable geometry such as meshes, while NVS is judged only on the rendered images, so a radiance field can render convincingly with fuzzy geometry. Stereo vision runs the warp in the opposite direction, measuring disparity from two real views. Frame interpolation creates new moments in time from a fixed viewpoint, and camera-controlled video generation blends both, moving a virtual camera through a generated scene. Dynamic scenes extend NVS to time with methods such as 4D Gaussian splatting.

Real-World Applications#

• 3D video calls: Google Beam, formerly Project Starline, captures each participant with multiple cameras and renders them from the viewer's eye position on a light field display.
• Virtual reality and mixed reality: Captured rooms, products, and landmarks become explorable scenes that a headset re-renders for every head position.
• Driving simulation: Neural reconstructions of recorded drives render camera views from trajectories the car never took, producing synthetic sensor data for testing perception. Waymo and Google researchers built Block-NeRF from 2.8 million images to render an entire San Francisco neighborhood.