Neural Rendering

Neural rendering is the generation of images and video with neural networks that replace or augment parts of the traditional computer graphics pipeline, while keeping control over scene properties such as camera viewpoint, lighting, geometry, and appearance. Conventional rendering computes an image from hand-built 3D models, materials, and light sources using the physics of light transport. Neural rendering instead learns some of these elements from photos and videos, or learns to produce the final pixels directly. As a field, neural rendering combines generative machine learning with physical knowledge from computer graphics, often by building differentiable rendering into network training. Neural rendering powers photorealistic 3D captures of real places, AI upscaling and frame generation in video games, and simulation for robotics and self-driving cars.

How Neural Rendering Works#

The most influential branch learns a 3D scene from ordinary photos. A scene representation, such as a neural network, a grid of learned features, or a set of 3D primitives, is rendered into an image from the camera position of a training photo by a differentiable renderer. Because every step of the renderer can be differentiated, the difference between the rendered image and the real photo can be traced back through backpropagation to every parameter of the scene. Repeating this over many views reconstructs the scene well enough to render viewpoints that no camera captured:

Camera poses for the photos usually come from Structure from Motion software such as COLMAP, which matches features across overlapping photos to recover where each one was taken.

Neural Scene Representations#

Neural radiance fields (NeRF), introduced in 2020, store a scene in the weights of a multilayer perceptron that maps a 3D position and a viewing direction to a density and a color. To render a pixel, NeRF samples points along the camera ray through it and blends their colors by volume rendering: each sample contributes according to its opacity and to how much light survives the samples in front of it. This PyTorch code trains a tiny field to reproduce one observed pixel along one ray:

import torch
from torch import nn

# A tiny neural field that maps a 3D point to a density and an RGB color
torch.manual_seed(0)
field = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 4))
optimizer = torch.optim.Adam(field.parameters(), lr=0.01)

# 64 points along one camera ray, and the color a training photo shows for that pixel
depths = torch.linspace(2.0, 6.0, 64)
points = torch.stack([torch.zeros(64), torch.zeros(64), depths], dim=1)
observed = torch.tensor([0.9, 0.4, 0.1])

for step in range(200):
    density, color = field(points).split([1, 3], dim=1)
    # Volume rendering: each sample's opacity times the light that reaches it through the samples in front
    alpha = 1 - torch.exp(-nn.functional.softplus(density.squeeze(1)) * (depths[1] - depths[0]))
    transmittance = torch.cumprod(torch.cat([torch.ones(1), 1 - alpha[:-1]]), dim=0)
    pixel = (transmittance * alpha) @ torch.sigmoid(color)
    loss = ((pixel - observed) ** 2).sum()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if step in (0, 199):
        print(f"step {step}: rendered {pixel.detach().numpy().round(2)}, loss {loss.item():.4f}")

It prints step 0: rendered [0.71 0.37 0.56], loss 0.2496 and step 199: rendered [0.9 0.4 0.1], loss 0.0000. The untrained network renders an arbitrary color, and gradients flowing back through the rendering equation adjust its weights until the rendered pixel matches the photo. A real NeRF does the same for millions of rays from dozens to hundreds of photos.

The original NeRF needed one to two days of training per scene on an NVIDIA V100 GPU, and each rendered image took 150 to 200 million network queries, about 30 seconds per frame. NVIDIA's Instant NGP replaced most of the network with a multiresolution hash table of learned features, cutting training to seconds or minutes and rendering to tens of milliseconds at 1920×1080. Gaussian splatting went further in 2023, representing scenes with millions of explicit 3D Gaussians that are rasterized in real time with no network evaluated at render time; it is usually grouped with neural rendering because it is optimized the same way.

Image-Space Neural Rendering#

A second branch keeps a conventional renderer and uses networks to finish or extend its output. NVIDIA's DLSS renders games at reduced resolution, upscales them with transformer models, and on GeForce RTX 50 Series GPUs generates up to five frames for every frame the game renders. Path-traced renders use learned denoisers to produce clean images from a few light samples per pixel. Other models synthesize images from scene layouts: NVIDIA's GauGAN turned a semantic map of sky, water, and trees into a photorealistic landscape, and neural avatar systems render a person's face from new angles or under new lighting.

Neural Rendering vs. Related Terms#

• Differentiable rendering: A renderer whose output can be differentiated with respect to its inputs. It is the mechanism that lets neural rendering learn from images, and it is also used without any neural network, for example to fit a mesh to photos.
• Inverse rendering: Recovering geometry, materials, and lighting from images, often with differentiable rendering, so the results can be edited and re-rendered.
• Novel view synthesis: The task of producing images from unseen viewpoints, which NeRF and Gaussian splatting address; neural rendering is the broader set of techniques.
• Generative AI: Text-to-image and video generation models create content from prompts, usually without an explicit scene that can be viewed from a chosen camera position.

Real-World Applications#

• Film and virtual production: Captured locations are reconstructed as radiance fields or splats and rendered from new camera paths for visual effects and previsualization.
• Digital twins and e-commerce: Phone captures of rooms, buildings, and products become 3D views that customers and planners can explore in a browser or AR headset.
• Robotics and autonomous vehicles: Recorded drives are reconstructed and re-rendered from shifted viewpoints, producing synthetic data for testing perception systems. People and cars that move during a capture leave semi-transparent artifacts, so pipelines mask them out of the training photos, and YOLO26 instance segmentation models generate those masks automatically for every frame. Waymo's Block-NeRF reconstructed an entire San Francisco neighborhood from 2.8 million images by stitching together a grid of separately trained NeRFs.