Differentiable Rendering

Differentiable rendering is rendering, turning a description of a 3D scene's geometry, materials, lighting, and camera into an image, implemented so that the derivative of every output pixel with respect to those scene parameters can be computed. A loss function that compares a rendered image with a photo can then be minimized by gradient descent, with backpropagation running through the renderer into the 3D scene. This makes it the engine of inverse rendering, the computer vision task of recovering a scene from its images, and of neural rendering methods such as neural radiance fields (NeRF) and Gaussian splatting, which train their scene models by rendering them.

Why Rendering Needs Special Handling#

A standard rasterizer is not differentiable where it matters. Whether a pixel is covered by a triangle is a step function of the vertex positions, so moving a vertex slightly changes no pixel until an edge crosses a pixel center, and the gradient with respect to geometry is zero almost everywhere; occlusion adds more discontinuities wherever one surface hides another. Differentiable renderers handle these edges in different ways:

• Approximate gradients: OpenDR (2014) and the Neural 3D Mesh Renderer (2018) keep a standard rasterizer for the forward pass and approximate the gradients in the backward pass.
• Soft rasterization: SoftRas (2019) replaces each pixel's hard coverage with a probability that falls off smoothly with distance from a triangle's edge and blends overlapping faces, so every pixel near an edge passes gradient to the geometry.
• Physically based differentiable rendering: Ray tracers such as redner (2018) and Mitsuba 3 (2022) sample silhouette edges or reparameterize the rendering integral, giving gradients for shadows, reflections, and indirect light.
• Volume and splat rendering: NeRF's alpha compositing of densities along each ray and the rasterized 3D Gaussians of Gaussian splatting are smooth functions of their parameters by construction.

A soft rasterizer for a single disk shows the idea in PyTorch: coverage is a sigmoid of the signed distance to the disk's edge, so gradients reach its center and radius from every pixel near the boundary.

import torch

# Pixel coordinates of a 64x64 image
ys, xs = torch.meshgrid(torch.arange(64.0), torch.arange(64.0), indexing="ij")


def render(center, radius):
    """Soft-rasterize a disk: pixel coverage is a sigmoid of the signed distance to its edge."""
    distance = torch.sqrt((xs - center[0]) ** 2 + (ys - center[1]) ** 2 + 1e-6)
    return torch.sigmoid(radius - distance)


# Target image: a disk at (40, 24) with radius 12
target = render(torch.tensor([40.0, 24.0]), torch.tensor(12.0))

# Recover the scene from a wrong guess by backpropagating the pixel loss
center = torch.tensor([20.0, 40.0], requires_grad=True)
radius = torch.tensor(6.0, requires_grad=True)
optimizer = torch.optim.Adam([center, radius], lr=0.5)
for _ in range(300):
    loss = ((render(center, radius) - target) ** 2).mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f"center {center.detach().round(decimals=1).tolist()}, radius {radius.item():.1f}")

It prints center [40.0, 24.0], radius 12.0: starting about 26 pixels away with half the radius, the Adam optimizer recovers the disk from pixel differences alone. With hard coverage, (distance < radius).float(), the loss would pass no gradient to either parameter.

Real-World Applications#

Silhouette losses like the one in the code compare a rendered object mask with an observed one, and for photos without annotations an instance segmentation model such as YOLO26 (yolo26n-seg.pt) supplies a mask for every object in the image.

• 3D reconstruction: NeRF (2020) and 3D Gaussian splatting (2023) fit a scene to dozens of photos with known camera poses, and Gaussian splatting then renders new views in real time, at 30 frames per second or more at 1080p.
• Human body fitting: Body-model methods fit a parametric mesh such as SMPL to an image by matching its rendered silhouette and projected joints to the person's mask and keypoints.
• Text-to-3D generation: DreamFusion (2022) optimizes a NeRF so that its renders from random viewpoints score well under a text-to-image diffusion model, creating 3D objects without any 3D training data.