Gaussian Splatting

Gaussian splatting, also called 3D Gaussian splatting (3DGS), is a technique for reconstructing a photorealistic 3D scene from ordinary photos and rendering new views of it in real time. It represents the scene as millions of 3D Gaussians, soft ellipsoids that each have a position, shape, opacity, and view-dependent color, and it draws an image by projecting ("splatting") them onto the screen and blending them. Introduced at SIGGRAPH 2023 in 3D Gaussian Splatting for Real-Time Radiance Field Rendering, it matched the image quality of the best neural radiance fields (NeRF) while rendering at 30 frames per second or more at 1080p, which no earlier method had achieved on large, unbounded scenes. It is a method for novel view synthesis and 3D reconstruction.

How Gaussian Splatting Works#

Each Gaussian stores 59 learnable numbers: a 3D position, a 3D scale and a 4-number rotation quaternion that together define its covariance (its shape and orientation), an opacity, and 48 spherical harmonics coefficients, 16 per color channel, that let its color change with viewing direction to capture reflections and sheen. Training starts from the sparse point cloud that Structure from Motion (SfM) produces while calibrating the cameras, with one Gaussian placed at each point. Each iteration renders one training view, compares it with the real photo using a weighted mix of L1 and D-SSIM losses, and updates every Gaussian by gradient descent through the differentiable renderer. Every 100 iterations, adaptive density control clones small Gaussians in under-reconstructed regions, splits large Gaussians that cover too much detail, and removes nearly transparent ones, growing each scene in the paper to between 1 and 5 million Gaussians.

To draw a frame, the renderer projects each 3D Gaussian into a 2D ellipse on the image plane, divides the screen into 16×16-pixel tiles, sorts the Gaussians that overlap each tile by depth with one GPU radix sort, and alpha-blends them front to back for every pixel. Because the scene is an explicit list of primitives, rendering works like a game engine's rasterizer and evaluates no neural network per pixel.

Gaussian Splatting vs. NeRF and Meshes#

A NeRF stores a scene implicitly in the weights of a neural network and renders by querying that network at many points along every camera ray. Gaussian splatting stores the scene explicitly, so rendering reduces to sorting and blending. On the same hardware in the original paper, Mip-NeRF360 averaged 48 hours of training and about 10 seconds per rendered frame, and fully trained Gaussian splats matched or slightly exceeded its quality after 35–45 minutes of training while rendering in real time. The explicit representation costs memory: at 59 numbers per Gaussian, a 3-million-Gaussian scene holds about 700 MB of 32-bit floats, and peak GPU memory while training large scenes could exceed 20 GB in the original implementation. Splats also lack a defined surface, so applications that need collision geometry or CAD-style edits still convert them to meshes with follow-up methods, and moving scenes require extensions such as 4D Gaussian splatting.

Capturing Data for Gaussian Splatting#

A capture is a set of overlapping photos or video frames taken while moving around a static scene. Anything that moves during capture, such as a passerby or a car, appears in some frames and not in others, and the optimization tries to explain it with semi-transparent artifacts known as floaters. Training pipelines therefore accept masks that exclude moving objects, and Nerfstudio's data format takes black-and-white masks in which black marks pixels to ignore. YOLO26 instance segmentation models generate these masks automatically by segmenting every person in each frame.

import cv2
from ultralytics import YOLO

# Load a YOLO26 instance segmentation model
model = YOLO("yolo26n-seg.pt")

# Segment only people (COCO class 0), a common source of floaters in a capture
results = model("https://ultralytics.com/images/bus.jpg", classes=[0], retina_masks=True)
result = results[0]

# White marks the static scene to train on, black marks people to ignore
people = result.masks.data.sum(dim=0) > 0
keep_mask = (~people).cpu().numpy().astype("uint8") * 255
cv2.imwrite("bus_mask.png", keep_mask)

The saved bus_mask.png has the photo's full resolution because retina_masks=True returns masks at the original image size, and every person in the frame is black.

Applications#

• Real estate and virtual tours: A phone or drone capture of a home, museum, or heritage site becomes a scene that visitors explore freely in a browser or VR headset, with reflections on floors and windows preserved.
• Visual effects and virtual production: Film and game teams capture real locations as splats and use them as photoreal backgrounds and set references.
• Digital twins and mapping: Geospatial platforms stream large splat reconstructions of cities, construction sites, and infrastructure alongside traditional 3D tiles.
• Driving and robotics simulation: Reconstructions of real drives let autonomous vehicle teams re-render recorded scenes from new camera positions to test perception models.

Open-source libraries such as gsplat provide CUDA rasterizers for training splats, and the Khronos Group has ratified KHR_gaussian_splatting, a glTF 2.0 extension for storing Gaussian splats, developed with contributors from Cesium, Esri, Niantic Spatial, NVIDIA, Huawei, and Autodesk.