Implicit Neural Representations (INRs)

Implicit neural representations (INRs) store a signal as a continuous function learned by a neural network, usually a small multilayer perceptron (MLP), that maps coordinates to values. An image becomes a function from a pixel position (x, y) to a color, a 3D shape a function from a point (x, y, z) to its signed distance from the surface, and a scene a function from a point and viewing direction to color and density. The network's weights are the representation: there is no pixel or voxel grid, so the signal can be queried at any coordinate, its memory does not grow with output resolution, and its derivatives are available for free through automatic differentiation. INRs are also called coordinate-based networks or neural fields, and they underpin neural radiance fields and much of modern 3D reconstruction.

How Implicit Neural Representations Work#

An INR is fit to one signal at a time. Training samples coordinates, compares the network's output with the true values at those coordinates, and updates the weights with gradient descent, so fitting is overfitting on purpose: the goal is to memorize this signal. A plain MLP with ReLU activations struggles, because networks learn low frequencies much faster than high ones, an effect Nasim Rahaman and colleagues named spectral bias in 2019. Fed raw coordinates, the network produces a blurred version of the signal. Two fixes made INRs practical in 2020: mapping the coordinates through sines and cosines at many frequencies, the Fourier features of Matthew Tancik and colleagues and the positional encoding of NeRF, and replacing ReLU with a sine activation function, as in SIREN by Vincent Sitzmann and colleagues.

This PyTorch example fits the same MLP to a photo twice, once from raw coordinates and once from Fourier features:

import torch
from skimage import data, transform
from torch import nn

torch.manual_seed(0)

# The signal: a 128 x 128 grayscale photo, and the (x, y) coordinate of every pixel in [-1, 1]
image = torch.tensor(transform.resize(data.camera(), (128, 128)), dtype=torch.float32).reshape(-1, 1)
grid = torch.linspace(-1, 1, 128)
coords = torch.stack(torch.meshgrid(grid, grid, indexing="ij"), dim=-1).reshape(-1, 2)

# Fourier features: sines and cosines of the coordinates at 128 random frequencies
proj = torch.pi * coords @ (10 * torch.randn(2, 128))
inputs = {"Raw coordinates": coords, "Fourier features": torch.cat([proj.sin(), proj.cos()], dim=-1)}

for name, x in inputs.items():
    # The INR: a small MLP from coordinate to gray value, fit to this one image
    mlp = nn.Sequential(nn.Linear(x.shape[1], 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(), nn.Linear(256, 1))
    optimizer = torch.optim.Adam(mlp.parameters(), lr=1e-3)
    for _ in range(1000):
        loss = (mlp(x) - image).pow(2).mean()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f"{name}: PSNR {-10 * torch.log10(loss).item():.1f} dB")

After 1,000 steps the raw-coordinate network reaches a peak signal-to-noise ratio of 22.0 dB, a blurry image, while the Fourier-feature network reaches 46.0 dB, a near-exact copy. The frequency scale, 10 here, is tuned per signal: too low and the fit stays blurry, too high and the network memorizes the samples with noisy values between them. Meta's Segment Anything Model, which runs in the Ultralytics package, encodes the coordinates of point and box prompts with the same random Gaussian Fourier features before they reach its mask decoder.

Shapes, Scenes, and Speed#

INRs for 3D shapes arrived in 2019, when DeepSDF predicted signed distance and Occupancy Networks predicted inside-or-outside probability for any 3D point, letting a mesh be extracted at any resolution. NeRF followed in 2020, mapping position and view direction to color and density for novel view synthesis. Pure MLPs are slow to train and query, so hybrid designs store learned features in a grid and decode them with a tiny MLP. Instant-NGP by Thomas Müller and colleagues at NVIDIA uses a multiresolution hash table of features and fits a NeRF in seconds instead of hours.

INRs contrast with explicit representations such as pixel grids, voxel grids, meshes, and point clouds, which store values directly and grow with resolution, cubically for voxels. Gaussian splatting is a recent explicit alternative to NeRF that renders faster by rasterizing millions of 3D Gaussians. Because a basic INR is fit to a single signal, generalizing across many signals takes extra machinery: hypernetworks that output INR weights from a latent code, or meta-learning of an initialization that fits a new signal in a few steps.

Real-World Applications#

• Medical imaging: INRs reconstruct CT and MRI volumes from sparse or undersampled measurements, since the continuous function fills gaps between the acquired samples with a smooth, data-consistent volume.
• Super-resolution: Local implicit image functions such as LIIF predict a color at any continuous coordinate from nearby features, so one model upsamples images by arbitrary factors.
• 3D content creation: Text-to-3D and image-to-3D generators often optimize or predict neural fields, which are then converted to meshes for games and visual effects.
• Compression: COIN, by Emilien Dupont and colleagues in 2021, stores an image as the quantized weights of a small MLP and outperforms JPEG at low bit rates.