Neural Texture Compression

Neural texture compression (NTC) is a lossy compression technique that stores material textures as compact learned features and a small neural decoder. The decoder reconstructs texture values when a renderer needs them. Intuitively, it replaces large collections of surface-detail images with a smaller learned description, reducing storage and graphics-memory requirements while preserving fine visual detail.

Karthik Vaidyanathan, Marco Salvi, Bartlomiej Wronski, Tomas Akenine-Möller, Pontus Ebelin, and Aaron Lefohn at NVIDIA introduced the influential random-access NTC method in 2023 through Random-Access Neural Compression of Material Textures. It jointly compresses a material's textures and their mipmaps, the progressively smaller texture versions used when surfaces appear farther away.

How Neural Texture Compression Works#

A physically based rendering material uses several aligned texture maps: base color describes surface color, normal maps describe small surface-orientation changes, and roughness controls highlight sharpness. These maps share structure. A scratch, for example, can affect several channels at the same location. NTC exploits those correlations by representing the maps together in a shared latent space, with compact feature vectors distributed across spatial grids. A small neural network, usually a multilayer perceptron, decodes the features into material values.

During compression, the feature grids and decoder's model weights are optimized for the material. A reconstruction loss function measures differences from the original textures, and backpropagation supplies gradients for optimization. The original method uses the Adam optimizer and combines sampled features with positional information and the requested mip level. At rendering time, a coordinate query reconstructs the corresponding texel—a texture's equivalent of a pixel.

This hybrid structure connects NTC to implicit neural representations, which represent signals through learned coordinate-dependent functions. Its defining graphics capability is random access: the renderer reconstructs requested texture locations on demand, supporting real-time inference during shading.

Memory Savings and Rendering Trade-Offs#

NTC trades texture memory for decoder computation on the GPU. Two deployment modes make that trade-off particularly clear:

• Inference on Load: Textures are decoded when an asset loads and can be converted into conventional BCn block-compressed textures. This reduces asset storage and transfer size while retaining the resident memory footprint of the destination format.
• Inference on Sample: Shaders decode requested texels during rendering, keeping the neural representation in graphics memory and adding decoder work to texture sampling.

A third mode, inference on feedback, decodes requested texture tiles into a sparse texture cache. These choices determine where compression savings persist and where decompression work occurs.

The original inkwell demonstration used 4096 x 4096 NTC textures against 1024 x 1024 block-compressed textures: 16 times as many texels with 30% less texture memory. The comparison demonstrates preservation of higher-resolution surface detail within a smaller texture allocation.

Applications and Related Concepts#

Interactive 3D rendering is a direct application. Developers compress material assets during content preparation and integrate decoding into the renderer. With the NVIDIA RTX Neural Texture Compression SDK, ntc-cli compresses texture sets, NTC Explorer compares settings, and the sample renderer displays glTF assets. Its bundled FlightHelmet model includes precompressed materials for testing the rendering modes.

Neural rendering is the broader use of learned components in image synthesis. NTC specifically reconstructs stored material textures; super-resolution reconstructs higher-resolution imagery from lower-resolution inputs. These techniques operate at different points in a graphics pipeline.

Model quantization reduces the numerical precision of network parameters and activations. Ultralytics YOLO applies this related low-precision technique through TensorRT export, producing INT8 YOLO26 engines that reduce model memory requirements and accelerate supported inference workloads.

Reconstruction Quality and Artifacts#

NTC quality depends on the bit budget and material complexity. Higher bits per texel provide more capacity for reconstructing detail; adding channels at a fixed budget increases compression pressure. Peak signal-to-noise ratio (PSNR), measured in decibels, quantifies reconstruction fidelity, with higher values indicating lower error. Shared decoding also allows features to leak between channels, altering reconstructed material properties. In NVIDIA's implementation, an explicitly designated alpha-mask channel receives special treatment to preserve values near 0 and 1 more accurately.