GPU (Graphics Processing Unit)

A graphics processing unit (GPU) is a processor with thousands of simple cores that run the same operation on many pieces of data at once. GPUs were built to render 3D graphics, which means computing millions of pixels and vertices in parallel, and they became the main hardware of deep learning because neural network training and inference are dominated by matrix multiplications that parallelize the same way. NVIDIA's CUDA platform, released in 2007, made its GPUs programmable for general computing, and in 2012 the AlexNet network, trained for five to six days on two NVIDIA GTX 580 GPUs, won the ImageNet challenge with a top-5 error of 15.3%, against 26.2% for the runner-up. Today PyTorch, TensorFlow, and JAX run on NVIDIA GPUs through CUDA, on AMD GPUs through ROCm, and on Apple silicon through Metal, and clusters of thousands of GPUs train the largest foundation models.

How GPUs Accelerate Deep Learning#

A CPU devotes most of its chip area to caches and control logic so a few cores can run branching code quickly. A GPU spends that area on arithmetic: an NVIDIA GPU is divided into streaming multiprocessors, each running groups of 32 threads, called warps, that execute the same instruction on different data. While some warps wait for memory, others compute, so the chip stays busy as long as there is enough parallel work. Since the Volta generation in 2017, NVIDIA GPUs also include Tensor Cores, units that multiply small matrices in one step at reduced precision, now including FP8 and, since Blackwell, FP4. An NVIDIA A100 peaks at 19.5 TFLOPS in standard FP32 math and 312 TFLOPS in BF16 on its Tensor Cores, which is why most GPU training uses mixed precision. This code times the same matrix multiplication on the CPU and the GPU:

import time

import torch

gpu = torch.accelerator.current_accelerator()  # cuda, mps, or xpu device on this machine
a = torch.randn(4096, 4096)
b = torch.randn(4096, 4096)

for device in (torch.device("cpu"), gpu):
    x, y = a.to(device), b.to(device)
    x @ y  # warm-up run
    torch.accelerator.synchronize()
    start = time.perf_counter()
    for _ in range(10):
        x @ y
    torch.accelerator.synchronize()  # GPU calls return immediately, so wait for them to finish
    seconds = (time.perf_counter() - start) / 10
    print(f"{device.type}: {seconds * 1000:.1f} ms per multiply, {2 * 4096**3 / seconds / 1e12:.1f} TFLOPS")

On an Apple M5 Pro laptop it prints cpu: 80.0 ms per multiply, 1.7 TFLOPS and mps: 18.3 ms per multiply, 7.5 TFLOPS, so the laptop's integrated GPU is about 4.4 times faster at FP32. Data center GPUs widen the gap further, both through more cores and through high-bandwidth memory (HBM) that delivers several terabytes per second, enough to keep the cores supplied with weights and activations. GPU memory, often called VRAM, also sets how large a model and batch size fit on one device, and techniques such as gradient checkpointing and distributed training exist to work around that limit.

Training and Deploying Models on GPUs#

A training job keeps the GPU busy only when the rest of the pipeline keeps up with it: CPU workers must decode and augment images fast enough, batches must be large enough to fill the cores, and math should run in reduced precision on the Tensor Cores. Low GPU utilization usually points to one of these steps:

Ultralytics YOLO trains on the first CUDA GPU by default when one is available, accepts device=[0, 1] for multi-GPU training, and with batch=-1 runs AutoBatch, which measures the GPU's free memory and picks a batch size that uses about 60% of it. Teams without local hardware can pick a cloud GPU on Ultralytics Platform for each training run. For inference, a YOLO26n model exported to TensorRT processes a 640-pixel image in 1.7 ms on an NVIDIA T4 GPU, compared with 38.9 ms using ONNX on a CPU. This code trains YOLO26 on the first GPU with AutoBatch:

from ultralytics import YOLO

# Load the smallest YOLO26 detection model
model = YOLO("yolo26n.pt")

# Train on GPU 0 and let AutoBatch size the batch to the GPU's memory
results = model.train(data="coco8.yaml", epochs=3, imgsz=640, device=0, batch=-1)

GPU vs. CPU, TPU, and NPU#

• CPU: A few powerful cores for general-purpose, branching code. CPUs run data loading and preprocessing in most training pipelines and handle inference where no accelerator is available.
• GPU: Thousands of programmable cores plus Tensor Cores. GPUs run almost any model and framework, come from several vendors, and are available in every major cloud.
• TPU: Google's application-specific chip built around large matrix multiply units. TPUs trade a GPU's flexibility for efficiency on large dense matrix workloads and are available through Google Cloud.
• NPU: A low-power neural network accelerator in phones, laptops, and edge devices, used for inference rather than training.

Real-World Applications#

• Training large models: Language and vision models with billions of parameters train on clusters of thousands of data center GPUs connected by high-speed links such as NVLink and InfiniBand.
• Real-time video analytics: A single GPU can run object detection and tracking on many camera streams at once for traffic monitoring, retail analytics, or factory inspection.
• Robotics and autonomous vehicles: Embedded GPU modules such as NVIDIA Jetson run perception models on board, where sending camera data to a server would add too much delay.
• Medical imaging: GPUs process large 3D CT and MRI volumes for segmentation of organs and tumors in seconds instead of minutes.