CPU

A central processing unit (CPU) is the general-purpose processor that runs a computer's operating system and programs. It executes instructions on a small number of powerful cores, each built to finish one stream of instructions quickly, with large caches, branch prediction, and out-of-order execution that keep complex, branching code fast. In artificial intelligence (AI) systems the CPU loads and decodes data, applies data augmentation, feeds GPUs and other accelerators, and handles the steps around a model, such as resizing images before inference and non-maximum suppression after object detection. It also runs the model itself wherever an accelerator is missing or not worth the cost, from laptops and cloud functions to a Raspberry Pi, which makes CPU inference latency a key design target for edge AI models.

How CPUs Run Neural Networks#

A neural network spends most of its time multiplying matrices, and CPUs speed this up in three ways. Several cores, from 4 in a small board to more than 100 in a server chip, work on different parts of a batch or layer in parallel. Within each core, SIMD (single instruction, multiple data) units apply one instruction to many numbers at once: AVX2 and AVX-512 on x86 chips from Intel and AMD, and NEON on Arm chips such as those in phones, Apple silicon Macs, and Raspberry Pi boards, with the wider SVE extension on newer Arm server chips. Recent chips add matrix units, such as Intel's Advanced Matrix Extensions (AMX) in Xeon server processors since 2023 and Arm's Scalable Matrix Extension (SME), which multiply small tiles of BF16 or INT8 values per instruction.

CPU throughput is still far below that of a GPU, because a CPU spends its chip area on caches and control logic for fast single-thread performance rather than on thousands of arithmetic units, and it reads weights from ordinary system memory with a fraction of a GPU's memory bandwidth. Inference engines close part of the gap: ONNX Runtime, Intel's OpenVINO, and Tencent's NCNN fuse layers, choose the fastest kernels for the instruction set they find, and run INT8-quantized models that move a quarter of the data that FP32 weights do. Choosing an engine and precision for a CPU deployment usually follows the hardware:

Ultralytics designed YOLO26 for this kind of hardware, and YOLO26n runs up to 43% faster than YOLO11n in CPU ONNX inference on an Intel Xeon CPU, while OpenVINO export speeds up CPU inference by up to 3x. This code runs YOLO26 on the CPU, then exports it to OpenVINO and runs the exported model:

from ultralytics import YOLO

# Run the smallest YOLO26 detection model on the CPU, even if a GPU is present
model = YOLO("yolo26n.pt")
results = model.predict("https://ultralytics.com/images/bus.jpg", device="cpu")
print(results[0].speed)  # milliseconds for preprocess, inference, and postprocess

# Export to OpenVINO's format and run the exported model on the same CPU
path = model.export(format="openvino")
ov_model = YOLO(path)
results = ov_model.predict("https://ultralytics.com/images/bus.jpg", device="cpu")
print(results[0].speed)

Each speed dictionary splits the time for one image into preprocessing, inference, and postprocessing, so the two printouts compare PyTorch and OpenVINO inference times on the machine that runs the code.

CPU vs. GPU vs. TPU vs. NPU#

• CPU: A few powerful cores for general-purpose, branching code. Every computer has one, and it runs any model with no extra hardware, at lower throughput than the accelerators below.
• GPU: Thousands of simpler cores plus Tensor Cores for matrix math and high-bandwidth memory. GPUs are the standard hardware for training and for high-throughput inference.
• TPU: Google's application-specific chip built around large matrix multiply units, available in Google Cloud and, as the Edge TPU, in small inference modules.
• NPU: A low-power neural network accelerator inside phone, laptop, and embedded chips. Intel Core Ultra, AMD Ryzen AI, Qualcomm Snapdragon, and Apple M-series chips combine CPU cores, a GPU, and an NPU in one processor.

Real-World Applications#

• Training data pipelines: While a GPU trains, CPU worker processes read images from disk, decode them, resize and augment them, and assemble batches. Too few CPU cores leave an expensive GPU waiting for data.
• Smart cameras and edge devices: Doorbells, retail sensors, and industrial cameras run small detection models on their own processors, so video stays on the device and only alerts leave it.
• Serverless and cloud inference: Cloud functions and low-traffic endpoints often run on CPU-only instances, which start quickly and cost less than reserving a GPU.
• Local large language models: Projects such as llama.cpp run quantized GGUF models on laptop CPUs, where generation speed is limited mainly by how fast memory can deliver the weights for each token.