Neural Processing Unit (NPU)

A neural processing unit (NPU) is a processor designed to run neural networks, built around large arrays of multiply-accumulate units fed from on-chip memory. Most NPUs are blocks inside a system-on-chip next to the CPU and GPU, where they run inference for camera, speech, and vision features at a fraction of the power the other two would draw. Huawei's Kirin 970 and Apple's A11 Bionic, whose Neural Engine was rated at 600 billion operations per second, put NPUs in phones in 2017, and Qualcomm's Hexagon NPU, Intel's NPU in Core Ultra laptop processors from December 2023, and AMD's Ryzen AI chips followed. Microsoft's Copilot+ PCs require an NPU that delivers more than 40 trillion operations per second (TOPS). The name also covers data-center chips: Huawei calls its Ascend data-center accelerators NPUs and uses them to train large models.

How NPUs Differ from CPUs and GPUs#

Deep learning inference is dominated by matrix multiplications and convolutions over predictable data, so an NPU drops most of the control logic and caching a CPU needs for general code and the flexibility a GPU keeps for graphics. It spends that silicon on fixed-function compute arrays and keeps weights and activations in local memory, because moving data costs more energy than computing on it. NPUs also compute in low precision, mainly INT8 and FP16, which is why a model usually needs quantization before it runs on one, and why NPU throughput is quoted in INT8 TOPS. Google's Coral Edge TPU, for example, performs 4 TOPS at 2 TOPS per watt.

• CPU: Runs any code, handles operations an NPU does not support, and is the slowest of the three at large matrix math.
• GPU: Programmable parallel processor that trains and runs models in floating point at high throughput, at higher power than an NPU.
• TPU: Google's family of matrix accelerators. Cloud TPUs train and serve models in data centers, while the Edge TPU is an NPU in all but name.
• Neuromorphic chips: Process spikes from spiking neural networks and suit event-driven sensors, a different computing model from an NPU's dense matrix math.

Running a Model on an NPU#

An NPU runs a compiled program rather than a PyTorch or ONNX graph. A vendor compiler converts the network, ahead of time or when the runtime loads it, often quantizes it to INT8 using calibration images, usually fixes the input shape, and maps each layer onto the hardware. Layers the NPU does not support are handed back to the CPU or GPU, and each handoff copies tensors between processors:

Every fallback adds transfers and leaves one processor waiting on the other, so architectures for edge devices favor standard operators such as convolutions, ReLU-family activations, and pooling. Runtimes such as ONNX Runtime, Apple's Core ML, Google's LiteRT, and Intel's OpenVINO handle this partitioning and choose the NPU when one is present.

Where NPUs Are Used#

Smartphones use their NPUs for computational photography, on-device speech recognition, facial recognition, and live translation, which keeps images and audio on the device for data privacy. On AI PCs, Windows Studio Effects run background blur, eye contact correction, and automatic framing for video calls on the NPU, leaving the CPU and GPU free. In edge AI, NPUs from Rockchip, Hailo, and DEEPX run object detection inside security cameras, drones, and robots without a network connection.

Ultralytics YOLO26 exports to the compiled formats these NPUs need through export mode, including OpenVINO for Intel NPUs, Core ML for the Apple Neural Engine, QNN for Qualcomm's Hexagon NPU, RKNN for Rockchip, and the .om format for Huawei Ascend. It also trains on Ascend NPUs, selected with device="npu:0" or device="npu:0,1" for multi-NPU training.