TPU (Tensor Processing Unit)

A Tensor Processing Unit (TPU) is an application-specific integrated circuit (ASIC) that Google designs to accelerate neural network training and inference. A TPU is built around large matrix multiply units, because matrix multiplication dominates the work of deep learning. Google has run TPUs in its data centers since 2015, trains its Gemini models on them, and rents them to other companies as Cloud TPU. The eighth generation, announced in April 2026, splits into two chips: TPU 8t for training and TPU 8i for inference.

How a TPU Works#

The core of each TPU is a matrix multiply unit (MXU) arranged as a systolic array, a grid of multiply-accumulate cells that pass data directly to their neighbors. Weights are loaded into the cells once, inputs flow across the grid one step per clock cycle, and partial sums flow down the columns, so every value fetched from memory is reused by many cells. Reading memory costs far more time and energy than arithmetic, which is where the design gains its efficiency. In Cloud TPU v2 and v3, each MXU was a 128×128 array that multiplied in bfloat16 and accumulated in FP32. A short simulation shows the data flow on a 4×4 grid:

import numpy as np

rng = np.random.default_rng(0)
W = rng.integers(-5, 5, (4, 4))  # weights, preloaded one per cell of a 4x4 grid
X = rng.integers(-5, 5, (6, 4))  # six input rows stream through the grid
Y = np.zeros((6, 4), dtype=int)

# Cell (k, j) holds W[k, j]. Input X[i, k] enters row k from the left and reaches
# cell (k, j) at cycle i + k + j, where it is multiplied by the stored weight and
# added to the partial sum for Y[i, j] passing down column j
cycles = 6 + 4 + 4 - 2
for t in range(cycles):
    for k in range(4):
        for j in range(4):
            i = t - k - j
            if 0 <= i < 6:
                Y[i, j] += X[i, k] * W[k, j]

print(f"{X.size * 4} multiply-adds in {cycles} cycles, equal to X @ W: {np.array_equal(Y, X @ W)}")

It prints 96 multiply-adds in 12 cycles, equal to X @ W: True: the 16 cells produce the exact matrix product while each weight is read from memory once and reused for all six input rows. The first-generation TPU ran this pattern on a 256×256 grid of 8-bit cells, 65,536 multiply-accumulates per cycle for a peak of 92 trillion operations per second, and its simple, deterministic execution suited Google's goal of meeting a 99th-percentile response-time target for inference.

TPUs run code compiled by XLA, which takes the whole computation graph from JAX, TensorFlow, or PyTorch and maps it onto the matrix units. Chips are linked by dedicated interconnects into pods of thousands, which is how the largest foundation models are trained across many TPUs at once.

TPU vs. GPU vs. CPU#

• CPU: A few powerful cores for branching, general-purpose code, which handle data loading and preprocessing in most training pipelines.
• GPU: Thousands of programmable cores plus Tensor Cores for matrix math. GPUs run almost any model and framework and are sold by many vendors and clouds.
• TPU: Trades that flexibility for efficiency on large dense matrix operations. Workloads with static tensor shapes run best, because a shape change makes XLA recompile, and custom operations, heavy branching, or high-precision arithmetic suit TPUs poorly.
• NPU: The general name for neural network accelerators in phones, laptops, and edge chips, a category that includes the Edge TPU.

The Edge TPU#

The Edge TPU is a small inference-only TPU, the chip in Google's Coral USB Accelerator, M.2 and PCIe modules, and Dev Board. It performs 4 trillion operations per second on 2 watts and runs only 8-bit integer math, so networks are fully converted to INT8 through model quantization and then compiled with Google's Edge TPU Compiler, which places supported operations on the chip and leaves the rest on the host CPU. Ultralytics YOLO automates both steps with format="edgetpu", which enables INT8 quantization and runs the compiler on an x86-64 Linux machine, producing a model that runs on boards such as a Raspberry Pi with a Coral USB Accelerator:

from ultralytics import YOLO

# Load the smallest YOLO26 detection model
model = YOLO("yolo26n.pt")

# Quantize to INT8 and compile for the Edge TPU (x86-64 Linux only)
path = model.export(format="edgetpu")
print(f"Edge TPU model: {path}")

Google's Coral platform now centers on Coral NPU, an open-source RISC-V accelerator design announced in October 2025 for wearables, which targets 512 billion operations per second on a few milliwatts.