Edge AI

Edge AI is the practice of running artificial intelligence (AI) models on or near the devices that produce the data, such as phones, cameras, vehicles, robots, and factory gateways, instead of sending that data to a remote data center. Processing a camera frame or audio clip where it is captured removes the network round trip, so decisions arrive in milliseconds. The device keeps working without a connection, sensitive inputs stay local for data privacy, and raw video never has to be streamed to the cloud, which saves bandwidth. The price is a tight budget of memory, compute, and power, plus the work of updating models across a fleet of devices.

How Edge AI Works#

Models are usually trained in the cloud on GPUs and then made small and fast enough for the target device. Model quantization to FP16 or INT8, model pruning, and knowledge distillation shrink them, and export converts them to the format an on-device inference engine runs, such as TensorRT on NVIDIA hardware, LiteRT on Android, Core ML on Apple devices, or NCNN on ARM CPUs. The device then runs inference locally and sends upstream only what matters, such as an alert, a count, or a short clip. Those uploads also feed the improvement loop: hard examples collected from the field are labeled, the model is retrained, and the new version goes back to the devices over the air.

Edge AI Hardware#

• Neural processing units (NPUs): Dedicated accelerators built into most current phone and laptop chips, such as Apple's Neural Engine and Qualcomm's Hexagon NPU, which run quantized models within a phone's power budget.
• NVIDIA Jetson: Embedded modules that combine an ARM CPU with an NVIDIA GPU for robots, drones, and smart cameras that need GPU-class throughput.
• Raspberry Pi: Low-cost single-board computers that run small models on their ARM CPU or on add-on accelerator boards.
• Microcontrollers: The smallest tier, where models of a few hundred kilobytes run on milliwatts of power.

Deploying a Model to the Edge#

Each device class rewards a different export format. On the Raspberry Pi, NCNN is the fastest of the formats Ultralytics benchmarks for Ultralytics YOLO26, and exporting once and loading the exported folder runs the same detector through that engine:

from ultralytics import YOLO

# Load the smallest YOLO26 detection model
model = YOLO("yolo26n.pt")

# Export to NCNN, an inference framework optimized for ARM CPUs
path = model.export(format="ncnn")  # creates 'yolo26n_ncnn_model/'

# Run inference with the exported model, as on the device
ncnn_model = YOLO(path)
results = ncnn_model("https://ultralytics.com/images/bus.jpg")
results[0].save(filename="bus_ncnn.jpg")

The saved image shows the detections computed by the NCNN engine. Applications on the device are often packaged in Docker containers so the same image runs across a fleet of identical boards.

Edge AI vs. Related Terms#

• Edge computing: The broader practice of processing any data near its source, from filtering sensor readings to caching content. Edge AI is edge computing whose workload is a machine learning model.
• TinyML: The smallest tier of edge AI, on microcontrollers with kilobytes of memory.
• Federated learning: Trains a shared model across many devices by sending weight updates instead of raw data. Edge AI usually refers to inference, and federated learning brings training to the same devices.

Applications of Edge AI#

• Autonomous vehicles: Cars and delivery robots detect pedestrians, vehicles, and lanes on board, because a delayed or dropped connection cannot be allowed to delay braking.
• Smart cameras: Retail, traffic, and security cameras run object detection locally and send counts or alerts instead of continuous video.
• Industrial inspection: Cameras on production lines reject defective parts within milliseconds, and vibration sensors on machines flag anomalies before they cause downtime.
• Smartphones: Phones process photos, transcribe speech, and run small language models on device, such as the roughly 3-billion-parameter model that Apple Intelligence runs on Apple devices and that apps reach through Apple's Foundation Models framework.