Autonomous Vehicles

Autonomous vehicles (AVs), often called self-driving vehicles, use sensors and artificial intelligence (AI) to perceive their surroundings, predict what other road users will do, plan a path, and steer, accelerate, and brake without a human driver. The term covers a spectrum, from cars that hold their lane on the highway while a person supervises to robotaxis with no one in the driver's seat. Driverless service is now commercial: Waymo, which opened fully driverless rides to the public in Phoenix in 2020, carries paying riders in at least ten US metro areas, and Baidu's Apollo Go runs driverless robotaxis in Chinese cities such as Wuhan. Consumer cars mostly offer driver assistance that requires constant human supervision, still the most common form of AI in automotive.

How Autonomous Vehicles Work#

A self-driving system repeats the same pipeline many times per second:

• Sensing: Cameras read lane markings, signs, and traffic lights; lidar measures precise 3D distances with laser pulses; radar measures range and speed and keeps working in fog and rain; GNSS, inertial sensors, and wheel odometry track the vehicle's own motion.
• Perception: Neural networks detect, classify, and track every car, cyclist, and pedestrian, find lanes and drivable space, and estimate distances.
• Prediction: The system forecasts where each road user is likely to go over the next several seconds, often as several possible trajectories with probabilities.
• Planning: A planner chooses a path and speed that make progress, obey traffic rules, and keep a safe margin from every predicted trajectory.
• Control: A low-level controller turns the plan into steering, throttle, and brake commands.

No single sensor works in every condition, so sensor fusion combines them. Perception relies on object detection, semantic segmentation of road, sidewalk, and vegetation, 3D object detection in lidar point clouds, and multi-object tracking, while camera-only systems recover distance with depth estimation and stereo vision. Path-tracking controllers often use model predictive control to follow the planned trajectory within steering and acceleration limits. Public datasets such as KITTI and Argoverse supply labeled driving data for training and benchmarking these models.

Tracking Road Users in Code#

Prediction needs each road user's motion, so perception must follow every object from frame to frame under a stable identity. Ultralytics YOLO26 in tracking mode does this, adding a persistent ID to each detected box. This example tracks pedestrians through a video of people walking:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")

# Track pedestrians (COCO class 0) across video frames, keeping one ID per person
video = "https://github.com/ultralytics/assets/releases/download/v0.0.0/decelera_portrait_min.mov"
results = model.track(source=video, classes=[0], stream=True)

ids_seen = set()
for result in results:
    if result.boxes.is_track:
        ids_seen.update(result.boxes.id.int().tolist())

print(f"Unique pedestrians tracked: {len(ids_seen)}")

It prints how many distinct people received IDs during the clip. Each ID links one person's boxes across frames, and the change in a box's position from frame to frame is the motion history that a prediction module extrapolates.

Modular and End-to-End Driving#

Traditional AV software is modular: separate, separately tested components for perception, prediction, planning, and control, connected by hand-designed interfaces such as lists of tracked objects. End-to-end systems replace most of that pipeline with one neural network that maps sensor data to a trajectory, trained by behavioral cloning on large amounts of human driving. Tesla moved its FSD (Supervised) driver-assistance software to an end-to-end network with version 12 in 2024, and the research model UniAD, which links perception, prediction, and planning in one network optimized for planning, won the CVPR 2023 best paper award. Many production stacks sit between the two, with learned modules trained jointly but still exposing interpretable outputs that engineers can test.

Simulation carries much of the validation load, because rare events cannot be collected on demand. A world model trained on fleet data, such as the Waymo World Model announced in February 2026, generates realistic camera and lidar data for scenarios the fleet has never recorded, and reinforcement learning in simulation helps planners handle situations that human driving data rarely covers.

Levels of Driving Automation#

The SAE J3016 standard, revised with ISO in 2021, defines six levels of driving automation. Levels 0 to 2 are driver support features: the human drives, even when the feet are off the pedals and the hands are off the wheel, and must supervise at all times. Levels 3 to 5 are automated driving systems that do the driving themselves, at least within their operational design domain, the roads, speeds, and conditions they are built for:

Hands-free highway systems such as GM's Super Cruise are Level 2, because the driver must keep watching the road. Mercedes-Benz Drive Pilot is a Level 3 system approved on certain highways in Germany, California, and Nevada, where the driver may look away but must take over when prompted. Waymo's robotaxis are Level 4, driverless within mapped service areas, and no Level 5 vehicle, able to drive anywhere a human could, exists today.

Real-World Applications#

• Robotaxis: Waymo, Baidu Apollo Go, and others carry passengers with no safety driver inside defined service areas, backed by remote staff who can answer the vehicle's questions.
• Trucking: Aurora began hauling customer freight with driverless trucks between Dallas and Houston in 2025, on highway routes where long, predictable driving suits automation.
• Driver assistance: Adaptive cruise control, lane centering, and automatic emergency braking use the same kinds of perception models in millions of consumer cars, often alongside driver-monitoring cameras that check attention.
• Industrial vehicles: Autonomous haul trucks in mines, yard tractors, and warehouse forklifts operate on private sites where routes and traffic are controlled.

Autonomous Vehicles vs. Related Terms#

• Advanced driver assistance systems (ADAS): Features at Levels 0 to 2 that help a human driver, who remains responsible for driving. An AV in the strict sense performs the whole driving task itself.
• Connected vehicles (V2X): Vehicles that exchange data with each other and with infrastructure, such as traffic signals. A car can be connected without being automated, and connectivity can extend an AV's view beyond its own sensors.
• Remote assistance and remote driving: SAE's 2021 revision distinguishes the two. A remote assistant gives a driverless vehicle advice, such as confirming that it may pass a stalled truck, while the vehicle keeps driving itself; a remote driver controls the vehicle directly from afar.
• Physical AI and robotics: An AV is a mobile robot specialized for public roads, the largest commercial deployment of physical AI, and its perception and planning methods are shared with warehouse robots, drones, and humanoids.