Embodied AI

Embodied AI is the field of artificial intelligence that builds agents with a body, either a physical robot or an avatar in a 3D simulator, that perceive their surroundings through sensors, act on them through motors or simulated controls, and learn from the results of their own actions. Because every action changes what the agent sees next, an embodied agent gathers its own experience instead of learning only from a fixed dataset of images or text. Typical embodied tasks include navigating an unfamiliar home to find an object, exploring a room to answer a question about it, following spoken directions, and picking up and rearranging objects, and the field supplies the learning methods behind modern robotics.

Origins of Embodied AI#

Embodied AI grew out of a reaction against purely symbolic AI. In the late 1980s, Rodney Brooks at MIT built insect-like walking robots controlled by layers of simple sensor-to-motor behaviors, and in his 1991 paper "Intelligence Without Representation" he argued that intelligence should be built up from a machine's direct interaction with the world. In 1988, Hans Moravec observed that computers could already match adults at checkers and on intelligence tests but could not match the perception and mobility of a one-year-old. That gap, now called Moravec's paradox, remains the central difficulty of the field.

The modern field took shape between 2017 and 2019, when fast 3D simulators made it practical to train agents on billions of steps of experience. AI2-THOR, from the Allen Institute for AI, offers interactive houses in which agents open drawers and move objects, and Meta's Habitat loads 3D scans of real homes and renders more than 10,000 frames per second on a single GPU. Standard benchmark tasks include point-goal and object-goal navigation, embodied question answering, and vision-and-language navigation. In 2019, Erik Wijmans and colleagues trained a Habitat agent with deep reinforcement learning for 2.5 billion steps, the equivalent of 80 years of human experience, and it reached goals in homes it had never seen almost perfectly, using only an RGB-D camera and a GPS and compass sensor.

The Perception-Action Loop#

An embodied agent runs a closed loop: it reads its sensors, builds a picture of the scene, chooses an action, executes it, and observes what changed.

Perception turns raw sensor data into the quantities a decision needs. Object detection and instance segmentation find the things an agent can interact with, depth estimation and sensor fusion place them in 3D, pose estimation follows people and their movements, and visual SLAM builds a map while locating the agent within it. The Ultralytics ROS quickstart runs Ultralytics YOLO26 inside a ROS1 or ROS2 node that subscribes to a robot's camera topics; in its depth example, a segmentation mask selects an object's pixels in the aligned depth image, and the node publishes the object's average distance from the camera.

Decisions come from classical planners, model predictive control, or learned policies that map observations directly to motor commands. World models, which predict how a scene will change in response to an action, let an agent test a plan in imagination before it moves.

How Embodied Agents Learn to Act#

Reinforcement learning in simulation lets agents learn from reward over millions of trials, and domain randomization varies textures, lighting, and physics so that policies trained in simulation transfer to real robots; OpenAI used it in 2019 to train a robot hand that solved a Rubik's Cube. Imitation learning trains policies on teleoperated demonstrations, from plain behavioral cloning to diffusion policies and action chunking, which predict short sequences of future actions at once.

Large pretrained models now supply general knowledge about objects and language. Google's RT-1 (2022) learned more than 700 tasks from 130,000 demonstrations collected with 13 robots over 17 months, and RT-2 (2023) fine-tuned a vision-language model to output robot actions as text tokens, introducing the term vision-language-action (VLA) model. The Open X-Embodiment collaboration pooled demonstrations from 22 robot types at 21 institutions to train foundation models across bodies. Physical Intelligence's π0 (October 2024) added a flow matching action expert to a 3-billion-parameter vision-language model to control robots at up to 50 Hz, and Google DeepMind's Gemini Robotics models, released in March 2025, build a VLA on Gemini 2.0.

These methods run in autonomous vehicles, warehouse mobile robots, vision-guided pick-and-place arms, agricultural robots, and drones, all of which must act on what they see in real time, often on edge AI hardware carried on board.

Embodied AI vs. Related Terms#

• Physical AI: A largely synonymous term, popularized by NVIDIA, for AI that perceives and acts in the real world through robots, vehicles, and other machines. Embodied AI also covers agents that exist only in simulation.
• Robotics: The engineering discipline of building robots, including mechanics, actuators, and classical control. Embodied AI supplies the learned perception and decision-making that let a robot handle situations its programmers did not anticipate.
• AI agents: Software agents that act through APIs, browsers, and code. They follow the same observe-decide-act loop without physics, contact forces, or continuous sensor streams.
• Spatial intelligence: The ability to understand and reason about 3D space from images and other sensors, which embodied agents need in order to move through and manipulate their surroundings.