Action Chunking

Action chunking is a technique in robot learning where a policy predicts a sequence, or chunk, of the next k actions from a single observation instead of one action per control step. The name borrows a concept from psychology and neuroscience, the grouping of a sequence of actions into a single unit, and entered robot learning with Action Chunking with Transformers (ACT) in 2023. ACT learned fine bimanual tasks such as slotting a battery with 80-90% success from only 10 minutes of demonstrations on the low-cost ALOHA arms. Predicting chunks is now standard in imitation learning and in the vision-language-action (VLA) models built for general-purpose robotics.

Why Predicting Chunks Helps#

In behavioral cloning, a policy learns to copy the actions in human demonstrations. Each small mistake moves the robot into a state the demonstrations never covered, where the next prediction is worse, so errors compound over a long task. A policy that commits to k actions per prediction makes k times fewer decisions per episode, which the ACT authors describe as a k-fold reduction in the task's effective horizon. In their simulated ablation, success averaged over two tasks rose from 1% when predicting one action at a time to 44% with chunks of 100.

Chunks also capture timing that a single observation cannot. Human demonstrators pause and hesitate, and a single-step policy that sees the same image before and after a pause cannot tell whether to wait or move; a chunk that includes the pause reproduces it. And when the demonstrations show two valid ways to do a task, predicting the whole sequence at once keeps the robot on one of them instead of averaging the two.

The robot runs a chunk and then asks the policy for a new one:

Replanning after the whole chunk gives the fewest model calls; replanning after part of it, as Diffusion Policy does, follows the receding-horizon idea of model predictive control and reacts faster to surprises.

Smoothing the Seams Between Chunks#

Within a chunk the robot moves open loop, and each new chunk starts from a fresh prediction, so motion can jump where one chunk hands over to the next. ACT's temporal ensembling queries the policy at every step and executes a weighted average of all the chunks' predictions for the current step, with weights exp(−m·i) that give the oldest prediction the most weight. This NumPy example compares plain chunking with temporal ensembling for a stand-in policy whose every chunk carries its own small error:

import numpy as np

rng = np.random.default_rng(0)
k, steps, m = 10, 100, 0.1  # chunk size, episode length, temporal-ensembling decay
target = np.sin(np.linspace(0, np.pi, steps + k))  # the demonstrated joint trajectory

# Stand-in policy: at step t it predicts actions for steps t..t+k-1, each chunk off by its own error
chunks = [target[t : t + k] + rng.normal(0, 0.05) for t in range(steps)]

# Plain chunking: query every k steps and execute each chunk open loop
plain = np.concatenate([chunks[t] for t in range(0, steps, k)])

# Temporal ensembling (ACT): query every step and average all predictions for step t, oldest weighted most
ensembled = []
for t in range(steps):
    preds = [chunks[s][t - s] for s in range(max(0, t - k + 1), t + 1)]
    weights = np.exp(-m * np.arange(len(preds)))
    ensembled.append(np.dot(weights, preds) / weights.sum())

for name, actions in (("plain", plain), ("ensembled", np.array(ensembled))):
    print(f"{name}: largest jump between consecutive actions = {np.abs(np.diff(actions)).max():.3f}")
print(f"demonstration: largest jump = {np.abs(np.diff(target[:steps])).max():.3f}")

Plain chunking produces a largest step-to-step jump of 0.085, about three times the demonstration's 0.029, at a seam between chunks. Ensembling cuts it to 0.044, because each executed action averages up to ten predictions made from different observations.

Larger models make the seams harder to hide, since inference can take longer than a control step. Physical Intelligence's real-time chunking generates the next chunk while the current one is still executing, freezes the actions that will run during the inference delay, and fills in the rest to match them, which kept dynamic tasks such as striking a match working with delays above 300 ms.

Action Chunking in Modern Robot Policies#

• ACT: A transformer trained as a conditional variational autoencoder that outputs chunks of 100 joint-position targets, two seconds of motion at ALOHA's 50 Hz control rate.
• Diffusion policies: These generate each action sequence by iteratively denoising it, which represents demonstrations with several distinct solutions well.
• VLA models: Physical Intelligence's π0 pairs a vision-language model with an action expert trained by flow matching that outputs chunks of 50 actions, for control at up to 50 Hz. Hugging Face's LeRobot library provides open implementations of ACT, Diffusion Policy, and π0.

Most of these policies encode raw camera images with a learned backbone. A policy can instead condition on object positions from a detector: an Ultralytics YOLO26 model fine-tuned on the parts in a work cell turns each frame into a few box or keypoint coordinates that the chunked policy takes as its state in place of raw pixels.

Action Chunking vs. Related Concepts#

• Action recognition: Identifies the activity in a video. Action chunking generates the commands a robot executes.
• Model predictive control: Optimizes a sequence of actions against a dynamics model at every step. A chunked policy learns its sequences from demonstrations and needs no model.
• Large action models: AI agents that act in software interfaces through clicks and API calls. Action chunking concerns low-level motor commands, such as the 14-dimensional vector of joint positions that ACT outputs for ALOHA's two arms at every step.