Large Action Models (LAM)

A large action model (LAM) is an AI model trained to turn a request into a sequence of actions that software can execute, such as API calls, clicks, and keystrokes, instead of a text answer. Most LAMs start as a large language model (LLM) and are fine-tuned on recorded task trajectories until they reliably choose the next action, fill in its arguments, and recover when a step fails, which makes them the decision-making core of an AI agent. The startup Rabbit popularized the term in January 2024, when CEO Jesse Lyu unveiled the R1 handheld and presented its LAM as a model that learned to operate apps by watching people use them. Research groups now use the name for models trained and benchmarked on taking actions, most prominently Salesforce's xLAM family: five models from 1 billion parameters up to an 8x22B mixture of experts (MoE), which its September 2024 paper reported in first place on the Berkeley Function Calling Leaderboard, ahead of GPT-4 and Claude 3.

How Large Action Models Are Trained#

Microsoft's paper Large Action Models: From Inception to Implementation, published in December 2024, documents the whole process for a model that operates Microsoft Word. Starting from Mistral-7B, the team trained it in four phases: task-plan pretraining on how requests break into steps, learning from expert trajectories recorded with GPT-4o, self-boosting exploration in which the model attempted tasks the expert had failed and kept its own successful runs as new training data, and reinforcement learning against a learned reward model. Running inside UFO, Microsoft's Windows GUI agent, the finished model completed 71.0% of live Word tasks in an average of 30.42 seconds. GPT-4o, given the same text description of the window's controls, completed 63.0% and took 86.42 seconds, and it succeeded more often only when screenshots were added, at 96.48 seconds per task.

At run time, a LAM repeats one cycle until the task is done: read the request and the current state, choose an action, let the harness execute it, and observe the result.

Actions Through APIs and Interfaces#

When an application offers an API, the model emits function calls: a named operation with arguments that match a JSON schema, which its agent harness validates and runs. When software offers only a graphical interface, the model works like a computer use agent, picking a control from the accessibility tree or locating it in a screenshot through grounding, then clicking and typing. API actions are faster and easier to verify, so agents prefer them wherever they exist and fall back to the screen for legacy software. Vision becomes one more API action: a YOLO26 model deployed to an Ultralytics Platform dedicated endpoint answers a single POST request carrying an image file or URL with its object detection results as JSON, the same kind of structured output the model reads back from any other tool.

LAMs vs. Related Terms#

• LLM: A model trained to generate text. A LAM is usually an LLM given further training so that its output is an executable action, and the line blurs as general models gain function calling and computer use.
• AI agent: The complete system of model, harness, tools, memory, and permissions, often built as an agentic workflow. The LAM is the model inside it that decides each action.
• Agentic RAG: Uses tool calls to search and gather information before answering a question. Actions taken by a LAM change the state of a system, such as sending an email, editing a document, or filing a form.
• Vision-language-action (VLA) models: The robotics counterpart in embodied AI, mapping camera images and instructions to motor commands, such as Gemini Robotics. Google DeepMind's RT-2, released in July 2023, showed the approach by fine-tuning a vision-language model to output robot actions as text tokens.