Computer Use Agents (CUAs)

Computer use agents (CUAs) are AI agents that operate a computer the way a person does: they look at screenshots of the screen, decide what to do, and act by moving the mouse, clicking, typing, and scrolling. Because they work through the graphical interface instead of an API, a CUA can use almost any desktop or web application, including legacy software that offers no integration at all. Anthropic released computer use for Claude 3.5 Sonnet as a public beta in October 2024. OpenAI followed in January 2025 with Operator, powered by a model it named Computer-Using Agent, and later folded Operator into ChatGPT agent. Google released its Gemini 2.5 Computer Use model in October 2025.

How Computer Use Agents Work#

A CUA runs a perceive-act loop. Its agent harness captures a screenshot and sends it, with the task and the history so far, to a vision-language model (VLM). The model replies with one action and its pixel coordinates, such as a left click at (512, 384) or typing a search term, and the harness performs it through the operating system's input APIs, captures a new screenshot, and repeats. Anthropic's documentation recommends a display of 1024 x 768 or 1280 x 720 for desktop tasks, and when a harness shrinks screenshots before sending them, it scales the model's coordinates back up to the real screen.

The hardest part of the loop is grounding: turning "open the invoice from March" into the exact pixel of the right row. End-to-end models predict coordinates directly from the screenshot. Other agents first parse the screen into a list of labeled elements with object detection and optical character recognition (OCR), then let a large language model (LLM) choose an element by its ID, which turns precise pointing into a multiple-choice question. Browser agents can also read the page's accessibility tree or DOM when one is available and fall back to pixels when it is not.

Screen Parsing with Object Detection#

Microsoft's OmniParser, released in 2024, is the best-known screen parser of this kind. It detects every clickable icon and text field, captions each icon with a Florence-2 model so the language model knows what it does, and reads text with OCR, and the parsed output raised GPT-4V's accuracy at clicking the right element on the ScreenSpot benchmark. Its interactable-region detector is an Ultralytics YOLOv8 model fine-tuned on screenshots of popular web pages whose clickable-element boxes came from each page's DOM. Detection-based parsing suits screens full of small, unlabeled icons, where a fast real-time detector finds every target in one pass and the VLM only has to pick among them.

Benchmarks#

OSWorld, introduced in April 2024, tests agents on 369 real tasks across Ubuntu, Windows, and macOS applications, from editing spreadsheets to changing system settings, and checks each result with a script. At release, humans completed 72.36% of the tasks and the best model only 12.24%. OpenAI's Computer-Using Agent reached 38.1% in January 2025, Claude Sonnet 4.5 reached 61.4% in September 2025, and Claude Sonnet 4.6 scored 72.5% on the cleaned-up OSWorld-Verified set in February 2026, about the human rate from the original paper. WebArena covers browser agents with 812 tasks on self-hosted shopping, forum, code-hosting, and content-management sites, and like OSWorld it scores each task by checking the final state with a script, the standard practice in agent evaluation.

CUAs vs. Related Approaches#

• Robotic process automation (RPA): Replays clicks and keystrokes recorded for one fixed layout, so a moved button breaks the script. A CUA reads the current screen and plans each step, at the cost of speed and some unpredictability.
• Function calling and tool use: Calls an application's API with structured arguments, which is faster and more reliable whenever an API exists. Computer use covers the software that has none.
• Large action models (LAMs): Models trained specifically to output actions in software interfaces. Many CUAs run on general multimodal models trained for computer use alongside everything else.

Applications and Risks#

CUAs automate data entry into legacy systems, fill out web forms from documents, run GUI tests that follow a workflow the way a user would, and gather information across sites that lack APIs. Everything on the screen is input to the model, so a web page, email, or document can carry a prompt injection that tells the agent to do something else. Providers recommend running CUAs in an agent sandbox such as a dedicated virtual machine with minimal privileges, keeping them away from logged-in accounts with sensitive data, and requiring confirmation for purchases, sent messages, and other consequential actions, and Operator handed control back to the user whenever a site asked for login details, payment details, or a CAPTCHA.