Google's Gemini 2.5 Computer Use model runs a screenshot-to-action loop for browser agents

The loop: screenshot in, UI action out

The model is exposed through a new `computer_use` tool in the Gemini API, and Google is explicit that it is meant to run inside a loop rather than as a single call. Each turn feeds the model three things: the user request, a screenshot of the current environment, and a history of recent actions.

The model responds with a function call representing a single UI action — clicking, typing, scrolling. Client-side code executes that action, then captures a fresh screenshot and the current URL and sends them back as a function response, restarting the loop. This continues until the task finishes, an error occurs, or a safety response or the user stops it.

That design is worth noting for frontend teams: the model does not read the DOM or wire into an app's internals. It works from pixels and a URL, the same surface a human sees. The reference implementations reflect this — Google points developers to build the agent loop locally with Playwright or in a cloud VM through Browserbase.

Scoped to the browser, not the desktop

Google is precise about where the model works. It is primarily optimized for web browsers and shows promise on mobile UI control, but it is explicitly not yet optimized for desktop OS-level control.

The demos stay inside that boundary: extracting pet records from one web form and entering them into a CRM to schedule an appointment, and dragging sticky notes into categories on a web app. Both are ordinary browser tasks — filling forms, manipulating dropdowns and filters, operating behind logins — which Google frames as the concrete capability that was missing when agents could only talk to structured APIs.

Per-step safety runs outside the model

Google names three risks specific to agents that control computers: intentional misuse by users, unexpected model behavior, and prompt injections and scams in the web environment. Safety features are trained into the model, but the more interesting piece for builders is the out-of-model check.

Per-step safety service: An out-of-model, inference-time safety service that assesses each action the model proposes before it's executed.Montana Labs

On top of that, developers can use system instructions to make the agent refuse or request user confirmation before high-stakes actions. The model itself can also flag that an end-user confirmation is required — for example, before making a purchase. Google lists concrete actions it treats as high-risk: harming system integrity, compromising security, bypassing CAPTCHAs, and controlling medical devices.

UI testing is the use case Google shipped first

Among early deployments, Google highlights UI testing in its own production teams, saying it can make software development significantly faster. Versions of the model already power Project Mariner, the Firebase Testing Agent, and some agentic capabilities in AI Mode in Search.

That framing matters. An agent that drives a real browser from screenshots is a natural fit for end-to-end frontend testing — clicking through flows, submitting forms, checking that interactive elements behave — without hand-written selectors that break when the UI shifts. The Playwright reference path lines up directly with how many teams already write browser tests.

What a pixel-driven agent changes for frontend work

Because the model operates on screenshots and URLs rather than app internals, it treats any web frontend as a testable, automatable surface as-is. There is no integration layer to build against your components; the agent uses the interface a user would.

The practical implication is that the same rendered UI now serves two audiences — human users and an agent that navigates it visually. Google's own caveat still applies: the safeguards reduce risk but do not remove it, and it urges developers to thoroughly test their systems before launch. For teams evaluating this, the near-term payoff is less about autonomous end-user agents and more about turning existing web flows into fast, self-driving test runs against the real interface.