Computer-use agents extend tool-calling from structured APIs into graphical user interfaces. Instead of asking a model to call a JSON function such as search_products(query), the host gives the model a screenshot, the user goal, recent action history, and a constrained action vocabulary. The model returns UI actions such as click, type, scroll, navigate, or wait. A client-side executor performs those actions in a real browser, captures the next screenshot and URL, and sends that state back to the model for the next turn.
This pattern matters because many useful workflows still do not expose clean APIs. Internal admin panels, ecommerce checkout flows, SaaS dashboards, legacy forms, and web-based testing environments often require visual interaction. A computer-use loop lets an agent operate those surfaces without writing a brittle custom scraper for every page. The tradeoff is that the agent is now acting in an environment designed for humans, so layout drift, pop-ups, hidden state, authentication, and prompt injection become core engineering concerns rather than edge cases.
The loop is related to ReAct, function calling, and browser automation, but it is not identical to any of them. ReAct describes reasoning and action interleaving; function calling provides structured calls; Playwright or ChromeDP provides deterministic browser control. A computer-use system composes all three: the model visually interprets the page, emits structured UI actions, and the host translates those actions into deterministic browser operations with safety gates and observability.
The defining mechanism is a repeated observe-act-observe cycle. The first request includes the user task and an image of the current environment. The model returns one or more function calls representing UI actions. After the executor performs the action, it returns a function response containing the new URL, action result metadata, and a fresh screenshot. The conversation history alternates between model action proposals and environment observations until the model stops, reaches a turn limit, encounters an error, or a human blocks a risky action.
The model is not allowed to emit arbitrary code. It chooses from a predefined vocabulary such as opening a browser, navigating, clicking at coordinates, typing text, pressing keys, scrolling, waiting, going back, or dragging. Production systems should implement every allowed action they expose, and they should exclude actions they cannot safely support. A mobile or desktop environment can add custom functions, but the same principle holds: the model proposes an action, while trusted client code executes it.
The executor is the trust boundary. It owns browser sessions, credentials, filesystem access, network restrictions, screenshot capture, coordinate conversion, and audit logging. The model can suggest a click, but only the executor can decide whether that click is allowed, translate normalized coordinates into pixels, perform the Playwright operation, and report what happened. Treating this boundary explicitly keeps model reasoning separate from environment authority.
Computer use is function calling where the model’s input state is visual and environmental rather than purely textual. The output is still structured, but the action schema describes human-interface operations instead of application-domain APIs.
The host starts a controlled browser environment, usually with a fixed viewport and an isolated browser profile. A stable viewport reduces coordinate ambiguity because the model predicts positions relative to the observed screen. An isolated profile keeps cookies, local storage, downloads, and service workers from leaking between tasks. For high-risk workloads, run the browser in a disposable container or virtual machine rather than the developer’s main desktop session.
from playwright.sync_api import sync_playwright
SCREEN_WIDTH = 1440
SCREEN_HEIGHT = 900
playwright = sync_playwright().start()
browser = playwright.chromium.launch(headless=False)
context = browser.new_context(viewport={
"width": SCREEN_WIDTH,
"height": SCREEN_HEIGHT,
})
page = context.new_page()
page.goto("https://example.com")The first model call includes the user goal, the computer-use tool configuration, and the current screenshot. The tool configuration can specify the target environment and can exclude predefined functions that should not be available for the task. Some systems also include custom functions when the UI surface is not a normal browser, such as a mobile emulator or a specialized remote desktop.
from google import genai
from google.genai import types
from google.genai.types import Content, Part
client = genai.Client()
config = types.GenerateContentConfig(
tools=[
types.Tool(
computer_use=types.ComputerUse(
environment=types.Environment.ENVIRONMENT_BROWSER,
excluded_predefined_functions=["drag_and_drop"],
)
)
]
)
contents = [
Content(role="user", parts=[
Part(text="Find the pricing page and summarize the free-tier limits."),
Part.from_bytes(data=page.screenshot(type="png"), mime_type="image/png"),
])
]
response = client.models.generate_content(
model="gemini-2.5-computer-use-preview-10-2025",
contents=contents,
config=config,
)The response may include one or more function calls. The executor parses each call, validates it, performs the corresponding browser operation, then captures the new state. The environment state is not just a screenshot; the current URL, action result, error text, and safety decision are all useful evidence for the next model turn and for later audit review.
def denormalize_x(x: int, width: int) -> int:
return int(x / 1000 * width)
def denormalize_y(y: int, height: int) -> int:
return int(y / 1000 * height)
def execute_call(call, page):
name = call.name
args = call.args
if name == "click_at":
page.mouse.click(
denormalize_x(args["x"], SCREEN_WIDTH),
denormalize_y(args["y"], SCREEN_HEIGHT),
)
return {"status": "clicked", "url": page.url}
if name == "type_text_at":
x = denormalize_x(args["x"], SCREEN_WIDTH)
y = denormalize_y(args["y"], SCREEN_HEIGHT)
page.mouse.click(x, y)
page.keyboard.type(args["text"])
if args.get("press_enter"):
page.keyboard.press("Enter")
return {"status": "typed", "url": page.url}
raise ValueError(f"unsupported action: {name}")Computer-use agents are useful when a workflow crosses sites or tools that were never designed to be integrated. A support operator might need to copy information from one dashboard, fill a vendor form, verify a confirmation page, and update a ticket. A browser-control loop can observe each screen and keep moving even when the page structure differs from a fixed API response.
The safest implementation narrows the task domain. Instead of giving the agent an unrestricted browser, start it on an allowlisted host, limit navigation, cap the number of turns, and require user confirmation before submitting forms, purchasing items, sending messages, or changing account settings. The model can still handle visual variation, but the system bounds the consequences of a bad action.
Traditional Playwright tests are deterministic and should remain the default for regression suites. Computer-use agents complement them by exploring flows that are expensive to script: “try to create a project as a new user,” “find a way to export this report,” or “verify that the billing page explains overage limits.” The agent can produce a trace of screenshots and actions that engineers later convert into stable tests.
Some research tasks require interaction rather than simple fetching: selecting filters, opening modal dialogs, sorting tables, paginating results, or comparing products. A computer-use loop can drive those interactions visually while maintaining a transcript of what it saw and clicked. For production research agents, pair browser control with citation capture so the final answer is grounded in pages the agent actually visited.
Every computer-use loop needs explicit stopping conditions. Use a maximum turn count, per-turn timeout, global task deadline, and clear stop criteria. Without these controls, an agent can loop on a pop-up, repeatedly click a disabled element, or spend unbounded time trying to recover from a layout it misread.
MAX_TURNS = 8
for turn in range(MAX_TURNS):
response = client.models.generate_content(
model="gemini-2.5-computer-use-preview-10-2025",
contents=contents,
config=config,
)
candidate = response.candidates[0]
contents.append(candidate.content)
calls = [
part.function_call
for part in candidate.content.parts
if getattr(part, "function_call", None)
]
if not calls:
break
results = []
for call in calls:
if is_high_risk(call):
raise PermissionError(f"confirmation required: {call.name}")
results.append((call.name, execute_call(call, page)))
screenshot = page.screenshot(type="png")
function_responses = [
types.FunctionResponse(
name=name,
response=result,
parts=[types.FunctionResponsePart(
inline_data=types.FunctionResponseBlob(
mime_type="image/png",
data=screenshot,
)
)],
)
for name, result in results
]
contents.append(
Content(role="user", parts=[
Part(function_response=fr) for fr in function_responses
])
)
else:
raise TimeoutError("computer-use task exceeded turn limit")Place policy checks between model output and browser execution. The gate should inspect the action name, target URL, text being typed, current domain, safety decision, user role, and task policy. If a step requires confirmation, pause the loop and ask the user before executing. If a step is prohibited, return a blocked result to the model or terminate the task.
Log every model request, model response, action, URL, screenshot hash, safety decision, and executor error. This is essential for debugging because failures are often visual: a cookie banner covered the button, the page loaded slowly, the coordinates were scaled incorrectly, or a modal changed the focus target. Logs also support abuse investigations when an agent interacts with external websites.