Computer use

The Computer Use tool lets you build browser, mobile, and desktop control agents that interact with and automate tasks. Using screenshots, the model can "see" a computer screen, and "act" by generating specific UI actions like mouse clicks and keyboard inputs. Similar to function calling, you will need to implement the client-side execution environment to receive and execute the Computer Use actions.

For the list of supported models, see Model versions. The Gemini 3.x models support several advanced capabilities:

• Multi-environment support: build agents for browser, mobile, and desktop environments.
• Streamlined actions with intents: actions include an intent field that explains the model's reasoning behind each step.
• Configurable safety policies: fine-tune safety behavior with built-in policy categories and overrides.
• Prompt injection detection: opt-in screenshot scanning to detect hidden adversarial instructions.

With Computer Use, you can build agents that:

• Automate repetitive data entry or form filling on websites.
• Perform automated testing of web applications and user flows
• Conduct research across various websites (e.g., gathering product information, prices, and reviews from ecommerce sites to inform a purchase)

Here's a minimal example of initializing the client and sending a prompt to the model with the computer_use tool enabled for a browser environment:

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-3.7-flash",
    input="Search for 'Gemini API' on Google.",
    tools=[{"type": "computer_use", "environment": "browser"}]
)

print(interaction)

import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

const interaction = await ai.interactions.create({
  model: 'gemini-3.7-flash',
  input: "Search for 'Gemini API' on Google.",
  tools: [{ type: "computer_use", environment: "browser" }]
});

console.log(interaction);

How Computer Use works

To build an agent with the Computer Use model, you need to set up a continuous loop between your application and the API. Here is what your code will do at each step:

1. Send a request to the model
   • Your application sends an API request containing the Computer Use tool, your configuration settings (like the target environment), the user's prompt, and a screenshot of the current screen.
2. Receive the model response
   • The model analyzes the screen and the prompt, returning a response which includes a suggested function_call representing a UI action (such as a click, scroll, or keystroke).
   • For Gemini 3.x models, the response also includes a reasoning intent explaining why the model chose that action.
   • The response may also include a safety_decision from an internal safety system that classifies the action as regular/allowed, require_confirmation (requiring user approval), or blocked.
3. Execute the received action
   • If the action is allowed (or the user confirms it), your client-side code parses the function_call, scales the normalized coordinates to match your viewport, and executes the action in your target environment using automation tools (such as Playwright). If the action is blocked, your client should halt the execution or handle the interruption.
4. Capture the new environment state
   • After the action finishes executing, your application captures a new screenshot and sends it back to the model in a function_result to request the next step.

This process then repeats from step 2, continually soliciting the next action from the model until the task is completed or terminated.

How to implement Computer Use

Before building with the Computer Use tool you will need to set up:

• Secure execution environment: Run your agent in a sandboxed VM or container to isolate it from your host system and limit its potential impact. The reference implementation includes a ready-to-use Docker-based sandbox you can use as a starting point.
• Client-side action handler: Implement client-side logic to execute coordinates, type text, and take screenshots.

The examples below use a web browser as the execution environment and Playwright as the client-side handler.

0. Set up Playwright

First, install the required packages:

pip install google-genai playwright
playwright install chromium

Then, initialize a Playwright browser instance to use for execution: