Gemini Robotics ER 2 can track task progress from continuous video feeds using two capabilities:
- Moment finding: identifies the precise timestamp where a key event occurs.
- Progress classification: assigns each video to one of five completion brackets (0–20%, 20–40%, 40–60%, 60–80%, 80–100%).
Moment finding
Moment finding identifies the exact video frame where a critical event occurs — for example, when a cup is full or a knot is tied. Robots use this to verify success, sequence steps, and trigger corrections.
The following example prompt asks the model to identify the completion moment for a given task in a video:
from google import genai
client = genai.Client()
uploaded_file = client.files.upload(file="task_video.mp4")
prompt = """
At what timestamp (in seconds) does the task reach successful completion?
Return a JSON object: {"completion_time_seconds": <float>}.
If the task is not completed, return {"completion_time_seconds": null}.
"""
interaction = client.interactions.create(
model="gemini-robotics-er-2-preview",
input=[
{
"type": "video",
"uri": uploaded_file.uri,