Gemini's video understanding just stopped watching every frame

Google DeepMind announced agentic video understanding for Gemini on September 1, 2026, and it changes a default that's been quietly expensive since video inputs arrived in the Gemini API: fixed-rate frame sampling.

The problem with watching at a fixed rate

Until now, Gemini processed video the way most multimodal models do — it ingested the file at a constant frame rate, 1 FPS by default, adjustable via the API. That's a reasonable default for a 30-second clip. It's a bad one for a 90-minute lecture or a multi-hour recording, where you're paying to encode thousands of frames that are static, redundant, or irrelevant to the question being asked. Developers were left choosing between high token costs or lossy tricks — downsampling further, chunking the video, running separate passes over audio and transcript — to keep long-form video analysis affordable.

That tradeoff is a token-budget problem in disguise, and it's the same shape as one DeepMind already solved on the image side. Agentic video understanding is the video counterpart to agentic vision, which paired code execution with Gemini's native image understanding. Here, the model gets native video tools instead: it can invoke an internal tool to load a specific segment of a video file, at a chosen frame rate, through a chosen modality — visual frames, audio, or transcript — and it decides what to fetch based on the question it's actually trying to answer.

What changes: search instead of scan

The mechanism is an agentic loop. Instead of ingesting the whole media stream up front, Gemini reasons about what it needs, calls a tool to pull that slice of the video, inspects the result, and either answers or fetches more. A query about a specific event doesn't require decoding the whole recording at a fixed cadence — it requires finding the right window and looking closely there.

DeepMind's benchmark numbers, measured on standard video analysis benchmarks including LongVideoBench: up to 88% fewer tokens consumed, up to 66% lower cost, and up to 7% better accuracy, with the gains most pronounced on long-form video — the 10-minute how-to guides, 90-minute lectures, and multi-hour recordings where fixed-rate sampling was most wasteful. Gemini 3.7 Flash with agentic understanding lands at what DeepMind describes as the accuracy-to-cost pareto frontier among tested models for video understanding — meaning no other tested configuration beats it on both axes at once.

That combination — cheaper and more accurate — is worth pausing on, because it's not the usual efficiency tradeoff. Normally you cut token spend by discarding information, which costs you accuracy at the margin. Here the model is discarding fixed-rate redundancy rather than content: it's still watching everything relevant, just not everything indiscriminately. The accuracy gain comes from the same mechanism as the cost gain, which is why both move in the same direction instead of trading off against each other.

Where this actually matters

DeepMind calls out four use cases, and they map cleanly onto the search-versus-scan distinction:

• Sub-second moment retrieval — catching split-second state changes or exact cut boundaries that fixed 1 FPS sampling simply misses between frames, useful for automated video editing.
• Long-form needle-in-a-haystack search — answering a specific question buried in a multi-hour video without paying to encode the whole thing.
• Anomaly detection — resampling a suspicious window at higher FPS once something looks off, rather than running the entire video at high frame rate just in case.
• Counting actions and objects — tracking repeated movements or distinct objects over time, where naive fixed-rate sampling tends to double-count or miss instances between sampled frames.

Each of these is a case where the right answer depends on non-uniform attention — spend more compute where the signal is, less where it isn't. That's a decision fixed-FPS sampling structurally can't make, because it commits to a sampling rate before it knows what the query needs.

Getting started

Agentic video understanding is live now for video uploads and YouTube videos via the Gemini API, in both Google AI Studio and the Gemini Enterprise Agent Platform, across Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. There's no separate feature fee — it uses standard Gemini API token pricing. You enable it by setting processing to "agentic" on a video input:

from google import genai

client = genai.Client()
interaction = client.interactions.create(
    model="gemini-3.7-flash",
    input=[
        {"type": "video", "uri": "https://youtu.be/7Z5Vy9JBANs", "processing": "agentic"},
        {"type": "text", "text": "What are the 3 most important announcements in this keynote?"},
    ],
)
print(interaction.output_text)

Broader rollout is coming: agentic video understanding will reach the Gemini app for Flash and Flash-Lite models, and it's slated to power YouTube's "Ask YouTube" feature on the video watch page in the coming months.

The takeaway

If you're running video analysis in production — content moderation, media search, QA over recorded meetings, anything touching long-form footage — fixed-FPS sampling was a tax you paid on every request regardless of whether the video needed dense attention or sparse attention. Agentic video understanding lets the model decide that per query. For anyone building on Gemini's video API today, switching processing to "agentic" is a one-line change worth testing against your own long-form workloads before you accept the current token bill as the cost of doing business.