Teaching audio-visual models to look and listen on demand

Most audio-visual models still work the way a nervous student crams for an exam: dump the entire lecture into context, then hope the answer surfaces somewhere in the pile. A new paper, OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning, argues that's the wrong default for long audio-visual sequences, and builds an agent that decides, turn by turn, whether it needs to look, listen, or both — and over which window of time — before it commits to an answer.

The framing matters because of where the cost actually lives. A full video-plus-audio pass is expensive twice over: once in the tokens needed to represent the raw signal, and once in the model's tendency to anchor on whichever modality is easiest to pattern-match, regardless of whether it's the modality that actually carries the evidence. OmniSeek attacks both problems by making evidence acquisition part of the reasoning loop rather than a preprocessing step.

Tool use as part of the reasoning, not before it

The core idea is architecturally simple: give an Omni-LLM a tool call for "look at this time window" and another for "listen to this time window," and let the model invoke them mid-chain-of-thought. Instead of a single forward pass over a fixed-length audio-visual sequence, OmniSeek runs an iterative multi-turn protocol — reason, decide what evidence is missing, call a tool, splice the retrieved raw segment back into the context, and continue reasoning with that new evidence in hand.

This is the same shape as text-based retrieval agents that learned to call a search tool instead of hoping the answer was in their training data, just moved into the audio-visual domain, where the thing being retrieved is a raw clip rather than a document chunk. The payoff is that the model's attention budget scales with the question's actual evidence needs rather than with the length of the recording.

Cold-starting a behavior that doesn't exist in pretraining data

No foundation model is pretrained to call a "listen to seconds 40–55" tool, so the authors had to manufacture the behavior before they could reinforce it. They built a data engine that synthesizes OmniTraj-170K, a corpus of 170,000 multi-hop chain-of-thought trajectories with audio and visual evidence interleaved into the reasoning chain. Supervised fine-tuning on this corpus is the cold start: it teaches the model the mechanics of multi-turn tool calling — when a turn ends in a tool call, what the retrieved segment looks like once it's spliced back in, and how to keep reasoning across turns — before any reward signal enters the picture.

That ordering is the practical lesson here for anyone building tool-using agents: reinforcement learning is very good at sharpening a behavior and very bad at inventing one from a near-zero base rate. If the base model almost never calls the tool correctly, the reward signal has almost nothing to grab onto. SFT on synthetic trajectories exists specifically to get the policy into a region where RL has traction.

The shortcut RL has to be stopped from taking

The RL stage is where the paper's more interesting design choice shows up. After cold start, the policy is optimized with a two-stage reinforcement learning process using verifiable rewards — standard practice for grounding a policy in checkable correctness rather than reward-model preference.

But verifiable correctness alone is a trap for any audio-visual model: plenty of questions in these datasets are answerable from vision alone, or from audio alone, even when framed as audio-visual tasks. A policy optimizing purely for correct answers will happily learn to ignore whichever modality is harder to use, because the easy modality already gets the reward. That's a shortcut, not reasoning, and it would quietly defeat the entire point of giving the model two different tools to call.

The paper's fix is an Audio-Visual Necessity objective: it explicitly rewards trajectories that are correct and whose correctness actually depends on evidence from both modalities, rather than rewarding correctness alone. That reshapes the incentive away from "get the answer" toward "get the answer by using the evidence you were actually supposed to need." It's a modality-level analogue of reward shaping used elsewhere to stop RL policies from gaming a proxy metric instead of solving the intended task.

Why this is worth tracking

The paper reports consistent improvements across a range of audio-visual reasoning benchmarks and demonstrates that the resulting policy learns adaptive cross-modal evidence seeking rather than a fixed retrieval pattern. I'd treat the benchmark gains as a secondary result; the primary contribution is the recipe — synthetic trajectory cold start plus a reward term that targets a specific known failure mode (modality shortcutting) rather than just correctness.

That recipe generalizes past audio-visual QA. Any agent that's been handed multiple tools and a single correctness reward is exposed to the same shortcut risk: if one tool is cheaper to call and still gets credit, the policy will quietly stop using the others, and you won't notice until the cases where the neglected tool was actually load-bearing start failing. OmniSeek's contribution is naming that failure mode precisely and building a reward term for it, in a domain — long audio-visual streams — where full-context ingestion was never going to scale anyway. Worth reading if you're building multi-tool agents of any kind, not just ones that look and listen.