Thinking while speaking: borrowing a frontier brain mid-sentence

The trade we keep making with voice

Every voice agent I have shipped or evaluated sits on the wrong side of a single trade. You can have a model that answers in milliseconds, or you can have a model that actually knows things — one that retrieves, reasons, and calls tools — but not both in the same turn. The frontier models that handle a genuinely hard question take seconds of iterative work to get there. A conversation does not grant you seconds. People expect the next turn to begin almost before they finish their sentence, and silence past a few hundred milliseconds reads as the system being broken.

A new paper, Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents by Vidya Srinivas, Zachary Englhardt, Shwetak Patel, and Vikram Iyer, proposes a way out of that trade that I find unusually practical. Instead of choosing between fast and capable, you run both — and let the fast model speak while the capable model is still thinking.

Conversational infill

The mechanism is called conversational infill, and it rests on two models with very different jobs. A small "talker" runs on-device; the paper tests seven off-the-shelf small language models ranging from 135M to 1.7B parameters. A large "reasoner" runs wherever frontier models run, slowly. When a question arrives, the talker does not wait for the reasoner. It immediately begins a contextually grounded response — the kind of fluent, on-topic opening that does not yet commit to the hard facts. That opening hides the reasoner's latency. Meanwhile the reasoner does its slow work, and as its knowledge becomes available it is streamed into the talker, which integrates that knowledge into the utterance it is already speaking.

The talker is, in effect, filling in the blanks of its own sentence with material arriving from a smarter model in real time.

The detail that matters is where the integration happens: during inference, mid-utterance, not as a separate follow-up turn. This is not a system that says "let me think" and then comes back with the real answer a beat later. The bridge between the two models is the sentence itself. The talker has to start a thought before it knows how the thought ends, accept foreign knowledge partway through, and still land somewhere coherent and grammatical. That is a harder behavior to learn than it sounds, because the model is committing to syntax and framing before the facts have arrived.

Is the behavior learnable?

The sharper empirical question is whether a 135M model can reliably perform that weave at all. The authors built a 290,571-example synthetic dataset spanning six domains to teach exactly this behavior, and they show the task is learnable across all seven small models they tested.

That breadth is more convincing to me than any single accuracy figure. A trick that only works on one carefully tuned checkpoint is a demo. A behavior that transfers across a 135M-to-1.7B range of stock models is a technique — something you could plausibly drop onto whatever small model you already run at the edge. It tells you the capability lives in the training signal and the framing, not in one lucky set of weights.

The numbers, and where they land

On accuracy, ConvFill — the system implementation in the paper — closes the gap to within 6.3% of the corresponding frontier reasoner it draws from, while holding time-to-first-response at the millisecond scale. Those two facts are normally in opposition; the contribution is reporting them together.

The right way to read this is as a new point on the latency-capability Pareto frontier. The old options were a small real-time model that meets the latency bar but cannot match a foundation model on complex tasks, or a frontier model that is capable but too slow to converse with. ConvFill claims the corner that was supposed to be unreachable: fast, and nearly as capable.

What keeps the result honest is the live user study. Eighteen participants interacted with talker deployments running on an Apple M2 SoC — real on-device inference, not a server pretending to be a phone. Participants ranked ConvFill on par with frontier models overall, preferred it for retrieval-heavy tasks, and rated it significantly more responsive. The retrieval-heavy preference is the tell: those are exactly the cases where a small model alone would be guessing, and where streamed reasoner knowledge does the most work. People felt the difference, not just the benchmark.

Why this matters in production

Most of the work I do to make voice agents feel responsive is latency engineering around the edges — speculative prefetch, caching, cutting tokens off the critical path, accepting a weaker model when the clock is tight. Conversational infill reframes the problem. The latency of the strong model stops being something you hide with tricks and becomes something you spend, productively, while the user is already being spoken to. The talker's opening is not filler; it is the carrier wave that the reasoner's knowledge rides in on.

There are open questions I would want to test before betting a product on it. What happens when the reasoner is wrong, and the talker has already committed to a confident framing? How gracefully does the weave degrade when the reasoner is slow enough that the talker runs out of grounded things to say? How does this compose with tool calls that have side effects rather than just returning facts? The paper is a strong existence proof, not a finished playbook.

But the framing is the part that will stay with me. We have spent the last few years treating reasoning and responsiveness as a dial to be set. This work treats them as two processes that can run at once and meet inside a single sentence. The code, models, and datasets are open, which means the rest of us can find out where it holds. That is the version of a result I trust most: one you can go break yourself.