NVIDIA's Nemotron team ships unified audio intelligence without breaking the text model underneath

Every team that has tried to bolt speech onto a capable text LLM has run into the same wall: the moment you fine-tune on audio, the model gets worse at the things that made it worth deploying in the first place — instruction following, multi-step reasoning, tool use. NVIDIA's Nemotron team has a new paper, Unified Audio Intelligence Without Regressing on Text Intelligence, that tackles this directly with a model called Audex, and it's worth a close read if you build anything voice-facing.

The tax nobody wants to pay

I've watched this failure mode up close. You take a strong text model, add an audio encoder and some speech data, fine-tune, and ship. Voice quality looks great in the demo. Then someone runs the model through the same reasoning or agentic eval suite it passed before the audio fine-tune, and the numbers are down — sometimes a little, sometimes a lot. Multi-turn tool calling gets flaky. Long-context recall gets shakier. Nobody planned for this; it's an emergent cost of multimodal fine-tuning, and it's the reason a lot of "voice mode" products feel like a worse version of the text model wearing a headset.

Audex — full name Nemotron-Labs-Audex-30B-A3B — is built on top of Nemotron-Cascade-2-30B-A3B, NVIDIA's text-only MoE backbone, and the headline claim in the abstract is specific: it delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, "while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression." That second half of the sentence is the actual contribution.

What's actually new here

To be clear about scope: the architecture itself isn't new. A single Transformer decoder where audio is projected into the text embedding space and audio output is generated as quantized tokens alongside text tokens is now a well-established pattern — Moshi, AnyGPT, Qwen2.5-Omni, and GLM-4-Voice all work from some version of this idea. Treating speech as "just another sequence of tokens" so you can reuse standard LLM training and inference infrastructure has become the default way to build unified audio-text models.

So Audex isn't proposing a new paradigm. It's demonstrating that at this point the hard problem has moved from architecture to training recipe, data curation, and scale — and that closing the regression gap is an engineering problem you can actually solve, not an inherent cost of unification.

One decoder, one vocabulary

The mechanics: audio inputs go through an encoder and get projected into the same embedding space the text tokens live in. From there, the model doesn't distinguish streams — text tokens and quantized audio tokens are "treated uniformly during generation," per the paper. That uniformity is what gives you "seamless multimodal generation" (their phrase) and compatibility with the KV-cache, sampling, and serving stack you already have for a text-only LLM. You're not maintaining a separate audio head or a bolted-on TTS pipeline; the decoder just emits whichever kind of token comes next.

The training order is the paper

Here's where I think the real signal is. The paper describes curating 157.4B audio tokens and 320.5B text tokens, then running multi-stage supervised training across that mixed corpus — the step where regression usually gets baked in, because gradient updates from audio data quietly overwrite what the text backbone learned. But Audex doesn't stop there. After the multimodal SFT stages, the recipe adds a text-only Cascade RL stage and multi-domain on-policy distillation.

Read as a sequence, that's a deliberate repair pass: mix in the audio modality, then spend a dedicated stage doing reinforcement learning on text only to re-anchor the reasoning and alignment behavior the backbone had before, then distill across domains on-policy to pull the unified model's outputs back toward what the strong text-only teacher would produce. It's less "train once and hope the mixture ratio is right" and more "train, measure the damage, and spend a stage explicitly undoing it." If the 30B-A3B naming follows the convention NVIDIA and others have used elsewhere (30B total parameters, roughly 3B active per token via MoE), that sparsity is also probably doing some of the work — a sparse backbone has more room to keep dedicated reasoning capacity untouched by audio-specific routing.

My take

For anyone building voice products on top of an LLM, this is the more useful kind of paper — not because the architecture is novel, but because it's an existence proof that you don't have to trade reasoning quality for speech capability if you're willing to pay for a training pipeline with an explicit recovery stage. Most teams don't budget for that; they do one SFT pass on mixed data and call it done, then wonder why the model's agentic behavior got worse. The lesson generalizes past audio: any time you fine-tune a capable model on a new modality or narrow task, plan for a stage that explicitly re-anchors the general capability you started with, rather than assuming the base model's skills survive by default.

Two caveats worth naming. First, the abstract doesn't give benchmark numbers, so "state-of-the-art" and "marginal or no regression" are the authors' framing until the full paper and evals are out — worth verifying against the released checkpoints rather than taking on faith. Second, NVIDIA says it's releasing model checkpoints for open research, which matters more than the paper text alone: a released model that holds up under independent regression testing is a much stronger claim than a benchmark table in a preprint. I'd want to run Audex against a fixed reasoning and tool-use suite before and after enabling audio inputs, the same way I'd stress-test any multimodal fine-tune before trusting it in production.