TokenRouter: what it takes to serve token-level model routing

Most model routing in production is coarse. A classifier guesses how hard a request is, and one model writes the whole answer. Research on routing within a response has moved further: a small model decodes the easy tokens, and a large model steps in only where the small one would go wrong. TokenRouter: Efficient Serving System for Token-Level LLM Routing (Fu, Liu, Wang et al., Tsinghua and CMU) is not a new routing algorithm. It is the serving system those algorithms have been missing, and it shows plainly what breaks when a multi-model technique runs on infrastructure built for one model.

Why token-level routing is attractive

The motivation comes from the authors' earlier work on R2R, which matches a 32B model's quality while sending only about 5% of tokens to it and decoding the rest with a 1.5B model. Query-level routing needs about 40% of queries on the large model for the same quality. Difficulty sits in a handful of divergent tokens, so routing at that granularity means paying for the large model only there.

The paper evaluates five algorithms: CITER, R2R, R-Stitch, Co-LLM, and a multi-model ensemble. Their signals differ, but they share one lifecycle: decode a step, decide whether to hand off, ship the request state to a peer model, and accept its tokens back.

Why existing engines struggle

SGLang and vLLM assume every request in a batch belongs to one model and advances in lockstep. Token-level routing breaks that, at three costs.

• Step desynchronization. A synchronized batch waits for the slowest model, so a 0.6B model idles while a 32B model finishes its step.
• Batch admission delay. A request handed to the large model usually arrives mid-batch and waits. Frequent, irregular handoffs fragment batches into small, inefficient slices.
• Implementation complexity. No mainstream engine exposes a per-step routing hook, so you end up editing scheduler internals coupled to continuous batching and prefix caching.

The design: request-centric code, model-centric execution

TokenRouter separates the algorithm from the runtime. Developers write three request-level functions. route runs after each decoding step and returns a destination. send packages the token suffix the peer hasn't seen. receive converts what comes back into local state, and the default (accept all tokens) covers most algorithms. The paper's R-Stitch example fits in about fifteen lines.

The runtime runs one autonomous subserver per model, each with its own scheduler and KV-cache pool, behind a single endpoint that speaks the normal single-LLM protocol. To the client it is a drop-in replacement.

Tri-loop execution with handoff-resume. Alongside the client and decoding loops, TokenRouter adds an inter-model loop for peer requests. A request routed away is marked pending: the scheduler skips it but keeps its KV cache and state. When the peer returns tokens, they are appended and the request is readmitted as if it had never left. A model switch becomes little more than a token append.

Delayed batching. Instead of launching a step whenever a peer request arrives, a subserver waits until B requests have queued, so batches get wider. B is not hand-tuned. The authors model the system as a discrete-time Markov chain over queue lengths, batch sizes, and residual step times, and pick the B that maximizes steady-state throughput. In a typical configuration the large model batches at B = 2 and the small model at B = 1.

What the numbers say

On an 8×A100 server (Qwen3-0.6B on one GPU, Qwen3-32B across two), TokenRouter reports 2.01–64.15× higher decoding throughput than the stronger baseline across 15 algorithm and workload combinations. I would not lead with the 64×: some official baselines are research scripts without continuous batching.

The ablation is more informative. At concurrency 8, official R2R runs at 134.9 tok/s. Engineering work, mostly extended CUDA graphs, brings that to 230.8. Asynchronous execution reaches 296.9, and delayed batching 372.5, so both ideas contribute measurably on top of a well-tuned base.

The detail I found most useful is in the appendix. In a baseline of one SGLang server per model plus an external dispatcher, actual small-model inference took 1.56 ms of a roughly 37 ms step, about 4%. The rest was prefix matching, cache-node locking, and radix-cache updates, because every returning token was treated as a fresh request. When one model is that cheap, bookkeeping is the workload, and handoff-resume exists to remove it.

One honest caveat from their Table 2: token-level routing does not always beat running the large model alone. At concurrency 4, TokenRouter-served R2R reaches 244.6 tok/s against 145.3 for the 32B model. Co-LLM reaches 76.0 against 134.8, and R-Stitch lands slightly below LLM-only throughput, though it cuts end-to-end latency from 361 s to 151 s. The win depends on the algorithm's routing rate, not only on the serving system.

My read

This is systems plumbing for a technique that is not mainstream yet. If you run query-level routing today, nothing here asks you to change. If your team is experimenting with small/large collaborative decoding, this paper removes the main excuse for not measuring it properly: until now, naive implementations hid whatever the algorithm was worth. The code is open and builds on SGLang.

The broader point is that serving engines encode a one-model-per-request assumption across batching, caching, and scheduling. Speculative decoding got away with it because its two models alternate in a fixed rhythm. Data-dependent handoffs don't. The fix here is structural: an autonomous loop per model, request state that survives handoffs, and batch thresholds chosen analytically.

The limits are stated clearly. The throughput model assumes geometrically distributed gaps between handoffs, co-locating models on shared GPUs needs joint KV-memory allocation, and the routers themselves are still mostly research artifacts. Still, if routing within a response is going to be judged fairly against routing whole queries, it needs a serving system like this one.