FreeToken: treating your desktop as one elastic memory pool for frontier MoE models

Mixture-of-experts models keep getting bigger, and the serving stack underneath them keeps assuming you have a datacenter. A new paper, FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, argues that assumption is the actual bottleneck — not GPU memory. The authors, a group spanning Berkeley systems researchers and MIT's efficient-ML lab (Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, and Ion Stoica), report running a 753B-parameter model — GLM-5.2 — on a single workstation GPU. That's a claim worth sitting with for a second, because 753B parameters at even 4-bit quantization is still hundreds of gigabytes, and no consumer or workstation card comes close to that in VRAM.

The actual constraint isn't VRAM, it's a fixed offloading plan

Offloading experts to CPU RAM when they don't fit on the GPU isn't new — llama.cpp and several MoE inference stacks already do it. The standard approach picks an offloading plan once, at load time: some experts pinned to GPU, the rest swapped in from CPU RAM or disk on demand, based on a static estimate of how much VRAM and CPU-GPU bandwidth you have. That plan is a reasonable average-case answer, and it's wrong twice over in practice.

First, agent workloads don't have a stable compute profile. A coding agent alternates between short planning steps, long generation bursts, and tool calls that pause generation entirely while waiting on external results. Each phase has a different ratio of compute to memory traffic, so the offloading plan that's optimal during a long code-generation burst is not the one you'd pick during a bursty tool-calling phase.

Second, edge hardware is not one thing. An 8GB laptop GPU, a gaming desktop with a mid-range discrete card, and a workstation with a single high-memory GPU all have wildly different ratios of GPU VRAM to CPU RAM to PCIe bandwidth. A plan tuned for one machine's balance of resources is not transferable to the next machine, because the bottleneck resource is different in each case.

FreeToken's answer is to stop treating offload placement as a one-time decision and instead treat the whole machine — GPU VRAM, CPU RAM, and the bus between them — as a single elastic pool that gets repartitioned continuously, informed by which experts are actually hot right now and what the agent is doing right now.

Why MoE structure makes this tractable at all

The reason this is even worth attempting is specific to mixture-of-experts architectures. A dense 753B model activates all 753B parameters on every token — offloading it to CPU RAM would mean streaming hundreds of gigabytes across a slow bus on every forward pass, which is a nonstarter. An MoE model only routes each token through a small subset of experts (the router picks the top-k for the layer), so most of the parameter count sits idle at any given instant. That's the opening: keep the experts that are getting hit repeatedly resident on the GPU, and let the ones that are cold — used rarely, or predictable-but-not-imminent — live in CPU RAM, streamed in only when the router actually calls for them.

The paper describes this as a full co-design across the serving stack rather than a single trick: model layout and loading, expert residency (which experts stay resident on GPU and for how long), CPU–GPU execution scheduling, agentic state reuse, and runtime memory management all have to move together. Agentic state reuse in particular stands out — reusing KV cache and other session state across a multi-turn agent loop instead of recomputing it means more of the machine's limited compute and bandwidth budget is left over for expert traffic, rather than being spent re-deriving context the model already produced a turn earlier.

What the numbers actually say

The headline results, per the abstract: a 35B model runs on a laptop with an 8GB GPU, a 284B model runs on a gaming desktop, and the 753B GLM-5.2 runs on a single workstation GPU. The system reportedly supports more than 20 MoE models and has been tested against real coding and tool-using agents, not just synthetic benchmarks — which matters, because the whole design premise is that agent workloads look different from single-shot chat completions.

What the abstract doesn't give you is throughput. "Runs" and "runs usefully" are different claims — fitting a 753B model into a mix of VRAM and system RAM says nothing about tokens per second once the working set starts thrashing across a PCIe bus that's orders of magnitude slower than HBM. For a coding agent, latency matters as much as feasibility; a technically-correct answer that arrives at one token every few seconds isn't a substitute for datacenter serving, it's a demo. I'd want to see the paper's actual throughput and latency tables — not just the model-size/hardware-tier headline — before treating this as a practical alternative to renting GPU time, especially at the 753B end of the range.

Why this is worth tracking anyway

The authorship is a signal in itself: Ion Stoica and Matei Zaharia built much of the distributed-systems lineage that underlies Ray, Spark, and Databricks; Song Han and Kurt Keutzer have spent years on quantization and efficient inference specifically aimed at constrained hardware. This isn't a weekend hack claiming a headline number — it's people with a track record in exactly this intersection of distributed systems and efficient ML, and the paper says the system will be released as open software, not just described in the abstract.

If the throughput numbers hold up on release, the practical shift is real: local-AI builders stop asking "which small model fits on my GPU" and start asking "which frontier open-weight model can my machine serve well enough to be useful." That's a meaningfully different planning question, and it's the one to keep an eye on once the code and the full paper are out.