Unified memory and the rise of the local AI box

A year ago, running a 120-billion-parameter model on a machine that sits next to my monitor was a party trick: technically possible, slow enough to be useless, and not something I would put inside a real workflow. That is changing, and AMD's Ryzen AI Halo is one of the clearer signals. The pitch is blunt — "local AI development and inference," 128 GB of unified memory, support for models up to 200 billion parameters, in a compact desktop that runs off a normal wall socket. The marketing language I can ignore. What I can't ignore is that the local-inference box is turning into a real product category, and the thing that makes it work is a memory decision, not a faster core.

What AMD actually shipped

Stripped of the page's adjectives, the Ryzen AI Halo developer platform is built on the Ryzen AI Max+ 395 with full ROCm support, 128 GB of LPDDR5x unified memory, a GPU rated at 60 FP16 TFLOPS, and a 50 TOPS NPU, running either Windows or Linux. AMD lists it at $3,999, with a 192 GB Max+ PRO 495 variant promised.

Those numbers describe a particular bet. The GPU is not a data-center part — 60 FP16 TFLOPS is a fraction of a single discrete accelerator. The NPU is real but modest. The memory is the headline: 128 GB shared across CPU, GPU, and NPU. AMD is not selling peak compute. It is selling capacity.

Why unified memory is the whole game

For LLM inference, the first constraint is not how fast your accelerator multiplies; it is whether the weights fit in memory the accelerator can address. A consumer discrete GPU tops out around 24–32 GB of VRAM. A 120B-parameter model quantized to 4 bits needs roughly 60–70 GB just for weights, before you account for the KV cache. On a discrete card it doesn't fit, and "doesn't fit" is a wall, not a slowdown — you fall back to streaming layers across the PCIe bus and throughput collapses.

A unified-memory architecture removes that wall by giving the GPU and NPU access to the same large pool the CPU uses. The model becomes resident, and inference proceeds without crossing a bus mid-layer.

A discrete GPU's small VRAM caps the model that fits; a unified pool trades peak bandwidth for the capacity to keep large weights resident.

The catch is in that caption. LPDDR5x offers far more capacity per dollar than the GDDR or HBM on a discrete card, but lower bandwidth — and LLM token generation is bandwidth-bound. So these boxes win on "can it run at all" and on dollars per gigabyte, while ceding raw tokens-per-second to a fed-and-watered discrete GPU. For a developer who wants to iterate against a real 70B–120B model locally, "runs at all, on my desk, with no per-token bill" is often the decisive property.

Reading AMD's benchmarks

AMD published two comparisons, and they split in a way worth reading carefully rather than as a single "we're faster" claim.

AMD's own figures, May 2026, pre-production hardware.

Against NVIDIA's DGX Spark — which also carries 128 GB of LPDDR5x — AMD reports LLM throughput gains of +4% to +14% across GPT-OSS-120B, Qwen 3.5-122B, Qwen 3.6-35B, and GLM 4.7 Flash-30B. Those are single-digit-to-teens margins, and they should be: when two machines share the same memory class on a bandwidth-bound task, the gap comes down to software and scheduling, not a different tier of hardware. AMD pairs that with a price difference ($3,999 vs $4,699), an NPU the Spark lacks, and Windows support the Linux-only Spark doesn't offer.

Against an Apple M4 Pro Mac Mini, the same page claims 3.3x to 7.3x speedups — but every one of those is an image or video generation workload (Stable Diffusion XL, Flux, Qwen Image, Hunyuan 3D). Diffusion is compute-bound, not bandwidth-bound, so a larger GPU with a mature kernel path pulls ahead by multiples. It's a real result for that workload; it is not evidence about token generation.

A few honest qualifiers, because these are vendor numbers from a single source. They were run by AMD on pre-production hardware, and the token rates were measured at a 100-token context — a short window that flatters decode throughput and sidesteps the KV-cache growth that dominates long-context serving. I'd treat the direction as credible and the exact figures as something to re-measure on my own prompts.

The software stack is the real contest

Silicon is the part everyone benchmarks; software is the part that decides whether you keep the machine. NVIDIA's durable advantage has never been raw FLOPS — it's CUDA, and the decade of libraries, kernels, and day-one model support built on it. AMD knows this, which is why the Ryzen AI Halo page leads with ROCm, "Day 0" model support, a Developer Center app, and ready-to-run playbooks rather than with clock speeds.

This is the right thing to compete on and the hardest to fake. The question for ROCm is not whether it runs a curated benchmark — AMD chose those — but whether the model I pulled this morning runs without a weekend of patching. That gap has narrowed a lot. It has not closed. Apple's MLX is a third path, narrower but unusually clean on its own hardware. The competition that matters now is between these three software stacks, not three spec sheets.

When a box on your desk beats the cloud

I run most production inference in the cloud and will keep doing so for frontier models, bursty scale, and high-throughput serving where bandwidth and batching win. A local box earns its place in a different set of cases: data that can't leave the building, iteration loops where a per-token bill quietly taxes every experiment, fine-tuning and prototyping against a real large model, and offline or edge deployment. The economics flip when utilization is high and steady — a $4,000 box amortizes fast against a metered API you hit all day.

What I'm watching

The number to watch is memory capacity, because it sets the ceiling on what runs locally at all. 128 GB puts most open-weight models in reach; the promised 192 GB part pushes that further. Bandwidth is the quieter variable that will decide whether local boxes stay good enough for development or become genuinely fast for serving. And the whole category only matters if a second credible vendor keeps NVIDIA honest on price and openness — which, looking at a $3,999 machine that runs the models I care about on Linux or Windows, is exactly what's starting to happen.