OpenAI's Jalapeño: a chip built for inference, not training

Yesterday OpenAI and Broadcom unveiled Jalapeño, what OpenAI is calling its first "Intelligence Processor" — a custom accelerator built specifically for LLM inference. I run generative and agentic systems in production, so the part of this that holds my attention is not the silicon itself but the choice of target: inference, not training.

What shipped

Jalapeño is a blank-slate design for serving LLMs, not a training accelerator repurposed for the job. OpenAI designed the architecture around its own roadmap — the kernels, memory movement, networking, and serving patterns its models actually use — while Broadcom handled silicon implementation, networking (including its Tomahawk switching silicon), and the path to production, with Celestica on boards, racks, and systems.

Engineering samples are already running real workloads in the lab at production target frequency and power, including GPT-5.3-Codex-Spark. OpenAI says early testing shows performance per watt "substantially better than current state-of-the-art," with a full technical report promised in the coming months. The detail that should not get lost: the chip went from design to tape-out in nine months — claimed as the fastest ASIC cycle ever for advanced semiconductors — and OpenAI's own models helped do the design work. Deployment is planned at gigawatt scale with Microsoft and other partners starting later this year.

Owning every layer of the stack lets each one be tuned to a single objective.

Why inference is the right thing to optimize

Training is a one-time capital cost per model. Inference is the recurring one, and it scales with every user, every token, every agent step. For the interactive and agentic products I work on, the cost and latency of a single response compound across long tool-using chains, so the economics of serving — not training — decide what you can actually ship.

The number to watch is performance per watt, because at gigawatt scale power is the binding constraint, not dollars or die area. And the architectural claim is the one that matters technically: reducing data movement to push realized utilization closer to theoretical peak. On general-purpose GPUs, LLM decoding is usually memory-bandwidth bound, not compute bound — the arithmetic units sit idle waiting on weights and KV-cache to arrive, so realized utilization runs well below the spec sheet. A chip co-designed with the serving stack can shape its memory hierarchy and interconnect around exactly those access patterns. That is where the watts get recovered.

The practical win is closing the gap between a chip's rated peak and what it actually delivers.

My take

This is a vertical-integration play, and a coherent one. When the same organization owns the model, the kernels, the serving system, and now the chip, every layer can be tuned to one objective instead of negotiating across vendor boundaries. A general-purpose GPU has to be good at everything; a captive inference chip only has to be good at the workloads you actually run. That is a defensible advantage — the same logic that put custom silicon in Google's and Amazon's data centers.

I would hold two caveats. First, "performance per watt substantially better" is a comparison without a baseline or a number; engineering samples at target power are promising but not the same as units-per-rack at scale, and the technical report is still pending. Second, bespoke silicon couples you tightly to your own roadmap — the bet pays off only if the workload it was shaped around stays the workload. OpenAI hedges this by stating Jalapeño is built to run all LLMs, not just its own, which is the right hedge to make.

The point I keep returning to is the nine-month cycle, accelerated by OpenAI's own models. If frontier models can meaningfully compress chip design — the slowest, most expensive part of the hardware stack — then the flywheel OpenAI describes is more than a slide. Better models help build the infrastructure that serves better models. That recursion, more than any single performance number, is what makes this worth paying attention to.