OpenAI's Jalapeño chip posts its first inference numbers

OpenAI just published the first performance results for Jalapeño, the custom inference chip it has been building with Broadcom. This is the first time the company has put numbers next to a claim it made back in June, when it revealed Jalapeño as the first product of its 10-gigawatt accelerator partnership with Broadcom: that a chip designed specifically for serving its own models could beat general-purpose GPUs at the one job that now dominates OpenAI's compute bill — inference, not training.

What actually shipped

OpenAI ran three open and widely-used models — GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 — on working Jalapeño silicon, benchmarked with SemiAnalysis's InferenceX suite against standard commercial setups built on Nvidia Blackwell. The reported gains:

• 1.7x to 3.6x lower end-to-end latency
• 1.5x to 1.9x more total work per watt at peak capacity
• 2.1x to 4.1x better performance on interactive, agentic workloads

The spread in each range reflects that these are three different models with different memory and compute profiles, not one clean number — which is normal for cross-model chip benchmarks and worth keeping in mind before treating the top end as the headline.

Why throughput and latency together is the actual point

The number that matters most here isn't any single multiplier — it's that OpenAI claims one architecture delivering both higher throughput and lower latency at once. On general-purpose accelerators, those two usually trade off: batch requests together to raise throughput and you add queueing delay; keep latency low and you leave compute idle between requests. Jalapeño's design attacks this by targeting the prefill and cross-chip communication phases specifically, since those are what get expensive when the model has to move a large KV cache — the running memory of a conversation or agent session — around while it works. Keeping that state local rather than shuttling it across the network is the mechanism the efficiency numbers are attributable to.

That framing matters more than it might sound. Agentic workloads — a model calling tools, waiting on results, and re-prompting itself in a loop — are latency-sensitive in a way that classic chat completions aren't, and they are precisely where OpenAI's own compute usage is shifting. Richard Ho, OpenAI's hardware lead, put it plainly: Jalapeño is meant to "serve more AI work per unit of power, while also returning responses more quickly." A chip that's fast at only one of those jobs doesn't help an agent pipeline; you need both, on the same silicon.

The context: everyone with enough scale is doing this now

OpenAI is not blazing a new trail here — it's catching up to a pattern set by every hyperscaler with enough inference volume to justify it. Google has TPUs, Amazon has Trainium and Inferentia, Microsoft has Maia. The logic is identical in each case: at sufficient scale, a chip co-designed with your own model architecture and workload mix beats a general-purpose GPU on cost-per-token, even after absorbing the non-recurring engineering cost of building it. OpenAI's version of that logic runs through Broadcom, which is fabricating the racks under the 10-gigawatt deal the two companies announced in October 2025.

What's notable is the timeline discipline: partnership announced, chip revealed roughly eight months later, first independently-benchmarkable results roughly two months after that. OpenAI says limited-volume deployment into its own infrastructure starts by the end of 2026, with broader rollout in 2027, and Nvidia and other suppliers stay in the mix throughout — this is diversification, not a GPU exit.

My take

The efficiency claims are plausible and the design choice — optimize for the prefill/communication bottleneck rather than raw FLOPs — is the correct one for where inference workloads are actually headed. But first-party benchmarks against your own definition of a "standard commercial setup" are a floor, not a ceiling, on what independent load-testing will show once the chip is running production traffic outside OpenAI's own facilities. The more durable signal is structural: OpenAI now has a credible answer to the question every large model lab eventually has to answer, which is whether it can keep growing inference volume without Nvidia's pricing and supply schedule setting the pace. Jalapeño doesn't replace that dependency in 2026. It puts a bound on how much leverage it has left.