OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show

Back At the Hot Chips conference on Tuesday, OpenAI shared a more detailed look at Jalapeño, including the first batch of benchmark results for the new system. Tested on Semianalysis’s InferenceX benchmark, Jalapeño filed both more tokens per user and more throughput per kilowatt than the currently available state-of-the-art inference processors.

“The bottom line is that the results show a very, very significant performance advance over state of the art,” said Richard Ho, OpenAI’s head of hardware, in a press call.

“Jalapeño can serve more AI work per unit of power, while also returning responses more quickly. It’s very efficient to serve a lot of customers, but it can also be very low latency.” Notably, that comparison is against an Nvidia Blackwell system — but Ho estimated that Jalapeño would deploy at the end of 2026 “in very small volumes,” with more significant deployment coming in 2027.

For related coverage, explore our detailed analysis on Technology and Digital Systems.

Key Analysis and Detailed Timeline

First confirmed last October, Jalapeño was developed The company plans to make Jalapeño a multigenerational platform, allowing AI products, models, chips and memory all developed in concert. Because of that full-stack approach, OpenAI was able to address specific phases in the inference process that often cause friction during inference processing.

Read More: Is it legal to train AI models on copyrighted books? It’s complicated

In particular, Jalapeño is designed to minimize delays during the prefill and communication phases of processing, which OpenAI says often act as bottlenecks.

“We designed Jalapeño to minimize data movement and communication delays,” the company said in a blog post presenting the results. “This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase.” When you purchase through links in our articles, we may earn a small commission.

This doesn’t affect our editorial independence. In less than 48 hours, your chance to save up to $300 on your tickets will end!

Two years after launch, Walmart’s Flipkart is closing in on India’s quick-commerce leaders Inherent, founded Michael Polansky is training an AI model on skin that’s still alive How AI accounting startup Rillet raised $100M and became a unicorn in 48 hours Tesla’s solar roof is dead — here’s what went wrong Oura faces legal case accusing it of misleading consumers about sleep-tracking accuracy Home batteries are suddenly cheap and everywhere.

Broader Impact and Sector Outlook