Inside OpenAI’s First Chip

Ben Lorica

Richard Ho on Custom AI Chips, Inference Economics, CUDA, and AI-Assisted Engineering.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • YouTube •  AntennaPod • Podcast Addict • Amazon •  RSS.

Richard Ho, VP of Hardware at OpenAI, explains how the team built Jalapeño, its first custom AI accelerator, and why controlling more of the hardware stack could lower inference costs and improve performance. He discusses the chip’s unusual combination of high throughput and low latency, how AI helped compress parts of the development cycle to nine months, and why increasingly capable AI tools could make expert engineering teams dramatically more productive. The conversation also covers CUDA, speculative decoding, reinforcement learning, next-generation chips, and what Jalapeño suggests about the future of AI-assisted engineering.

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Jump to transcript

Related content:

Support our work by subscribing to our newsletter📩

Transcript

Below is a polished and edited transcript.

Ben Lorica. All right, so we’re in for a treat today. We have Richard Ho, VP of Hardware at OpenAI. We’re recording this a week after they announced their new chip called Jalapeño. So, Richard, welcome to the podcast.

Richard Ho. Oh, thank you, Ben. It’s a real pleasure to be here and get a chance to chat with you.

Ben Lorica. Actually, the name Jalapeño is great because the name of the conference is Hot Chips.

Richard Ho. It is. I should have brought—I actually like that the program committee of Hot Chips wears these shirts with hot peppers on them, and I’m like, that’s so perfect. They gifted me one of those at the end of the talk, so yeah, I should have worn that today.

Ben Lorica. So, this chip is actually super impressive in a few dimensions, right? First of all, from the beginning to tape-out, I hear variations of either 16 or 19 months—or nine months, right? Which is unheard of. And for our listeners, tape-out is this kind of inside-baseball term hardware people use to say basically that the chip went to actual production and manufacturing.

Richard Ho. Manufacturing, yeah.

Ben Lorica. And then the other thing that’s impressive about it is that normally first-generation chips are actually not that great, honestly. As many of our listeners know, I’ve been around. I’ve known the Cerebras guys, for example, from the beginning, so I followed their journey. For a lot of chips, frankly, the first one is basically a freebie pass, right? And this chip seems to have not only been produced rather quickly, but actually been quite impressive and competitive, at least in the benchmarks that you folks have put it through.

So, Richard, going back to the beginning of this project, what was the mission statement? What were the four bullet points on the slide that kicked off this project?

Richard Ho. I think this project really kicked off when Sam and Greg decided that infrastructure was going to be a key differentiator and one of the key drivers of AI and intelligence to users. They wanted to look really hard at what we could do to lower the cost of inference and lower the cost of compute.

They’d been talking to a lot of different startup companies and things like that, and then they came around and started talking with me about what it was that we wanted to do. We had not decided originally that we were going to do our own chip. It was a possibility that we would look into.

So I came on board and we looked at it. What are the key things?

One, really lower the cost of inference so that we can provide more intelligence to our users faster and at a lower cost. That’s the primary, number-one bullet point, and the most important one.

The second bullet point is being able to take full control of our stack. OpenAI already has the models, it has the compiler—which is Triton, the open-source compiler—and all the software in between. But at some point, that control has to turn over to merchant silicon. We use a lot of GPUs, standard commercial GPUs. But if you had the ability to control even down to the rack and down to the chips and silicon, what could you do there? What trade-offs could you make up and down the stack that would go toward bullet point one, which is better infrastructure: cheaper, faster, and better?

And point three is that we want the ecosystem to do really well. If you are dependent on silicon, what could you do with your own AI models to make better silicon, to make silicon come faster, and overall improve the entire ecosystem—not just our own internal development, but the development of NVIDIA, AMD, Intel, Cerebras, as you were mentioning, and all the other companies? How would you do it?

Having that ability to do our own proof point of what you could do is really valuable. And that goes back to your earlier introductory comments about the nine months it took us to get from initial RTL to tape-out. That’s a proof point, and I can go into more detail about that if you’d like to hear more. That’s a really fun story, actually.

Ben Lorica. So, it sounds like the mission is really to make sure that inference is cheap, and primarily that OpenAI’s own costs get cheaper. To what extent did you have to pay attention to other open-source projects? For example, do you care that we should be able to run vLLM fast? Are those extraneous, or are those also important?

Richard Ho. It’s actually interesting. The team and I arrived, and we looked at the way OpenAI was doing LLM training and inference. We looked at it and said, “Hey, it’s actually very interesting. There are a few things that OpenAI does differently, but a lot of stuff is kind of standard LLM models. What would we need to do?”

We looked at it and said, let’s just tackle LLMs in general. We went with the knowledge that we got from the way OpenAI serves these inference models, but we didn’t say we’re going to customize to OpenAI models because they’re constantly changing. The research is constantly moving on. There are new techniques being put in, and at that point it was right at the beginning of when chains of thought were being introduced and analyzed, and those kinds of thinking models were emerging.

So we built for general LLMs.

The key thing here is that we got the chip back basically in May of this year, and we had not done any analysis of the other open-source models on it. Within a very short number of months—within two months—we got three open-source models running: GPT-OSS, which is based on internal models but is actually not the same as our internal models; DeepSeek R1; and Kimi K2.5.

They’re very different models, ranging from smaller models to really large models. I think what we’re showing here is that the hardware we produced is very general-purpose. We had not designed it for these models, and yet we were able to get them up and running really fast, and they’re very performant, as we show in the benchmarks.

So we used our knowledge of OpenAI models, but we didn’t fine-tune the hardware to OpenAI models only.

Ben Lorica. Speaking of the class of models, you mentioned LLMs, but obviously listeners who interact with ChatGPT, for example, know that ChatGPT is inherently multimodal, right? It can generate images, analyze images, edit images, and maybe even do video and things like that. To what extent did this multimodality play into your design?

Richard Ho. Again, it was not designed specifically for multimodality, but it actually does really well.

One of the things we’re finding is that in these different modes, like image generation or bidirectional voice, latency really matters. The user is expecting a response really fast, and what we’re finding is that Jalapeño actually has really good response times on those models, so you can use it in those places.

We’re expecting to be able to do that. It’s going to be a matter of getting it out and at scale, but we intend to have it serve a lot of different models, and those are definitely things we’re thinking about.

One of the things we’re considering is: What are the first applications that really care about high throughput and low latency? We’ll have a range of these. We’ll make those decisions as we go into volume in 2027, and then we’ll be able to decide. Is it going to be codecs first? Is it going to be the multimodal stuff? What is going to be served by Jalapeño?

All of those things will be under consideration next year.

Ben Lorica. And as you mentioned, the key challenge here is that every week we look at the news, there’s a new model, right? And it’s not just that the models themselves are new. Sometimes they’re actually introducing new ideas and new techniques. So, what did you decide to hardwire?

Richard Ho. The one thing that is really unique about our architecture, that is kind of hardwired in, is the locality and affinity between memory and compute. That’s the key architectural difference of this device.

What that means is that when data goes into our HBM, it has a very short path to the core that’s going to compute on it. That gives you all those benefits: performance, power, and everything else.

That’s really the only thing that’s kind of hardwired. Everything else is very programmable, so this is a very programmable accelerator.

And I want to stress this: We designed this with a blank slate. We didn’t have any legacy. This device is not being derived from some other kind of accelerator that was targeted for other models, and it wasn’t a general graphics processor. It was really designed from a blank slate to do this.

But as we were doing this, we wanted to be as programmable as possible.

Going back to one of your earlier comments about first generation, I would point out that the reason we’re getting such a performant first generation is that a great deal of the team, including myself, spent over 10 years building ML accelerators at our previous company. This is the core of our team. It’s basically the team that built many, many generations of the TPUs at our previous company.

So we’ve learned a lot about what should be done, how it should be done, what works and what doesn’t work, and how to analyze it and simulate it. We were able to bring all that knowledge here, which is why our first generation, right off the bat, was programmable, performant, and able to serve a lot of models. That was our goal here, and that’s what I think we’ve achieved.

Ben Lorica. So correct me if I’m wrong, Richard, but typically these hardware accelerators are either great at throughput or latency.

Richard Ho. Yeah.

Ben Lorica. But you guys seem to have lucked out.

Richard Ho. We’ve got both. Exactly. You understood this very well, right? That’s the point of the metrics that we presented at Hot Chips.

Most accelerators have tried to focus on one or the other. GPUs are very good at high throughput. Things like Cerebras and Groq are very good at low latency.

What we have in a single device—and I think this is really critical for our infrastructure and being able to lower the cost of the infrastructure—is a single device that can operate at both points, or at any point along that curve. We call it the Pareto curve between high throughput and low latency.

At any one of those points, we’ve shown through those metrics, on the models that we ran it on—which include very large models as well as diverse models—that it has higher throughput and lower latency than competitive systems that have been benchmarked up to now.

In a fleet, in the data center, this matters because you don’t want to fix the ratio of hardware that’s used for prefill versus the ratio of hardware that’s used for decode, because it’s going to change. And it does change.

If you have Jalapeño in there, you can basically select, for any pod or any rack, what operating point it’s going to be. Is this pod or rack going to be at the high-throughput point, or is it going to be at the low-latency point? You can rebalance the amount of hardware you have dedicated to it as your users’ needs change and as your models change.

So this hardware is much more fungible. It’s better to have this than to have regret over deployed hardware at a fixed ratio that you then can’t use because the ratio of the workload has changed over time.

That’s one of the things we believe gives a lot more flexibility to our infrastructure and our data centers, and you can lower the cost of that data center as a result.

Ben Lorica. Yeah. And as you kicked off this discussion, you alluded to cost, which means cost per kilowatt.

Richard Ho. Yeah. Right.

Ben Lorica. So energy efficiency is obviously super important. In fact, our listeners may not be aware of this, but power is more of a bottleneck. A lot of the articles that are written talk about, “Oh yeah, we’re having a hard time buying chips,” but really it’s power that’s the bottleneck, right?

Richard Ho. It’s the bottleneck, right?

Imagine a data center. It has a certain amount of power going in—a certain number of megawatts, or even gigawatts these days—and that doesn’t change. You can’t increase that once it’s built. That’s what it has.

So what you want to do is eke out the maximum amount of intelligence you can from that amount of power entering the data center. The performance-per-watt metric is a very important metric, and that’s the metric that we really want to measure all our devices on.

It’s not about performance per chip, or even performance per GPU or device. It is about how much intelligence you can get from every watt of power you put into a data center.

And if possible, we want to do more with less. If you can reduce the amount of power you need at any period of time because you’re able to serve your customers with less power, obviously that’s a good thing. It’s good for the environment, it’s good for society, but it’s also good for the bottom line—being able to serve the tokens at a lower cost point, to get a lower cost for intelligence for users.

Ben Lorica. So, you mentioned that this hardware is good at both prefill and decode.

Richard Ho. Right.

Ben Lorica. Why don’t the other hardware vendors do that too? Why do they optimize for one or the other? And, by the way, you may want to start by describing to our listeners what prefill and decode are.

Richard Ho. Okay. So prefill is the part where you’re basically making the KV cache. You have the context coming in, and you’re basically computing what the cache looks like—the key-value cache. That’s a lot of matrix multiplications, so it’s very compute-heavy.

Decode is really about pulling in the data, running it through the cache, and getting the tokens out. That is very memory-bandwidth limited. How much bandwidth you can get from your memory, whether it be HBM or SRAM, really dictates how fast that is.

To do a full inference, you need to do both of those things. You need to do one, and you need to do the other.

There’s been a lot of talk about disaggregating those: Do you use a certain type of hardware that’s very compute-heavy versus another type of hardware that’s very bandwidth-oriented? But like I said, Jalapeño does both.

When a computer architect or researcher looks at the hardware, they have this thing called a roofline plot. Basically, in the early part of the compute, you can increase the amount of compute, and it’s like a diagonal line that goes up: the more compute you give it, the more throughput you get.

At some point, the compute tops out, meaning all the hardware’s compute is being used, and at that point you’re memory-bandwidth limited. So there’s a memory-bandwidth part and there’s a compute part. Actually, I described that in reverse. There’s a memory-bandwidth part and there’s a compute part.

I think the key here is that if you’re able to reduce the amount of data movement you need for both parts of the compute—as well as decode, prefill and decode—you can actually do better.

Why hasn’t anyone else done this? I think everyone is trying to do this, to be honest. This is something that’s been known in computer architecture for a while: the less data movement you need, the better it is.

I think the advantage we had is that we started with a blank slate, so we didn’t have any legacy.

One of the other reasons people sometimes hesitate to go down that path is that they think the programming model might be challenging. It may be challenging to lay out your data and your matrices in such a way that you don’t have to move them very much. The shapes of the matrices in the different models may be different and may be hard to lay out. That was an early concern: maybe the programming model is different.

What we’ve shown is that, with AI, we can actually optimize and write these kernels for this type of memory architecture and this type of computer architecture, and it’s not that hard.

We were able to bring up all three of these models in the space of about two months, from scratch. We did not design for this. We had not done pre-work on this. We had the silicon back, and we said, “Okay, let’s do the kernels. Let’s see if we can make these things work and see if we can make them performant.”

And we used our own internal models—models that include Sol, GPT Sol, and Astra. We used the internal model that would ultimately become Astra to do not only the initial work on it, but also the real fine-tuned optimization.

It turns out that with that, these programming models become less of a challenge, and you can actually do this type of architecture. It’s not going to be months and months and months of optimization work on the software to make it performant and able to serve customers.

Richard Ho. I would highlight that in the SemiAnalysis write-up, they did a deep dive on Jalapeño. One of the things they noticed—which came as a surprise to us as well, by the way—is that we were dropping our latest benchmark results to them over a period of a number of weeks.

We thought, “Oh, we just want to give them better results. We got better results, and we want to give them to them.”

I think what they noticed was that there was a really fast increase in the performance level over that very short number of weeks. The reason we were able to do it is because of our use of AI and models to do that kind of kernel optimization and get the performance up.

Ben Lorica. So, the kernels and owning the whole stack. Most people who listen to this podcast have never touched it, but have heard of CUDA, and that’s supposed to be one of the reasons why NVIDIA has such a moat, right? Can you map everything that you’ve done around that and explain, compare, and contrast it with CUDA, for example?

Richard Ho. Sure. CUDA is a framework for people to basically be able to code up efficient software for NVIDIA GPUs.

The way it works is that you take your top-level model, which is represented in something like one of the Python-derivative languages, and it maps it down through a compiler, through CUDA, and it basically issues fine-grained instructions to the GPU in a language called PTX.

That transformation means model writers can write in a very high-level language, and then the CUDA system can transform it into a very efficient implementation that is able to run on the GPUs.

The finest-grained things that are issued are these things called kernels. Kernels are basically small subfunctions. They do a particular task, and the GPU does that task on some amount of data. You can optimize it to get really high performance on that particular task.

Ben Lorica. NVIDIA invested years and years of—

Richard Ho. Years, yeah.

Ben Lorica. —man-hours on this, right?

Richard Ho. Yes, totally. And it was one of the reasons why NVIDIA devices perform so well at ML. That stack was really good, and it still is really good.

What we’ve done here at OpenAI is, one, we wanted to use a more open-source stack. So we have a stack that’s based on the Triton compiler, which is open source.

Then what we’re building is basically the lower end of that, the equivalent of those kernel dispatches over into the hardware. That’s always been difficult to optimize because, as I said, you need to be able to overlap the operations so that you don’t wait for memory to move between where it’s needed and where it’s reserved.

Ben Lorica. So it’s usually only just a handful of people.

Richard Ho. That’s right. It’s a really fine-grained skill set, and a really valuable skill set.

I think what we’re seeing here is that with AI and with new architectures, you can do it just as efficiently, and you can do it very fast as well.

I think that’s always been a problem. When new hardware has come out in the past—like when a new NVIDIA GPU has shown up—it’s taken a little bit of time to get to where it’s really highly performant, to get close to the spec numbers.

What we’re seeing here, I think, is kind of new for the industry: you’ve got the hardware, and you’re getting very close to your spec numbers in a very short amount of time.

I think that speaks both to the architecture, the programming model, as well as the use of AI to help with that programming.

Ben Lorica. And so you were also able to run two other models, Kimi and what? What’s GLM?

Richard Ho. DeepSeek.

Ben Lorica. DeepSeek. Okay. Kimi and DeepSeek, right? Were those just random choices?

Richard Ho. We wanted to show the performance of the chip, and we wanted to do it in a way that was as neutral and objective as possible. So we chose a good end-to-end benchmark that was available to everybody, which is the InferenceX benchmark.

When we got the chip back, we looked at the InferenceX benchmark and said, “What are the best models here? What models would give us a diverse set of shapes and memory layouts and things like that, just to show that we could do it all?”

We chose those three in particular because they represented a range between a small model, which is GPT-OSS, and a very large model, which is Kimi, which has something like one trillion parameters.

By being able to do all of that, we demonstrated that it’s not focused purely on OpenAI models. You can take any model and you can make it run really well. That was really the goal.

Ben Lorica. And the timeline for getting Kimi to run fast was how long?

Richard Ho. It was two months.

Ben Lorica. What? Two months?

Richard Ho. Two months, which is very, very fast. And that’s two months with all three models. We didn’t separate out how much of that two months was on Kimi, how much was on DeepSeek, and how much was on the others.

Ben Lorica. And this is what you were alluding to, that you were using AI tools to do this, right?

Richard Ho. AI tools. That’s right.

We have very great engineers—let me give them a shout-out. We have fantastic engineers. But they were made so much more productive and so much more effective by using the models and AI tools to help them.

These three models, I think, give us a representative set. There are other benchmarks in InferenceX, but we thought this would give good representative data to demonstrate that we’re not trying to do this just for the benchmarks. We really wanted to demonstrate that this architecture is good, the target model is good, and where we’ll end up for our end customers in terms of the infrastructure.

Ben Lorica. So, have you tried the AgentX benchmark?

Richard Ho. We haven’t gotten there yet. AgentX was introduced while we were in the middle of this, so that’s why it wasn’t included in our list of things we were going to do.

We’ll probably do that at some point. Our next priority right now is to get to production with our own internal production models, so we’re going to be doing a lot of work on that. But we’ll probably get to AgentX at some point too, just so the performance is understood relative to the alternative hardware that’s available to the rest of the world.

Ben Lorica. And it seems, based on the things I’ve read, that you folks are already working on Gen Two, and maybe even already talking about Gen Three.

Richard Ho. Yeah.

Ben Lorica. So is Gen One meant to be just—it isn’t actually going to be used in production?

Richard Ho. No, it will be used. It will be used.

Our intent is to go to pretty high volume on Gen One, and it goes back to your earlier point: How many Gen Ones actually go to volume? We wanted to make a case that you can go to production on Gen One.

Our goal is to go to pretty high volume on Gen One and use it with our customers—being able to serve some very meaningful models to our customers and have an impact.

Why are we already on Gen Two and Gen Three? This is infrastructure, and this is the world of AI that we live in today, where models are constantly evolving and the cost needs to come down very rapidly for the end user. So we’re on this very aggressive path to do that.

Gen Two is already well underway. We’re actually very close to being able to lock it down in terms of the design and everything. And in order to get ahead, Gen Three is there.

Let me talk a little bit about our timing here. Part of it is also about when the technology is ready.

One of our goals is to make use of new hardware technologies as soon as they become available and as soon as they’re ready to go to volume.

With Jalapeño, we’re one of the earliest HBM4 devices, and HBM4 is an upgrade over previous HBM devices where you get more bandwidth coming through and a little bit more capacity.

Then in Gen Two, we’re looking at new technologies that we want to introduce, so the timing is based on when we feel those technologies are ready to go to volume. We think it’ll be ready relatively soon.

And in Gen Three, we’re looking way beyond there. What else is out there in the world?

It really attacks these three things: How many FLOPS do you have? How much compute do you have in your device? How much memory and memory bandwidth do you have in your device? And then how much interconnectivity do you have?

If you think about that, there actually aren’t that many knobs in the world of computer architecture.

We think that the way we’re doing it, with this very close affinity between memory and compute, is the right way to do it, and we’re going to push that direction as far as it’s going to go.

Ben Lorica. What do you think of the hype around high-bandwidth flash?

Richard Ho. It’s good. We’re obviously looking at that ourselves.

With flash, you have to worry about the endurance of the device, so you have to do that calculation very carefully. You have to put it in the right spot in your design.

It’s a good question, Ben, because I think computer architecture is no longer about just the chip. It’s about your entire system and how your entire system delivers.

High-bandwidth flash, HBF, is likely to be used somewhere in your system. I would say there’s a high probability that there’ll be a good use for it.

A lot of these new advances in memory are going to be very important to keep an eye on as we progress.

Ben Lorica. So, reinforcement learning—

Richard Ho. Yeah.

Ben Lorica. Some of reinforcement learning I can even do on CPUs, right?

Richard Ho. Yeah.

Ben Lorica. So how is Jalapeño going to play a role in RL?

Richard Ho. RL, reinforcement learning—there’s a part of it which is chain of thought, which is a lot of inference.

Those chain-of-thought rollouts are all inference. The faster you can do those chain-of-thought rollouts, the faster the model can think about them and bring it together, or it can do more and be more intelligent as a result.

So yes, I think faster inference becomes a very important part of reinforcement learning, and it already is. That’s clearly part of the things we’re thinking about in terms of why we want to have fast and cheap inference—and a lot of it.

If you think about where fleet composition is, obviously pretraining uses general-purpose GPUs, and it does so because that’s where the stack is, and it’s good for that.

But as you move forward, as more reinforcement learning is being done and more agents are being used—agents do a lot of this as well, in terms of doing lots of chains of thought and a lot of analysis—I can imagine that the amount of inference needed, and the fleet, is going to keep growing.

There’s going to be very strong growth in the amount of inference compute that a company like OpenAI needs, and so I see a big opportunity there to continue increasing our fleet in that way.

Ben Lorica. So, Richard, going back to our earlier discussion around CUDA: Is one of the takeaways that CUDA is no longer a moat?

Richard Ho. That’s a spicy take.

I think it helps everybody to be able to operate on the best hardware. And if the best hardware for a particular application is—

Ben Lorica. But the whole point of this is hardware is only one thing. There’s software too, right?

Richard Ho. Software, yeah.

Ben Lorica. Software was always an important part of that NVIDIA ecosystem, but now we have tools that may be able to unlock other hardware.

Richard Ho. It’s true, and I think there are a lot of people who would like to see that happen.

Is that a goal of ours? Not necessarily. But is that something we’re enabling with this? I think it is enabling that optionality for developers and software people to be able to use other software stacks.

Ben Lorica. So, the people you—I don’t know if you went to Hot Chips, but if you did—the people you ran into who were not working in one of these AI labs, so not Google or NVIDIA or somewhere else, what was the reaction?

Richard Ho. What was the reaction? Oh, good. Yeah, I was there. We presented. It was a lot of fun.

Hot Chips is almost like a reunion of old friends, actually, for the most part.

I think the people I met who were not in one of these AI labs were very encouraged that this is a new direction in computer architecture. I think they mostly focused on how fast it happened. They focused on the nine months, and there were a lot of questions about that.

Ben Lorica. By the way, nine months—in the past, you’re just at FPGA.

Richard Ho. Yeah, exactly. Let me talk a little bit about that. I think it’s actually an interesting story.

We came here to OpenAI, and there was nothing here from scratch in terms of hardware. There were no people who were focused only on hardware, so we had to build this up.

I think the key thing we found—and it was kind of a growing together—was that the AI models helped us and the people, and we learned from each other and stepped forward.

That was a real synergy that I don’t think we fully understood we were going to be able to have when we got here, when the hardware people started getting hired.

But as we were progressing, the excitement level just kept growing. We could say, “Hey, we can get this level of performance. We can have the AI models rewrite our code. They can optimize stuff. They can suggest stuff.”

And as the AI models got better, we did more with them. There was almost this snowball effect as people started using them. People got excited.

We were using AI in physical design, in verification, in the actual microarchitecture design, in performance analysis, and, as I said, in the kernels and software. The amount of AI people were using just kept growing and growing.

What we found was that if we had not had it, that nine months would have been impossible. It would not have been possible had we just been doing our usual engineering processes. But with AI, yes, that’s what we achieved.

I would say that we had done some simulation and some performance analysis prior to the RTL start, so that’s why you probably hear some other dates and some other durations. But from the moment we started RTL to when we taped out, that was the nine-month part that we’re highlighting.

And for something as large as this—and this is full reticle—for your listeners, what full reticle means is that this is the largest that we can manufacture.

Fabs like TSMC, Intel, and other fabs use light to basically print the circuits onto silicon, and the largest thing you can do is this size. It’s roughly 800-ish square millimeters.

That’s a lot of silicon. It’s very rare to be able to get a chip this size done, with that many transistors and gates and things like that, in nine months. It’s extremely hard, especially starting with no previous legacy design to build on top of. It was a completely brand-new design.

Ben Lorica. By the way, I think I read this, but I’m not sure, so I want to clarify. Do you or do you not use speculative decoding?

Richard Ho. In our benchmarks, we did not do speculative decoding, and I can tell you why. Some of the other benchmarks do speculative decoding, or something called MTP.

Ben Lorica. By the way, give our listeners a one-sentence definition.

Richard Ho. Exactly.

Speculative decoding basically takes a smaller model. You run the smaller model a few times and get some speculative tokens, meaning the smaller model guesses that this is going to be the next token, and you use that information to get the real next token.

If you’re able to do this, you go faster. In general, what we’ve found is that if you use speculative decoding, you get somewhere between three and five times faster response out of your hardware. So it’s a good thing.

The other name for that is MTP, meaning multi-token prediction. If you don’t use speculative decoding, it’s called STP, or single-token prediction.

Basically, in order to run it, you have to have these smaller models in order to do that speculation.

In our sprint to do this benchmarking, we didn’t have time to train a smaller model, so we just used single-token prediction. In other words, we did not use speculative decoding.

Ben Lorica. So this is still something extra that you can do, that you’ll have to—

Richard Ho. Right, exactly. In production, we will do it. In production, we will have these smaller, what they call draft models.

Ben Lorica. So the performance numbers will be even better.

Richard Ho. They will be even better. You could predict three to five times better.

And that was the interesting thing. We compared our single-token prediction without speculative decoding against the published numbers of some of the other GPUs that were using speculative decoding, and our single-token prediction number was comparable and slightly better than the multi-token prediction of the other hardware that we compared against.

So you can expect that when we get multi-token prediction running, once we train up the draft models, we should get another three-to-five-times performance boost on top of that.

So it’s looking very promising, should I say, Ben?

Ben Lorica. Yes. Yeah.

So, in closing—of course, I can talk to Richard for hours, but we have time constraints here—for our listeners, I guess, Richard, the takeaway more broadly is that AI takes a pool of experts with deep domain expertise and helps them get things done faster, move faster.

Because the one thing that I think remains true, a truism, and has always been true in the hardware space, is that there are only a few teams out there that can actually build hardware. And usually within these teams, they’ve built several generations of hardware because, basically, the first few you build don’t actually work that well.

So experience still matters, but AI unlocks that. It supercharges that experience. Right?

Richard Ho. Perfect. Perfect summary. Exactly right.

That’s one of the key messages here. With AI, with a good team, you unlock a lot of productivity and a lot of performance benefits.

Ben Lorica. And by the way, your team is not the only team that is using AI to build hardware, right? I’m pretty sure a lot of other teams are.

Richard Ho. I’m sure that’s true. In fact, I think—

Ben Lorica. But you’re the only one that I think I’ve seen where, okay, they’re using AI, but they actually produced a chip, and it actually works.

Richard Ho. Yeah, I think we were one of the early ones. I think this is the proof point.

We want to publicize the fact that this is a proof point, and I think it’s not only chips. It’s general productivity, engineering, and other productivity.

This is the proof point. I think people talk a lot about, “Oh, when AI comes, what’s going to happen?”

What we’re trying to show as a proof point is that your team becomes better faster, and you get a better product at the end of the day.

Ben Lorica. And you can imagine mapping this same thing to another field, like drug discovery. Right?

Richard Ho. Exactly.

Ben Lorica. Right. Exactly.

Richard Ho. Exactly.

So my take on this—and this is my personal take—is that there’s no world where the human experts are not needed. We need to have the human experts there to be able to do this work.

What you’ve basically done is turn them into super engineers, or super scientists, or super productive workers in some way. You’ve given them these superpowers.

You just have to make sure they understand how to use it and how to get the best use out of it.

And from there, you can do a lot more than we thought was possible a year ago or two years ago. That’s the proof point that we’re trying to demonstrate here.

Ben Lorica. And with that, thank you, Richard.

Richard Ho. Thanks.