Disrupting the AI infrastructure with MicroLEDs

[MUSIC] 

PAOLO COSTA: Hi, I’m Paolo Costa from Microsoft Research in Cambridge in the UK. And today, I would like to talk to you about how we can reimagine the next-generation infrastructure using microLEDs.  

So the trend where we’re seeing across many generations of AI accelerators is that while the compute has been able to scale at a reasonable pace, the network and the memory have been lagging behind. And indeed, what we are seeing is that the AI workloads, especially inference ones, are becoming I/O bound. What it means is that we cannot fully utilize our GPUs because they are bottlenecked on either the memory or the network or both. One of the key reasons for this trend is that today’s communication technologies force you to make a tough choice. 

On the one hand, you can use optical communication with lasers. You can have very long reach, like10s, 100s meters and beyond, but the cost and power is very high, and also, reliability is very low. And so whenever you can, you’re forced to use copper cables, [which] are very low cost, extremely low power, and highly reliable, but the reach is very short. In fact, it’s less than 2 meters for GPU to GPU and only a few millimeters for GPU to memory. And this is actually creating a number of challenges for system designers. 

For example, recently, NVIDIA introduced the latest Blackwell [NVL72] rack, where you have 72 GPUs just inside one rack. This is 120 kilowatts per rack or the equivalent of 60 dishwashers. And projections for future generations is to go beyond half a megawatt per rack. And the only reason you have to put so many GPUs inside a single rack is because you’re forced to use copper for their high-bandwidth connectivity, multiple terabits [per second] per GPU.

But as you can imagine, this is becoming extremely challenging from a cooling, thermal, and mechanical standpoint. And ultimately, this will limit the number of GPUs that you can interconnect at high bandwidth. Similarly for the memory, since we can only put the memory next to the GPUs, there is only so much real estate that you can use. And so again, this is constraining the amount of memory that you can make available to each GPU.  

So why is optics so high cost and high power? The reason is because today we use lasers that are extremely expensive; they are very high-power hungry. And so in order to amortize their cost and power, we have to transmit faster and faster. We used to do 25 gigabit [per second] and 50 gigabit, and now we do 100 gigabit [per second] per channel. Next generation is going to be 200 gigabit [per second] per channel.

So, for example, today in one transceiver, we have eight lasers, each one transmitting 100 gigabit [per second]. However, as you go up in speed, you need more and more complex electronics to compensate for transmission impairments. And this also adds up to the power and also the cost because you have to use the latest CMOS process node technology like 5 nanometers or 3 nanometers in the next generation. Also, since lasers operate in the infrared region of the space, you cannot use silicon photodetectors. You have to use more complex materials like silicon germanium or gallium arsenide. And this is where the high power and high cost comes into the picture. And also, when you operate at such high speed, margins become smaller and smaller, and this also ultimately leads to higher failure rate.

So in our approach, we decide to go to the completely opposite direction. Rather than having a few channels operating at very high speed, we adopt a wide and slow architecture, which means having many channels, each one operating at very low speed.  

So, for example, to get to the 800 gigabit, rather than having eight channels at 100 gigabit, we can have 400 channels, each one operating at 2 gigabit. The advantage is that now, since we operate at a very low speed, the electronics become much more efficient. We don’t need to use advanced signal processing. We can do anything in the analog domain, and we can also relax the requirement in terms of process node size.

Now, we can’t use lasers because it would just break the bank. So instead, we use microLEDs. MicroLEDs are very similar to the LEDs, for example, that we have in this room, but they’re much smaller, like 5 to 10 microns in size. And this is a technology that has been developed for next-generation displays like smart watches or head-mounted devices. And they provide three advantages. The first one is that they are extremely cost effective. Second, they can operate at very, very low power. And also, they’re very small, and so what it means is that you can pack many of them inside a single space. Also, they operate in the visible range like green, blue, red. Why is this important?  

Because now on the receiving side, we can just use a standard camera sensor detector like the one that we have in our phone. And so when you combine all of this together, what you’re seeing is that we can get very low cost and very low power. Also, since we can put many of them in a very small space and they’re very low power and cost effective, it’s easy to overprovision them, which also makes them much more reliable. 

Now, I make it sound, like, very simple, but actually realizing this vision in practice required a lot of innovation across the stack and also required a very multidisciplinary team with expertise across many fields from integrated photonics, material science, lens design, and digital and analog design.

So one of the first challenges we had to solve is how to make these microLEDs that are designed for a very slow rate to operate them very, very fast. But also, we have to look across the stack. For example, we designed new type of optics that allow to better couple the light into the fiber. Also, we don’t use standard fiber; we use imaging fibers with thousands of cores. This is a fiber typically used for medical applications like endoscopy, and instead, we use that for communication. And also, we’ve been looking at beyond the single module. For example, looking at designing new connectors and optical circuit switch that are compatible with these new technologies.

Now, we really have great support both from Microsoft Research and from the Azure team. And together with our suppliers, we were able actually to demonstrate that it’s possible to be the technology that breaks this trade-off. So we can have at the same time low power, low cost, and high reliability while still adding a reach of up to 50 meters. And one of the strengths of this technology is that it’s fully compatible with today’s form factor. And so what it means is that it doesn’t require any changes to existing switches or NICs and can be immediately deployed. Also, since it operates at the physical layer, it’s completely [protocol agnostic]. So you can use it for Ethernet, CXL, PCIe, and so on.

This is already a big win because it can address today’s pain points. But what makes me excited is the fact that it can also open up new, exciting opportunities as we think of the next generation of AI infrastructure. For example, on the networking front, since now we are not constrained by the short distance of copper cables, we can now expand the high bandwidth domain to multiple racks. And also, we can use more flexible topologies; for example, using optical circuit switch, which will improve our efficiency and overall utilization.

Also, on the memory front, we are not constrained to put the memory next to the GPU, but we can now disaggregate the memory by moving the memory off the package, which would allow to increase the memory bandwidth and capacity that is available to a GPU but also make it possible to explore novel type of memories that are different from the ones that we use today.

And finally—and this is really our North Star—would be the idea actually to put the microLEDs directly on top of the GPU because this can unlock the full aerial bandwidth. Now, this would require a lot more innovation happening at the packaging side and thermal and microLED devices. But if we can get there, this can really be game changing because this can enable to have multiple petabytes of bandwidth coming out from the GPU. 

And so what we really would like to do is to engage with the broader research and the industry community to work together, understanding how we can best leverage this property and design together the next-generation AI infrastructure.

So thanks so much for your time and looking forward to connect with all of you.