Break The Memory Barrier And Integrate HBM And DDR5

What Is DDR5

DDR5 (Double Data Rate 5) is a computer memory standard used in high-performance computing and data center applications. It is the successor to DDR4, providing higher bandwidth and lower latency.

Here are the main features and improvements of DDR5:

Increased bandwidth: DDR5 provides higher data transfer rates and bandwidth than DDR4. The data transfer rate of DDR5 memory modules can reach 8 data bits per pin, which is equivalent to two data bits per clock cycle. 

This allows DDR5 to have higher theoretical bandwidth to meet the demand for large-scale data processing and high-performance computing.

Capacity expansion: DDR5 supports higher memory module capacity. DDR5 memory modules typically range in capacity from 16GB to 128GB and beyond. This provides a greater choice of memory capacity and is suitable for handling large data sets and complex computing tasks.

Low power consumption: DDR5 achieves lower power consumption by adopting low voltage operation and higher efficiency optimization. This is important for applications that require high performance but require energy savings, such as data centers and cloud computing environments.

Latency reduction: DDR5 further reduces the latency of memory access. With improved circuit design and higher frequencies, DDR5 is able to provide faster response times and lower access latency. 

This is very beneficial for applications that require high-speed data transfer and low latency, such as high-performance computing, artificial intelligence, and large-scale data processing.

Memory Error Repair: DDR5 introduces more powerful error repair and correction capabilities to improve data reliability. 

It supports higher levels of Error Correction Code (ECC) and Cyclic Redundancy Check (CRC) capabilities to detect and correct more memory errors, reducing the risk of data corruption and system failure.

Overall, DDR5, as the latest generation memory standard, offers higher bandwidth, greater capacity, lower power consumption, and lower latency.

 It is suitable for applications that require high memory performance and throughput, such as large-scale data processing, high-performance computing, artificial intelligence, and so on. 

However, the widespread adoption of DDR5 memory requires corresponding motherboard and processor support, so factors such as compatibility and cost need to be considered when choosing and upgrading a system.

What Is HBM

HBM (High Bandwidth Memory) is a high bandwidth memory technology designed to provide high performance memory access speed and bandwidth. It is widely used in high-performance computing, graphics processors (Gpus), and other areas that require a lot of memory bandwidth.

The following are the main features and working principles of HBM:

Vertical Stacking: HBM uses a vertical stacking design that stacks multiple DRAM chips together to form a 3D structure. This stacked design helps reduce the physical size of memory modules, increases memory capacity density, 

and allows for shorter interconnect distances, thereby reducing latency and power consumption.

High bandwidth: HBM provides very high memory bandwidth. Each HBM stack can transmit data simultaneously through multiple channels, with multiple data pins on each channel. 

This parallelism and the high number of channels enables HBM to achieve extremely high data transfer rates, reaching bandwidth levels of hundreds of GB/s.

Low power consumption: HBM achieves low power consumption by using low voltage operation and efficient energy management mechanisms. 

Compared to traditional GDDR (Graphics Double Data Rate) memory, HBM provides higher performance at the same bandwidth and is more energy efficient in terms of power consumption.

Low latency: Due to HBM's vertically stacked and tightly interconnected design, the distance between memory and processor is shorter, which reduces the latency of data transfer. 

This is important for applications that require fast response and low latency, such as high-performance computing and graphics rendering.

High density: HBM has a high memory capacity density because multiple DRAM chips can be stacked together to form a stack that takes up less physical space. 

This is useful for systems that require large amounts of memory in limited Spaces, such as high performance computing, artificial intelligence, and data centers.

It is important to note that HBM is a memory technology specifically designed for high-performance computing and graphics processors, and its high bandwidth and low latency give it an advantage in these areas.

 However, HBM also has some limitations, such as high cost and relatively small capacity range. As a result, HBM is often used for applications with high memory bandwidth and performance requirements, while traditional DDR memory remains the more common and economical choice.

Future Connections Between HBM And DDR5 Are Anticipated

By 2024, if tens, hundreds, thousands or even tens of thousands of accelerators need to be stitched together, there will be no shortage of interconnects.

Nvidia has NVLink and InfiniBand. Google's TPU Pods communicate with each other using optical path switches (OCS). AMD has Infinity Fabric for ie-to-die, chip-to-chip, and soon node-to-node traffic. And, of course, there's good old-fashioned Ethernet, which Intel uses in Gaudi2 and Gaudi3.

The trick here is not to build a large enough grid, but to avoid the huge performance penalties and bandwidth bottlenecks associated with going off packages. Nor does it do anything to address the fact that the HBM memory on which all AI processing depends is tied to computation at a fixed rate.

"The industry is using Nvidia Gpus as the world's most expensive memory controllers," Dave Lazovsky said his company Celestial AI just raised $175 million in a Series C round backed by USIT and numerous other venture capital giants to commercialize its Photonic Fabric.

Last summer, we looked at Celestial's Photonic Fabric, which contains a series of silicon photonic interconnects, intermediary layers, and small chips designed to separate AI computing out of memory. Less than a year later, light wranglers says it is working with several hyperscale customers and a large processor manufacturer to integrate its technology into their products. To our disappointment, but certainly not surprising, Lazovsky did not name names.

But the fact that Celestial counts AMD Ventures as one of its backers, and that Sam Naffziger, vice president and product technology architect, discussed the possibility of co-encapsulating silicon photonic chips on the same day of the announcement, certainly raised some eyebrows. That being said, AMD funding this photonics startup doesn't mean we'll see Celestial's little chip in Epyc cpus or Instinct GPU accelerators.

While Lazovsky couldn't say who Celestial is working with, he did offer some clues about how the technology will be integrated, as well as a preview of upcoming HBM memory devices.

As we discussed when we first explored Celestial's product strategy, the company's components fall into three broad categories: small chips, intermediary layers, and based on Intel EMIB or TSMC CoWoS (known as OMIB).

Not surprisingly, much of Celestial's appeal is focused on small chips. "What we're doing is not trying to force our customers to adopt any one particular product implementation. The current risky, fast, and easy way to provide photonic structured interfaces is through small chips, "Lazovsky told The Next Platform.

In general, these small chips can be used in two ways: either to add additional HBM memory capacity, or as an inter-chip interconnect, sort, or similar optical NVLink or Infinity Fabric.

These small chips are slightly smaller than the HBM stack and provide an opto-optical interconnect suitable for 14.4 Tb/ s or 1.8 GB/ s of total off-chip bandwidth.

That being said, we are told that small chips can be made to support higher bandwidth. The technology can support about 1.8 terabytes per second per square millimeter. At the same time, Celestial's second-generation Photonic structure will increase from 56 Gb/ s to 112 Gb/ s PAM4 SerDes and increase the number of channels from four to eight, effectively quadrupling the bandwidth.

Therefore, 14.4 Tb/ SEC is not the upper limit, but the result of what existing chip architectures can handle. This makes sense, because otherwise any extra capacity would be wasted.

This connection means Celestial can achieve interconnect speeds similar to NVLink, but with fewer steps.

While chip-to-chip connectivity is relatively self-explanatory - placing a small photonic structure chip on each package and aligning the fiber connections - memory expansion is a different animal altogether. While 14.4 Tb/ SEC is far from slow, it is still a bottleneck for multiple HBM3 or HBM3e stacks. This means that adding more HBM will only push your capacity past a certain point. However, replacing one stack with two HBM3e stacks is not nothing.

Celestial has an interesting solution to this through its memory expansion module. Since the bandwidth is capped at 1.8 GB/ s anyway, the module will contain only two HBM stacks totaling 72 GB. A set of four DDR5 DIMMs complements this, supporting up to 2 TB of additional capacity.

Lazovsky hesitated to reveal all the details of the product, but did tell us that it will use Celestial's silicon photonic intermediary layer technology as the interface between HBM, interconnect, and controller logic.

Speaking of the module's controller, we learned that the 5 nm switch ASIC effectively converts HBM into a direct-write cache for DDR5. "It gives you the capacity and cost of DDR and all the advantages of HBM's bandwidth and 32 pseudo-interconnect channels, which hide latency," Lazovsky explained.

That's not far off from the performance of Intel's Xeon Max or Nvidia's GH200 superchip, he added. "It's basically a supercharged Grace-Hopper that doesn't have all the overhead and is much more efficient."

How much more efficient? Lazovsky claims: "Our memory transaction energy overhead is about 6.2 pejoules per bit, and the energy overhead for remote memory transactions via NVLink, NVSwitch is about 62.5 pejoules," he said, adding that the latency is not too bad.

"The total round-trip latency of these remote memory transactions, including travel through photonic structures and memory read times, was 120 nanoseconds," he added. "So it's going to be a little bit more than about 80 nanoseconds of local memory, but it's going to be faster than going to Grace and reading the parameters and pulling it to Hopper."

We understand that sixteen of these memory modules can be combined to form a memory switch, and multiple of these devices can be connected using fiber shuffling.

The implication here is that, in addition to computing, storage, and management networks, chips built using Celestial Interconnect will not only be able to connect to each other, but will also be able to build a common pool of memory.

"This allows you to do machine learning operations, such as broadcast and reduce, in a very, very efficient way without having to switch," Lazovsky said.

The challenge for Celestial is timing. Lazovsky told us that he expects to start offering samples of Photonic Fabric small chips to customers sometime in the second half of 2025. Then, he expects it will be at least another year before we see products using the design hit the market and significantly increase sales in 2027.

Celestial, however, is not a startup pursuing silicon photonics. Ayar Labs, another photonics startup backed by Intel Investments, has integrated its photonic interconnect into a prototype accelerator.

Then there's Lightmatter, which raised $155 million in Series C funding in December and is trying to do something very similar to Celestial through its Passage intermediary layer. At the time, Lightmatter executive Nick Harris claimed that its customers used Passage to "scale up to 300,000 node supercomputers." Of course, like Razovsky, Harris won't tell us who its clients are.

Then there's Eliyan, which is trying to get rid of the mediation layer and its NuLink PHY entirely, or enhance the performance and scale of the mediation layer if you must have them.

No matter who comes out on top in this race, it seems only a matter of time before the shift to co-encapsulating optics and silicon photonic intermediaries really begins.