The Future of System Memory Is Mostly CXL | Hacker News Reader

The Future of System Memory Is Mostly CXL

nextplatform.com

40 pointsrbanffy14 comments

Figure 3 (on the right) looks a lot like putting the memory controller back on the north bridge chip. And there’s reasons the memory controller was moved onto the CPU in the first place. Most workloads are very latency sensitive, with low latency ddr4 dimms outperforming much higher bandwidth ddr5 dimms.

The article is really about disaggregating memory from CPU capacity, and rightly points out that sometimes a marginal CPU in an SMP system is used purely as a memory controller.

The article mentions CTX is similar to IBM’s OMI, which they use in their high end POWER boxes. Their goal is to have separate processor and memory drawers on their servers to better mix and match workloads and capacity, trading latency for flexibility and bandwidth. I suspect their mainframes are next, as their RAM is located in the same drawer as the processors, but attached to a system controller chip (z15’s chip had almost a gigabyte of EDRAM working as an L4 cache on each drawer). With z16’s cache architecture, it may be better shielded from extra memory latency and able to work on a simpler NUMA topology.

CXL isn't replacing system ram, it's just allowing you to have slow (compared to ram) connection to a big chunk of ram that's wicked fast (compared to storage). So you could for instance buy 16 nodes with 256GB ram then buy a couple 8TB connect CXL units. Then as needs dictate you could add any fraction of the CXL connected ram to any of those systems.

Nobody is ditching CPU connected ram for CXL.

So memory and I/O won't be on separate busses anymore? Will all CXL devices--formerly known as PCIe--be memory mapped now? Will IOMMU as a thing separate from MMU no longer need to be a thing?

Thank you! Nowhere in the article do they tell you what CXL is!

This is a sin this website often commits. They just assume we know what they are talking about and they sometimes talk about the most esoteric tech we muggles never get to see.

PCIe devices are memory mapped in most systems now. The IOMMU and the MMU are basically the same thing, but there are separate ones because the IOMMU has to handle multiple "bus masters" - the IOMMU's main job now is to protect memory from transfers initiated by malicious accelerators on the PCIe bus. The MMU is separate to prevent contention.

The fiction of separate I/O and memory space is another abstraction that carries over from the 80's.

It's kind of fun that my first personal computer, the Apple II, used memory-mapped IO (because the 6502 was designed to be cheap, but still). :-)

I have no idea about any of these things, but running memory bus via PCI-e sounds like a major regression in terms of power usage. Also.. what about latency? I mean, having all your RAM pooled into one coherent logical space sounds neat on paper until you realise that now you have to deal with a non-trivial NUMA architecture with unpredictable performance hits.

I'm sure that this technology will have it's place for RAM in certain markets, but high-performance computing seems to prefer ultra-wide memory interfaces with energy efficient RAM.

CXL = Compute Express Link https://en.wikipedia.org/wiki/Compute_Express_Link

It uses the physical and electrical layers of PCIe and is all about coherent memory access between CPUs and devices allow each to potentially access the others memory.

So DMA ? What could go wrong ?

DMA over PCIx16. Probably will have some severe latency issues to overcome just with the distance the electricity will have to cover.

obviously, latency is the big question. it's fine to say "let's aggregate memory across multiple systems via a coherent layer on PCIe", but you're talking about taking away 50ns DRAM. enthusing about PCIe 7 bandwidth is not an answer...

and when is CXL-over-fabric coming?

Reply on news.ycombinator.com