Comment by e63f67dd-065b | Hacker News Reader

Comment by e63f67dd-065b | Hacker News Reader

CXL is useful for many more things beyond just memory expansion (which is a "type 3 device").

The purpose of CXL is to allow for memory coherency between different CXL devices. To quote the spec on type 2 devices:

> CXL Type 2 devices, in addition to fully coherent cache, also have memory, for example DDR, High-Bandwidth Memory (HBM), etc., attached to the device. These devices execute against memory, but their performance comes from having massive bandwidth between the accelerator and device-attached memory. The main goal for CXL is to provide a means for the Host to push operands into device-attached memory and for the Host to pull results out of device-attached memory such that it does not add software and hardware cost that offsets the benefit of the accelerator.

There's some cool things you can do with CXL, like resurrecting the whole persistent memory idea with low-latency flash, making hardware offload devices more capable since you now get free cache coherency, and a whole bunch of other stuff.

But yes, it's really not for consumer use-cases. The applications I've seen colleagues work on are mostly enterprise stuff like cool RDMA integrations, cache-coherent flash, and more I can't talk about here.

I've got to imagine that at some point in the future when it's been common on server stuff for a while it'll end up as part of a next-gen "thunderbolt" style connection, simply because the silicon by then will be a commodity and it'll be cheaper to use the same thing everywhere. I'm imagining a docking station that doubles the ram, and gives it a powerful GPU/NPU/TPU for a workstation. Since the ram would be close to the accelerator over CXL they can talk directly and make full bandwidth use of it even if the laptop uses the remote ram as a "very very very fast swap" or something else to prioritize things. I'm not so sure it'd be useful for persistent storage at that level but it'd make for some really cool options for a portable workstation since you wouldn't need everything inside the device.

CXL 3.0 is where it gets interesting, where you can start to have switched fabric where many hosts can talk to many devices. Having that pool of ram & GPUs be on demand usable by whomever has some attractive possibilities. Also, one can just imagine having some pools of data in CXL memory that a host can just attach to & read, which seems like a cool possibility.

It does kind of upset me a bit that CXL 3.0 still seems purely host-to-switch-to-device oriented. If you have your formerly PCIe slots on your cores speaking CXL and doing directory memory over fabric, I'd really really love to be able to talk to other hosts. Maybe that happens & is possible in 3.0, but it feels like CXL isnt paving that cowpath, isn't making is obvious, and that there will be a bunch of proprietary nasty ways to bridge computers & chat over CXL that are all non-standard, & I wish CXL had been more direct about making themselves & their upcoming switched fabric viable & interesting for host-to-host.

I think a large part of that is probably because the needs for doing that kind of host to host thing aren't well understood yet for the general cases. Right now I think that's being fleshed out in the market using the whole DPU model still. I think sadly you're going to be right about how it turns out for a while and it won't be until something like CXL 6.0 or whatever comes after it that starts to really address that.

I'm wondering when we'll see the first CXL DPU style device that uses some of the fancy new 800Gbps or even 1.6Tbps networking stuff that's being developed right now. That'd be enough bandwidth to put things on the other end of a data-center with very little other than the latency penalty.

CXL for Persistent Memory, as the Optane successor (or rather, sole-survivor if you will) is what excites me.

Persistent Memory databases are such a neat idea, it's a shame that the hardware for it isn't commonplace.

Yeah optane seemed like it had so much potential -- with the right abstractions, it neatly dealt with the eternal problem of the separation between RAM and disk. Much of DB and storage system engineering comes down to decisions around when to persist what, and optane was very promising in allowing for new architectures that had much better performance at lower complexity.

Alas, optane's dead now. I do know people actively working on resurrecting a lot of pmem work on low-latency flash, however, and it seems like this is one area with a low of momentum behind it.

I'm utterly unqualified to discuss using it in practice but it always seemed like PMEM would have required a fundamental idiom shift away from the DRAM concept.

If you treat it like memory it's always going to be slow memory. If you treat it like storage, it can be very fast storage. If that makes sense.

Software that "treats storage like memory" would end up looking a lot like javacard imo. Or something like Samsung's in-memory key-value database stuff. But it wouldn't really look like a linux kernel allocating memory inside an all-pmem partition.

Difference, in my head, being that essentially the filesystem is an unnecessary layer/abstraction in the middle. You want something that either looks like a garbage-collected runtime, or an in-memory database that vacuums, or LISP collections of object-trees, etc.

There is no point to having a separation between memory and disk anymore, that is the point - "resource allocation is disk persistence" so to speak.

It would have required a big-bang rewrite/second-system that is just not possible with the dominance of the existing RAM/storage dichotomy. Or at least a couple killer apps from large vendors/etc that really outperformed what was possible with RAM.

Some support has been added to the Linux kernel like DAX [0] to avoid extra copy operations [0].

SNIA has some interesting talks regarding using non volatile memory and its applications (DBs). Where using pmem and DAX to directly to store logging operations [1].

[0] What is Direct-Access (DAX)? https://kb.pmem.io/faq/100000008-What-is-DAX/

[1] SDC2020: How can Persistent Memory Make Databases Faster, and How Could we go Ahead? https://www.youtube.com/watch?v=rTgITrVhpQM

Pavlo has some good papers on architecting DB's around PMEM

"How to Build a Non-Volatile Memory Database Management System"

https://db.cs.cmu.edu/papers/2017/p1753-arulraj.pdf

Under 3.1 Allocator Interface, there is a great point about reducing wear levelling by rebalancing the B+ tree operations to reduce the number of writes thus extending the lifespan of the hardware. Not much of a concern on the small scale, however in large deployments even single-digit improvements will make a difference.

Sometimes it becomes useful to appreciate the hardware and what it is doing, reminds me of the quote.

* You don't have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy *. Jackie Stewart

>CXL for Persistent Memory, as the Optane successor (or rather, sole-survivor if you will) is what excites me.

wasn't Optane (hardware) trying to use CXL?

CXL for CPU access to DRAM is overkill. Plenty of server mobos allow you to directly attach 2 TiB of DRAM.

Some 4 and supposedly 6TB now too.

Last one I put physical hands on was running 24TB, and was expandable to 32TB.

Rack level disaggregated compute-memory-storage-accelerator architectures allow for dynamic partitioning/aggregation of hardware to suite concurrent workloads and their evolution over time and make it easier to achieve both cost efficiency as well as incremental, continuous and non-disruptive hardware upgrades.

Wendell Level1techs has been talking about his Liqid PCIe Fabric stuff lately, sounds pretty sick, this mention inspired me to go search for his other videos on the topic

https://www.youtube.com/watch?v=x1NWdoXmVsE

https://www.youtube.com/watch?v=Iwep9D_q4e0

https://www.youtube.com/watch?v=zXlWeVrsk14

Reply on news.ycombinator.com