Even a laptop can run RAM externally thanks to CXL | Hacker News Reader

Even a laptop can run RAM externally thanks to CXL

techradar.com

111 pointsashvardanian81 comments

This wasn't mentioned in the article but CXL Stands for Compute Express Link which is an open standard, CPU-to-Device Interconnect, I guess they're on v3.0 now. First I had heard of it as well. 3.9GB/s on the low end and as high as 121.0GB/s with the latest 16x 3.0 spec over serial connection

https://www.computeexpresslink.org/about-cxl https://en.wikipedia.org/wiki/Compute_Express_Link

I'm surprised that it is not known on HN. This is standard developed to enable different server configurations in the data center from what I've read through the years. The universal interconnect allows for adding and removing devices without special allowances for each device type as far as I get it.

It might work for laptops, but it is not the main goal and it might be only Framework who will do something in that direction.

These days that bulk of us people doing dev work aren't getting to play in the actual DC, it's all so abstracted away. Frankly looking to get a used rack or two at home here, just so I can play with the HW a bit.

(I do embedded and not server stuff mostly these days anyways, but generally interested in systems stuff, so)

I was talking with a colleague about this just the other day. We all used to have home labs, because we could get reasonably near to what business used and learn that way. Now, what goes on in is so different, frankly you won't learn much with one.

Obviously if you're just interested then go for it. It's just not practical generally speaking.

For myself, I've got 3gbps fiber to the house, and a need to host something. Pondering buying a used server instead of dishing money out to AWS or GCP. Question is if the power bills make sense.

I've spent a lot of time down this rabbit hole recently. I have a Dell M1000e blade server (16 blades, each with dual CPUs). A comical amount of compute for a home, sounds like a leaf blower. Hosted a couple dozen small apps on it. Power usage was 1.5kW (ouch), made less painful by large rooftop solar.

Just today, I replaced the whole thing with a single mini PC with a laptop Ryzen chip (6900HX). Pulls like 60W at the wall under load.

This is a circuitous way of saying that, for non-mission critical stuff, you can self-host and save a ton of money over the equivalent hosted option. You don't even need server hardware necessarily, depending on what you're trying to learn from the experience. Standard caveats about securing your home network apply.

(Note: I'm not actually seriously saying a 16x blade server and a mini PC are equivalent. I got the blade server to learn things (more networking, mostly), it was very, very overpowered for my workloads)

Yeah I think this likely where I'll end up going as well. What MiniPC did you end up working with?

The thing I'm building/hosting can benefit from plenty of cores, but for the start it's just prototype with a limited audience, so I don't need to go hardcore.

But I have a friend who has rackspace in a DC, and my thought was if I bought something in a rack form I could move it there and pay him to host it later. But I suspect it doesn't make a lot of sense, really.

I ended up with a Beelink GTR6, picked up used on eBay for $430. It's been great so far, but it's too early to vouch for it. You lose the nice IPMI/other server management features using consumer hardware, but that's part of the tradeoff.

Rack stuff can be efficient (<100W idling), especially recent model year stuff, but it gets pricey quickly. You could spec a used 1U single socket Epyc Rome or Milan for under $2,000 if you're willing to shop around, but if you don't have other rackmount equipment, I'd stay away as it's such a price (and power) premium

There's plenty of 10-yearish old PowerEdge servers around here for $500ish CAD full of SAS drives and high core count Xeons and a pile of memory. But yeah, power and noise. And I don't have ethernet running down to the basement yet, so not super enthused about having that share office space with me.

The laptop I'm typing on is a Ryzen 6850H, so not too far off from the 6900 you're talking about and it has plenty of oomph. I might consider such a thing.

> I have a Dell M1000e blade server (16 blades, each with dual CPUs). A comical amount of compute for a home, sounds like a leaf blower.

That's pretty awesome! I would have loved that at one point but they're extremely loud and I remember them needing 16 amp IEC leads

Loud is an understatement, I wear ear protection when I'm physically in the room for more than a minute.

And indeed, the power supplies run on ~200V, my house serendipitously is wired with a dedicated 240V/50A circuit because the previous owner did glasswork and had a massive electric kiln.

IT is an extremely diverse field with extremely diverse subprofessions - knowledge about esoteric hardware should not be assumed.

Is it possible to run an eGPU with this? Like the TH3P4G3 one can buy on AliExpress

This is based on serial PCI Express (PCIe), and allows block based transfer, cache coherence, etc. In the Device Types they talk about "Accelerators with Memory", type 2: https://www.logic-fruit.com/blog/cxl/compute-express-link-cx....

CXL is useful for many more things beyond just memory expansion (which is a "type 3 device").

The purpose of CXL is to allow for memory coherency between different CXL devices. To quote the spec on type 2 devices:

> CXL Type 2 devices, in addition to fully coherent cache, also have memory, for example DDR, High-Bandwidth Memory (HBM), etc., attached to the device. These devices execute against memory, but their performance comes from having massive bandwidth between the accelerator and device-attached memory. The main goal for CXL is to provide a means for the Host to push operands into device-attached memory and for the Host to pull results out of device-attached memory such that it does not add software and hardware cost that offsets the benefit of the accelerator.

There's some cool things you can do with CXL, like resurrecting the whole persistent memory idea with low-latency flash, making hardware offload devices more capable since you now get free cache coherency, and a whole bunch of other stuff.

But yes, it's really not for consumer use-cases. The applications I've seen colleagues work on are mostly enterprise stuff like cool RDMA integrations, cache-coherent flash, and more I can't talk about here.

CXL for CPU access to DRAM is overkill. Plenty of server mobos allow you to directly attach 2 TiB of DRAM.

Some 4 and supposedly 6TB now too.

Last one I put physical hands on was running 24TB, and was expandable to 32TB.

Rack level disaggregated compute-memory-storage-accelerator architectures allow for dynamic partitioning/aggregation of hardware to suite concurrent workloads and their evolution over time and make it easier to achieve both cost efficiency as well as incremental, continuous and non-disruptive hardware upgrades.

Wendell Level1techs has been talking about his Liqid PCIe Fabric stuff lately, sounds pretty sick, this mention inspired me to go search for his other videos on the topic

https://www.youtube.com/watch?v=x1NWdoXmVsE

https://www.youtube.com/watch?v=Iwep9D_q4e0

https://www.youtube.com/watch?v=zXlWeVrsk14

CXL for Persistent Memory, as the Optane successor (or rather, sole-survivor if you will) is what excites me.

Persistent Memory databases are such a neat idea, it's a shame that the hardware for it isn't commonplace.

Yeah optane seemed like it had so much potential -- with the right abstractions, it neatly dealt with the eternal problem of the separation between RAM and disk. Much of DB and storage system engineering comes down to decisions around when to persist what, and optane was very promising in allowing for new architectures that had much better performance at lower complexity.

Alas, optane's dead now. I do know people actively working on resurrecting a lot of pmem work on low-latency flash, however, and it seems like this is one area with a low of momentum behind it.

I'm utterly unqualified to discuss using it in practice but it always seemed like PMEM would have required a fundamental idiom shift away from the DRAM concept.

If you treat it like memory it's always going to be slow memory. If you treat it like storage, it can be very fast storage. If that makes sense.

Software that "treats storage like memory" would end up looking a lot like javacard imo. Or something like Samsung's in-memory key-value database stuff. But it wouldn't really look like a linux kernel allocating memory inside an all-pmem partition.

Difference, in my head, being that essentially the filesystem is an unnecessary layer/abstraction in the middle. You want something that either looks like a garbage-collected runtime, or an in-memory database that vacuums, or LISP collections of object-trees, etc.

There is no point to having a separation between memory and disk anymore, that is the point - "resource allocation is disk persistence" so to speak.

It would have required a big-bang rewrite/second-system that is just not possible with the dominance of the existing RAM/storage dichotomy. Or at least a couple killer apps from large vendors/etc that really outperformed what was possible with RAM.

Some support has been added to the Linux kernel like DAX [0] to avoid extra copy operations [0].

SNIA has some interesting talks regarding using non volatile memory and its applications (DBs). Where using pmem and DAX to directly to store logging operations [1].

[0] What is Direct-Access (DAX)? https://kb.pmem.io/faq/100000008-What-is-DAX/

[1] SDC2020: How can Persistent Memory Make Databases Faster, and How Could we go Ahead? https://www.youtube.com/watch?v=rTgITrVhpQM

Pavlo has some good papers on architecting DB's around PMEM

"How to Build a Non-Volatile Memory Database Management System"

https://db.cs.cmu.edu/papers/2017/p1753-arulraj.pdf

Under 3.1 Allocator Interface, there is a great point about reducing wear levelling by rebalancing the B+ tree operations to reduce the number of writes thus extending the lifespan of the hardware. Not much of a concern on the small scale, however in large deployments even single-digit improvements will make a difference.

Sometimes it becomes useful to appreciate the hardware and what it is doing, reminds me of the quote.

* You don't have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy *. Jackie Stewart

>CXL for Persistent Memory, as the Optane successor (or rather, sole-survivor if you will) is what excites me.

wasn't Optane (hardware) trying to use CXL?

I've got to imagine that at some point in the future when it's been common on server stuff for a while it'll end up as part of a next-gen "thunderbolt" style connection, simply because the silicon by then will be a commodity and it'll be cheaper to use the same thing everywhere. I'm imagining a docking station that doubles the ram, and gives it a powerful GPU/NPU/TPU for a workstation. Since the ram would be close to the accelerator over CXL they can talk directly and make full bandwidth use of it even if the laptop uses the remote ram as a "very very very fast swap" or something else to prioritize things. I'm not so sure it'd be useful for persistent storage at that level but it'd make for some really cool options for a portable workstation since you wouldn't need everything inside the device.

CXL 3.0 is where it gets interesting, where you can start to have switched fabric where many hosts can talk to many devices. Having that pool of ram & GPUs be on demand usable by whomever has some attractive possibilities. Also, one can just imagine having some pools of data in CXL memory that a host can just attach to & read, which seems like a cool possibility.

It does kind of upset me a bit that CXL 3.0 still seems purely host-to-switch-to-device oriented. If you have your formerly PCIe slots on your cores speaking CXL and doing directory memory over fabric, I'd really really love to be able to talk to other hosts. Maybe that happens & is possible in 3.0, but it feels like CXL isnt paving that cowpath, isn't making is obvious, and that there will be a bunch of proprietary nasty ways to bridge computers & chat over CXL that are all non-standard, & I wish CXL had been more direct about making themselves & their upcoming switched fabric viable & interesting for host-to-host.

I think a large part of that is probably because the needs for doing that kind of host to host thing aren't well understood yet for the general cases. Right now I think that's being fleshed out in the market using the whole DPU model still. I think sadly you're going to be right about how it turns out for a while and it won't be until something like CXL 6.0 or whatever comes after it that starts to really address that.

I'm wondering when we'll see the first CXL DPU style device that uses some of the fancy new 800Gbps or even 1.6Tbps networking stuff that's being developed right now. That'd be enough bandwidth to put things on the other end of a data-center with very little other than the latency penalty.

It's been a long day, managed to take a few seconds wondering what this interesting new "best CPUs" technical architecture was before realizing the article is SEO'd blogspam.

Maybe switch to the source article? https://www.servethehome.com/fadu-cxl-2-0-switch-and-pcie-ge...

Thank you, that's way better.

> 1TB of RAM

What are you going to do with the other 999,999,360KB?

999,999,360KB ought to be enough for anybody

If there's a reference I'm missing, enlighten me...

Anything to do with data processing, machine learning, llms- hundreds of gigabytes of ram can be incredibly nice.

Especially as pandas recommends ram equal to 5-10x the size of the dataset.

When you're an individual- not having to think about managing a cluster...

Edit: One of the lucky 10000- famous bill gates joke- got it. I walk away cultured

It's the classic Bill Gates quote taken out of context where he says that 640kb ought to be enough for anyone.

You're missing Gates 640KB of ram reference

I think Bill Gates once said 640Kb should be enough for anything

It's a joke based on the fake quote "640k (ram) should be enough for anyone"

To my eyes, it looks like this article consists almost solely of "external RAM exists".

OK? What laptops even support it? How much capacity is there ("over 1TB" is kind of vague)? So many questions...

The current devices are full size PCIe. And you need the latest Xeon CPU.

Are these the same Xeons that require a recurring subscription to unlock the silicon you've already purchased?

If anyone wishes to dive deeper and attempt some emulation* of CXL and to find out more, checkout both links as QEMU supports emulating CXL devices.

[0] Emulating CXL Shared Memory Devices in QEMU https://memverge.com/cxl-qemuemulating-cxl-shared-memory-dev...

[1] CXL support in QEMU https://www.qemu.org/docs/master/system/devices/cxl.html

How's the latency with these vs on board memory?

170-250ns was what they were saying a year ago, which they were saying was 2x native latency. https://www.servethehome.com/compute-express-link-cxl-latenc...

But with Genoa for example, socket to socket latency has climbed up to 220ns, and going across nodes on a socket is 110ns. I feel like CXL will be less than 2x a hit, if only because cores themselves are having higher and higher latencies. https://chipsandcheese.com/2023/07/17/genoa-x-server-v-cache...

CXL is a neat tech, but many innovative hardware systems go no where despite hundreds of millions of dollars of work and investment.

I have hopes. I can't help it. Cool shit is inspiring. I, however, am not holding my breath.

REST IN PIECES: SERVERS AND CXL https://www.nextplatform.com/2023/06/15/rest-in-pieces-serve...

If CXL is not being used by servers, will it ever become an offering for laptops?

I've recently read some PC chipset datasheet and it was almost 1000 pages of obscure tech: https://www.intel.com/content/dam/www/public/us/en/documents...

The slowest clock speed for DDR5 is 2,000Mhz or 0.5ns per cycle. At 2/3c, electricity should be able to travel about 10cm max in one cycle. I suppose 4Ghz RAM would be only 5cm. Is having an external unit with much greater than 10cm of copper travel ultimately a limiting factor here with modern day RAM speeds?

You're conflating latency vs. bandwidth.

It is entirely possible to have many bits in transit on a single channel, wired or wireless.

Note that modern RAM protocols assume that multiple bits are in the transmission line that is the wires connecting your CPU to RAM.

Electrical Engineers since at least DDR2, probably earlier, need to ensure that all lines are delay matched to about 100picoseconds.

That is, in DDR2, if DataBit#1 takes 1.1nanoseconds from start-of-wire to end-of-wire, then all other bits must be somewhere between 1.0ns to 1.2ns in length.

This requires impedance controlled pcbs from the manufacturer, and length tracking software for PCB design. (Note: advanced PCB CAD software will even recalculate the speed of light across different lengths, as "inner" tracks have more dielectric surrounding them, slowing down electricity's speed, while "outer" tracks are a bit faster)

-------

The dielectric of FR4 (the glass/resin used to make PCBs) is what determines the speed of light of the copper actually. And it can be 3.6 or 4.2 or whatever, but the PCB manufacturer will tell you in the PCB specsheets.

Copper guides the wave, but the actual wave travels in the dielectric / insulation between the wires (FR4 in modern PCBs, but if you had open-air wire the dielectric would be the open-air surrounding the wires that form the ground-loop return path). There's been a huge amount of improvements to the understanding of electricity in the past 3 decades, and the old circuit models (sometimes still taught today) are kind of obsolete btw.

-------

I think I heard that DDR4 is now 10picosecomds (10x more accuracy in delay matching), which is doable with modern CAD and PCB software. I dunno the delay requirements of DDR5.

------------------

https://en.wikipedia.org/wiki/Transmission_line

Anyway, look through this Wikipedia on Transmission Line theory, which is a closer model to how electricity "actually" works (but still isn't perfect). But its the level you need to think of electricity to understand modern CPU-to-RAM connections.

In particular, DDR2 / DDR3 / DDR4 / DDR5 connections are almost certainly either Microstrip or Stripline connections, with a fair amount of PCB / Electrical Engineering going into the design to make sure everything works as expected.

Long long time ago, when Ethernet used coax cables, the max. length of the segment was not because of the signal strength. It was because with the max. length of an Ethernet segment the first half of the ethernet data packet was on the cable, if now the computer on the other side started to send a packet too, a collision would not be detected, and the packet would be lost.

The article seems to reference having external AI etc accelerators colocated with the RAM they use, but then tied to a host machine via CXL. So that makes a certain amount of sense. Latency sensitive stuff happens co-located with the RAM, and then you just take advantage of throughput back to the host.

I'm personally now imagining a specialized database appliance which takes the role of the whole of the pager and buffer pool management from a DB (or KV store or whatever); a physical box which ties secondary storage arrays + large quantities of RAM + buffer pool mgmt firmware together on a box, then connect to host system via CXL. Host system does query planning end execution and everything else...

Is anybody doing this? Does anybody want to found a startup with me to do this? <sips more and more coffee...>

There are enterprise database offerings that do something like that with RDMA. I'm almost certain CXL is being looked at for improvements.

I just finished my master’s thesis on database buffer management with CXL [1]. It’s not published due to confidential hardware information, but we are working on a paper.

[1] https://hpi.de/rabl/teaching/master-theses/ongoing-masters-t...

Very cool. I look forward to the paper!

For some reason the whole issue of buffer mgmt is something I nerd out on a bit.

DRAM latency is tens of clock cycles. There's a minimum of about 10ns of latency happening within the DRAM chip itself, so a slightly longer bus doesn't affect total latency by much.

Wikipedia is showing a cycle time for CAS of less than 1ns for modern RAM

https://en.wikipedia.org/wiki/CAS_latency#Memory_timing_exam...

The figures under 1ns are not latency numbers but merely the duration of a clock cycle.

This is not exactly the reason, but trace length/loss is a major problem.

Laptop SODIMMs are already severely limiting DDR5 bandwidth, server/desktop DIMMs are getting close.

Users aren't gonna like it, but unswappable packaged RAM is coming to most CPUs.

What do you mean by packaged RAM and what protocol does it use that would be faster than DDR5?

> What do you mean by packaged RAM

Depends. Apple uses LPDDR5X on a cheap substrate, which is basically just RAM chips soldered to a tiny motherboard, and that is already very fast. But it could also mean HBM on an expensive interposer or cheaper Intel EMIB, or something like Samsung's proposed Wide I/O, or even something new like stacked RAM with TSVs.

CPU/RAM packaging is getting increasingly complicated.

> what protocol does it use that would be faster than DDR5

Depends, but it can just be straight up DDR5. SODIMMS are terrible because they need 1.35V (vs 1.1V stock DDR5) for really slow speeds and terrible ram timings, while soldered DDR5 and LPDDR5X do not. The closer the DDR5 gets to the CPU, the faster, more power efficient and lower latency it gets.

CXL can drastically change ratio of memory to CPU and potentially even drive down demand for latest memory gens (mixing different gens of memory via CXL). Could be an interesting shift to think in terms of designing and building large scale data storage and retrieval systems at low latency.

Dell has been pushing a new memory standard for laptops (https://news.ycombinator.com/item?id=33048948) and even there you face the realities of latency tradeoffs when physically moving the memory away from the CPU.

i would like to see whether the high memory use cases (like running LLMs) can benefit more from direct memory access to GPU that efforts such as DirectStorage is working on (https://news.ycombinator.com/item?id=33265666) and just faster, low-latency drives can bridge this gap faster.

Apple's dream come true. Sell one model with everything soldered and unupgradable. One universal computing core. Want extra? Plug it in.

From an operations point of view (Cook's forte), it makes a lot of sense.

Jony Ive would be proud as well.

Honestly I think I'd love this. Being able to augment an existing MacBook pro with a bunch of RAM or a good GPU would be very useful to me.

External GPUs working over thunderbolt is a thing. So, you can at least get part of this today.

Unfortunately not with the M series of apple CPUs :(

I really like my e gpu set up I got 6 - 7 years ago. I'd love to see a laptop dock that granted you a gpu, additional hard drive and ram when you need it.

Who's using this? What company doesn't already have a rack they can add a few more sticks of RAM in for their workloads?

If you need a lot of RAM, usually you need to buy servers with multiple CPUs to which you can attach the memory. Because the amount of DRAM you can attach to one CPU is limited.

If you don't have the need for all the extra CPUs, just being able to attach more memory to a single CPU through CXL may be cheaper.

Isn't this already possible with things like Thunderbolt, which is essentially "external PCIe"?

Yes, if you don't mind the latency and throughput restrictions of Thunderbolt. Eg. Apple's M1 has a memory throughput of 66GB/s, which is an order of magnitude more compared to Thunderbolt 40Gbit/s. So it's more like a really fast SSD instead of a local memory upgrade.

little-known?

The universe isn't Hacker News :P

But if you ask the people that would actually consider external RAM, I think you'd see a lot of awareness.

How could one consider if they don't know it was possible?

Even more so on a laptop.

This is fake news. Laptops don't support CXL.

Why buy this when you can just download more ram for free?

How much do one of these cost and is the memory persistent?

It's volatile memory.

Reply on news.ycombinator.com