Hacker News Reader

Comment by jiggawatts | Hacker News Reader

A shower-thought that popped into my brain this morning is that with ~100ns latency to main memory, that's the equivalent of "only" 10 million IOPS (per channel). I say only, because my laptop has an NVMe SSD that can do nearly 1 million IOPS.

If you have 96 CPU cores per socket, and only 6 channels per socket, that's a "mere" 625K IOPS per core. Up that to 128 cores (coming soon), then it's just 470K IOPS per core! That's worse in some sense than a laptop SSD for a single-threaded program. Not directly comparable, of course, but you can see why AMD would want to bump this number up.

For comparison, the equivalent IOPS for L2 cache is a blistering 260 million IOPS, and L1 cache is 850 million.

I see now why there's the phrase "memory is the new disk" is starting to get popular.

View on HN

> A shower-thought that popped into my brain this morning is that with ~100ns latency to main memory, that's the equivalent of "only" 10 million IOPS (per channel). I say only, because my laptop has an NVMe SSD that can do nearly 1 million IOPS.

That is not how any of this works. Neither RAM nor SSD can get anywhere near their peak throughput if accesses are serialized. If your laptop has DDR5, it can likely do > 1B iops. 100ns is the time to get a single random access to a closed row, but while you are waiting for that access to happen, you can issue new operations every 3ns to different banks and bank groups on the same channel.

I don't know how any of this works, and there seem to be a number of experts in this thread so please allow me some very newbie questions.

> [can't] get anywhere near their peak throughput if accesses are serialized

But first of all, doesn't DRAM have an optimisation for serial accesses, which is (I hope I get this the right way round) to hold the RAS steady then do CAS/CAS/CAS/CAS instead of RAS/CAS + RAS/CAS + RAS/CAS + RAS/CAS?

In addition, if you do access RAM serially then there is some interleaving which is automatically inserted so that sequential accesses at some level are sent off to different banks (but I've never been clear if there interleaved at a 64 bytes, to fit a cache line, or at a 4K page size to suit the TLB, or some other size, and I'd really like to know – anyone please?)

Sorry, two entirely different definitions of serial, and mine is less useful.

What I mean is that if you are doing an infinite pointer hop (that is, load a value, then use the that value as a pointer to load another value, use that as a a pointer... etc), you only get a tiny fraction of the IOPS you get than if you launch 100 different loads to different addresses that you already know. And the inverse of your load to use latency is basically how many IOPS you get if you do that infinite pointer chase.

> In addition, if you do access RAM serially then there is some interleaving which is automatically inserted so that sequential accesses at some level are sent off to different banks (but I've never been clear if there interleaved at a 64 bytes, to fit a cache line, or at a 4K page size to suit the TLB, or some other size, and I'd really like to know – anyone please?)

They are interleaved at the size of the DRAM row, which is usually 2^16 bits, or 8192 bytes. The reasoning here is that if you are launching a ton of linear accesses, that is, doing CAS/CAS/CAS/CAS... to the same row, you can get full throughput from it. This is because when you have opened a row, you are not reading from the DRAM anymore, the entire row is in a SRAM buffer, and can read full interface throughput from it. Then you only have to open a new row once the current one runs out, which is always in a different bank so can be done in parallel with the read operations from the current row so long as your memory prefetchers are smart enough to start doing it early enough.

Thanks that's really good, and the interleaving question's been nagging me a long time. There's a lot of depth here when you start asking. Much appreciated.

Yeah, DRAM controllers optimize for serial access in the mapping from physical address space to DRAM DIMMs, banks, rows, and columns. There's actually a pretty sophisticated algorithm for mapping "physical" addresses to DRAM addresses.

I am curious how this all maps on software. Is it kernel which will will try to do many parallel RAM reads? What is the state if it, and are there any benchmarks..

It's done almost entirely in the CPU's memory controller. The operating system only controls the page table entries, which is a "high level" concept a layer of abstraction above this.

Your NVMe SSD gets 1 million IOPs with overlapping requests. That means each new request issued does not wait for the previous one to finish. There's a queue depth, and keeping it full enough is needed to reach 1 million IOPS throughput. The latency will be higher than 1µs for each request.

RAM also does overlapping requests and has a queue depth, so that's a reasonable comparison. The headline of this article says they are standardising RAM with a throughput of 17600 million IOPS.

If you have 96 CPU cores per socket, and 6 channels per socket going to separate MRDIMMs, that's 1100 million IOPS per core. At 128 cores, 825 million IOPS per core.

So, this new RAM standard is 825-1100 times faster throughput than your laptop SSD, when counted in IOPS (without regard to the data size).

If you want to compare latency instead of throughpt, the IOPS measurement for that occurs when the sender waits for each request to complete before issuing the next one. In that case you're right that it's will be much lower IOPS per CPU core, but this also makes it lower IOPS for the SSD so that comparison still favours RAM.

Each CPU core is independent, so each core would likely wait for its own previous request before issuing the next one (this actually happens when a CPU core is following a linked list), but the cores are doing this independently of each other, in parallel.

So with 96 CPU cores per socket, each core waiting for its own request to complete before issuing the next one, and assuming ~100ns latency, that's 960 million IOPS total, and 10 million IOPS per core. With 128 CPU cores it's still 10 million IOPS per core.

Moreover, at least last time I played around with it on reasonable hardware (3rd gen EPYC), your first real limitation is your operating systems’ IO scheduler, not the hardware — at least on a “single node.” At least a year and change ago, you started slamming into a wall around 15-16M random IOPS (from your backing store). Wasn’t CPU bound. Wasn’t memory bound or a NUMA issue. Wasn’t an issue of PCIe lanes. 100% the IO scheduler as dtrace very much laid that bare.

We’re very much in a situation where the software is catching up to the hardware. For example, look at the proposed NEST Scheduler for Linux where they were seeing 10-100% (yes, as much as double!) performance improvement with a 10-15% reduction in power consumption, mostly by focusing on keeping, “hot cores hot.”

IO scheduler and process scheduler improvements will offer truly material gains in coming years, even if you still use the same high-end hardware you have today.

I love the responses in this thread. Turned out that my "Fermi estimate" was wrong, by about a factor of 100x.

A simple back-of-the-napkin maths is that for a system with 30GB/s memory bandwidth and 32-byte cache lines works out to about a billion I/O operations per second. This would be a typical laptop system, or the like.

However, 100ns per read is still a valid scenario for a single-threaded program "chasing pointers" as in a linked-list.

I think it's still fair to say that an un-optimised single-threaded program accessing memory randomly is only 10x faster than a parallel program efficiently utilising a modern NVMe SSD.

The corollary is that there is (still) no scenario where an SSD would "beat" the performance of RAM since the worst-case for RAM still beats the best-case for SSD.

People don't seem to understand that IOPS is a meaningless measure from a practical perspective.

It is just a weird way to express throughput. An SSD that loads 4KB at a time has 64 times less IOPS than RAM that loads 64 bytes at a time assuming both have the same bandwidth.

The average fetch is not that big. IOPS and GB/s are both very important numbers.

From a practical perspective, the IOPS of RAM is critical for running programs. And if you're doing anything other than big file transfers, an SSD plugged into USB 2.0 at 35MB/s will beat a hard drive on USB 3.0 at 200MB/s.

Even better is IOPS at queue depth 1, IOPS at queue depth 16 or 32, and max bandwidth.

> with ~100ns latency to main memory, that's the equivalent of "only" 10 million IOPS (per channel)

I think you're mixing up latency and throughput. IOPS is throughput, and any single CPU core can have several RAM accesses in flight, not just one. So with DDR4-3200 you have more than 3 billion "IOPS" per channel, not 10 million as in your calculation.

Not quite so many. Each memory operation on modern CPUs is a minimum of 64 bytes, so on DDR4-3200 if you saturate a single 64-bit channel, you get 400 million IOPS.

On DDR5, channels are 32 bits wide, but there are two in a DIMM, so a normal desktop system is actually "quad-channel", with each channel doing up to that 400 million IOPS if the ram speed is DDR5-6400.

In practice, it's of course rare to completely saturate channels, just because of bank conflicts and refreshes, etc.

IOPS are not throughput at all... throughput can't even be calculated until you factor in the size of the I/O request.

You can have 1M IOPS at 5MB/sec if your blocks of data are small enough, like 8k... vs 100k IOPS/sec at 1GB/sec throughput with 1MB request sizes...

In this case 100k IOPS are getting you more throughput than your 1M IOPS.

> IOPS are not throughput at all

Yes, it is. You're thinking of "bandwidth"; the terms "throughput" and "bandwidth" are not interchangeable. IOPS and bandwidth are two different forms of throughput: the former has number of operations in the numerator, the latter has bytes.

While it’s true that if you were e.g. traversing a random linked list doing pointer chasing, that might be true, it should be noted that the effective number of operations is significantly improved by the prefetcher. For example, on an i7 9700k, memcpy can copy at 13.7GB/sec, which is probably 400M operations in practice.

Today, compute speed is roughly 5-6 orders of magnitude faster (single threaded) than it was 30 years ago. Today, RAM is roughly 5-6 orders of magnitude faster than HDD was 30 years ago.

All the old-school HDD-optimization techniques from traditional database programming, etc. are now fully relevant again for memory management.

> Today, compute speed is roughly 5-6 orders of magnitude faster (single threaded) than it was 30 years ago

Wat? No it's not. It's closer to 5x-6x than it is to 5 orders of magnitude if you're looking at instructions the CPU normally executes.

> All the old-school HDD-optimization techniques from traditional database programming, etc. are now fully relevant again for memory management.

Those techniques were mainly managing the fact that it took 100s of milliseconds to random seek (like, human perceptible times to do one seek) so the algorithms tried hard to minimize that. It doesn't have much bearing on RAM with something like 100ns random access.

> It's closer to 5x-6x than it is to 5 orders of magnitude

1993 was when the original 60-66Mhz Pentium first shipped; 5 orders of magnitude may be an exaggeration, but every step since then has had of increases beyond just clock speed, and clock speed alone has increased nearly two orders of magnitude.

Memory latency went down mere 2x between SDR and DDR around 2000, and then stayed flat for the last 20 years:

https://en.wikipedia.org/wiki/CAS_latency#Memory_timing_exam...

> Memory latency

The widening gap between memory latency vs speed of processing in the processor (and how that arguably makes access optinizations that used to make sense between processor and HDD sensible between processor and main memory) is the issue that was being discussed, so memory latency can't be counted as part of processor speed in that context.

And I still reckon that it's closer to 5-6x than it is to 5or6 orders of magnitude.

> And I still reckon that it’s closer to 5-6x than it is to 5or6 orders of magnitude.

The geometric midpoint (the only one that makes sense here) between the high end of 5-6x and the low end of 5-6 orders of magnitude is about 774×.

The best info I can find (unsurprisingly, direct comparisons of performance of long-separated-in-time processors in aggregate is hard to find) is that between 1995-2011, integer performance increased somewhere well over 128 times but less than 256 times and floating point significantly more than 256x, and was trending in the last several years of the period to increase at a pace of 21% per year for both. [0] Seems likely to be close to or significantly past the 774× level between 1993-2023.

[0] https://preshing.com/20120208/a-look-back-at-single-threaded...

The access patterns are very different which makes old optimizations less useful.

For example HDD latency was very sequential with the heads stacked on top of each other. Meanwhile modern memory has has latency for a single request but you can make several requests at the same time and latency doesn’t stack linearly.

The comparison is about speed, not total capacity. Sure, memory hasn't scaled like that. But in terms of CPU stalling, it's effectively the same as disk was.

I edited the point a few times trying to get the idea across, but the point was one of access patterns.

Modern RAM allows paralleled access where HDD effectively only had a single read/wrote head. People also on average do a lot more computation on any given bit of memory.

> All the old-school HDD-optimization techniques

Kind of. With memory, at least, we don't need to do elevator seeks.

Each block of CPUs have it's own controller so your math is already wrong.

But yeah, random access to memory is bad, you can only get to the max performance if each thread requests for somewhat contiguous memory blocks

> Each block of CPUs have it's own controller so your math is already wrong.

Wrong how? The internal structure doesn't matter very much. Channels per socket and cores per socket, to calculate channels per core, is the correct math.

NVME drives made for PCIe gen 4 and 5 use DRAM to get their high burst read and write rates, apart from optane devices. Most in gen 3 did as well, the 970pro being an exception.

I don't think this is accurate, because the memory protocol doesn't require you to wait for each IOP, it's much more complicated than that.

That's why there's parallel banks and parallel bank-groups.

Any _particular_ RAS + CAS to read a 64-burst will be ~100ns latency. But you can perform 16 of them in parallel on DDR4 (probably 32?? in parallel on DDR5?).

That is: Bank-group#0-Bank#0 RAS -> Bank-group#1-Bank#0 RAS -> ... Bank-group#4-Bank#0 RAS -> Bank-group#0-Bank#0 RAS ... Bank-group#4-Bank#4 RAS -> Bank-group#0-Bank#0 CAS (finally finish the 1st read/write)... etc. etc.

DDR4 sticks are designed to only be fully utilized when you have 16+ concurrent RAS/CAS operations going all at once.

-------------

If you have two sticks on a line of DDR4, you have a "rank" as well. So Rank#0-Bankgroup#0-Bank#0 -> Rank#1-Bankgroup#0-Bank#0... (etc. etc.) 32x in parallel (alternating between Stick#0 and Stick#1).

DDR4 sticks already can be multi-rank. So this MRDIMM seems to be another layer of parallelism (and DDR5 upgrade was probably another layer of parallelism again, because 100ns latency doesn't seem to be improved no matter how many years go by).

Note that DDR5 doubled the number of channels per module. That compensates for a lot of CPU growth, in addition to those big CPUs getting more channels over the last few years.

Intel currently has a max of 8 channels (for 60 cores) and AMD 12 (for 96 cores) on the server side.

Not to discount what you've said but it's slightly better :p

multiple socket multi-cpus solve this, right?

Reply on news.ycombinator.com