Is this now a factor? I know memory has gotten faster, and multi-channel of course helps, but is that all really enough for typical workloads?
In 2005, Opterons used a 940-pin socket and had dual-channel DDR, for 6.4GB/s total bandwidth to feed two cores. In 2009, Xeons the LGA1366 socket to provide triple-channel DDR3, 32GB/s for four (later six) cores.
Now, we're up to 4094 pin sockets providing 8 channels of DDR4 for a total of 204.8GB/s of memory bandwidth for 64 cores, which is exactly the same per-core bandwidth we had in 2005.
(And for comparison, a $200 graphics card will also have around 200GB/s of memory bandwidth.)
As you can imagine, processor performance has grown a lot more than 6x over the years. The amount of cores in a package alone has increased more than 6x! So how do we get around this?
1) Larger caches: the most efficient optimization to any slow task is to simply not perform said task. Having a larger L3 cache means sending requesting fewer pages from DRAM.
2) “Hiding” latency with bandwidth: if I have to incur a (relatively) large time penalty tp go to DRAM, I better make sure to make it worth my time. Instead of always getting data in 4KiB pages, often times modern processors support “superpages” all the way up to the gigabytes. If you’re going to go drive out to the store, it makes sense to buy groceries for the week instead of just the candy bar you’re craving right now, right? That way when you actually have to cook the work has already been done.
I am probably missing some nuance here, especially when it comes to memory controllers and putting caches on DRAM, but this is the gist.
The specific implication details make a difference, but nothing is going to provide a 4x improvement in latency so other considerations take precedence. The real change is ever growing cache sizes allowing CPU’s to minimize random RAM reads.
Further, every layer of cache adds latency and cost as you need to check it before going to main memory. In the 486 and early pentium days External Cache was common, but it’s no longer useful.
“For a completely unknown memory access (AKA Random access), the relevant latency is the time to close any open row, plus the time to open the desired row, followed by the CAS latency to read data from it. Due to spatial locality, however, it is common to access several words in the same row. In this case, the CAS latency alone determines the elapsed time.”
btw 10cm of track on pcb is about 2 clock cycles at 3GHz, ~0.6ns. Now you will love this one - in the last 20 years we went from ~15ns CAS down to ... 7ns, twice as fast! Even better - we got down to 7ns in 2006 https://www.newegg.com/g-skill-2gb-240-pin-ddr2-sdram/p/N82E... and stayed there for the last 14 years.
What we got instead is higher density and susceptibility to rowhammer.
Further, MB traces are just part of the story. It’s the physical distance traveled from a CPU’s memory controller to the furthest physical cell location on a chip which is significantly more than 10cm and it’s a round trip timing. Even just the physical DIMM’s are 13 CM wide.
A more accurate ~25 cm each way = 50 cm total ~= 3ns out of 6.56ns timing for a DDR3-2133 & CAS 7 ~47%. CAS 18 DDR4-4600 is 7.82 ns or ~40%.
PS: As to progress, DDR2-1066 at CAS 4 was 7.5n where CAS 18 DDR4-4600 is 7.82 ns that’s some progress.
Page size is irrelevant for purposes of getting data from RAM to CPU / cache. This is done by cacheline size, which is usually 64 B. Higher page size helps with better TLB efficiency and also for disk to RAM copy (if that is strictly page-based).
And God said, "Let there be Cache." And there was Cache.
https://www.nextplatform.com/2017/02/23/promises-challenges-...
"AMD mentioned a latency increase of 7-8 ns when memory encryption is enabled, which results in a 1.5% performance hit in SPECInt"
Here we would be compensating with smaller data transfers. Wouldnt have to be an amazing algorithm, even RLE would probably give latency improvements.
Modern CPU cores are way faster than ones in 15 years old CPUs.
That 2015 processor has AMD K8 cores. I wasn’t able to find exact info, but slightly newer AMD K10 can do 8 single-precision FLOPs/cycle, at 2.8 GHz.
Modern Zen2 cores can do 32 FLOPs/cycle at approximately same frequency, i.e. per-core performance improved by a factor of 4.
Both numbers are theoretical limits only achieved with very specific workloads and heavy use of manual vectorization. Still, real-life general-purpose code scales quite similarly between them. CPU benchmark says their single-threaded performance improved by a factor of 3.5.
The saving grace is not that there is a lot of memory bandwidth to go around; there isn't; the saving grace is that a lot of processing either doesn't use too much data (making caches effective) or is complex enough to not be limited by memory bandwidth. Rarely using all cores and threads at the same time helps a lot as well.
Wow wow wow.
Your references are TIGHT.
----------
The main RAM / CPU issue is latency: it takes over 100-clock cycles to communicate with DDR4, which a CPU could process ~400 instructions if they were lined up just right (modern CPUs are super-scalar, executing multiple instructions per clock tick).
Only once you have sizable caches + data locality will you solve the latency problem.
Without locality, I'm not sure if the latency problem can be solved at all. Fortunately, many problems have an element of locality that can be taken advantage of.
(Anyone have pointers to current EPYC vs Xeon STREAM results?)
My experience is that it takes well structured memory access to actually max out memory bandwidth even if the numbers seem like the CPU would be starved.
Things like adding two huge images with SIMD and writing out a new image could be like this. It would be much better of course if the image is tiled and more operations are done per tile, which would use the cache.
Good use of the prefetcher but poor use of the cache could still run through memory bandwdith.