Dumb question, but why don't Intel/AMD massively increase memory bandwidth by adding more memory lanes? For example instead of the usual 4 slots on a desktop motherboard you would have 16 slots surrounding the CPU on all sides.
Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU?
On a GPU, the memory access patterns are often very wide & easily controlled, such as fetching a block of texture memory. This is like streaming reads for a disk, where you can actually just throw essentially RAID 0 at the problem to scale up really easily. So scaling all the way up to ridiculously wide 2048-bit bus widths (ex. AMD Vega 64) is completely viable since you're working with relatively bulk array data almost constantly. Now most GPUs aren't going quite that wide to hit the speeds they want, but consider that 192-bit is still considered "low-end" here and that's already a lot wider than you'll find on most CPUs.
On a CPU, especially in languages like Java, JavaScript, etc..., memory access patterns are often very random and very small. Consider just something like a simple boolean accessor call `if (foo->isEnabled()) doThing();` - you're reading let's say 8 bytes (foo pointer) to in turn access 1 byte (a boolean). Now the CPU is already reading in chunks of 64-bytes typically (cache line width), so you're already blowing 2 memory fetches to get 128-bytes of data just to read 9 bytes total. Going wider, which is how GPUs scale bandwidth, is utterly pointless here. You'd just be fetching even more data you didn't want in the first place just to occupy precious cache space on something you're probably not going to use anyway. In fact this is why DDR5 actually essentially shrinks the bus-width, from 64-bit in prior DDR generations to 2x32-bit in DDR5
a major part of JVM, JS Engines and even memory allocators, pools and similar is to make this patters less random, more predictable and more local.
For example a common optimization for lisp like languages it to layout a linked list in a continuous way in memory.
Stuff like that is needed even if you have more memory bandwidth as sometimes even more important then memory bandwidth is cache locality, which is WAY harder to scale then adding more memory channels. And branch prediction a major performance factor for modern CPUs also benefits quite a lot from cache locality.
Furthermore having more bandwidth also only helps if you can bring the bandwidth to each core as needed. But if you ad more memory bandwidth it still has a limit per memory region which can be an issue.
So in practice nearly all applications benefit from more bandwidth but the degree they do benefit is limited outside of highly paralelizable use-cases (like some graphics and AI use-case).
So for most typical consumer use cases it's not worth the additional (high) cost this would impose to various aspects (especially motherboards).
And for pro-sumer/server CPUs _we do have more channels_!
But always a balance between added cost and expected need for more bandwidth.
> In fact this is why DDR5
AFIK the reason they DDR5 has a split channel is way more complicated then that
They don't, though. The JVM memory model doesn't even really allow it. This is a big reason for value types, but that keeps getting perpetually delayed...
Not only there is sequential array in both the language, it is very often used. Things like iterating the array is one of the most common operation and is almost always memory bandwidth bound even for interpreted language.
In Java only primitive arrays are memory-efficient for iteration; ArrayLists are not because every element is boxed, so even when iterating sequentially a lookup of some random memory address is needed for every element.
I think that you are mixing ram channels with ram channel width. Channels, could be fetching totally different address at same time. On a multi core world this is very helpful. One core could being fetching a single 32 bit integer, and another core could being fetching a a struct of 128 bytes from another address.
Also, server CPUs usually have many more RAM lanes that desktop CPUs . For example, EPYC server CPU could have 12 RAM channels!
Yes, but also it's due to simple physical limitations. If you look at how close CPUs are to RAM as well as the number of PCB lines coming out of CPUs, it becomes pretty readily apparent that there's not much room for additional lines.
You have to recognize that each of those lines is creating a magnetic field, the closer you stack them, the more likely cross talk is.
We could continue to lower the voltage on the memory lanes so you can pack them closer together, but then you have to start worrying about environmental interference on the lines. It likely won't be too long in the future (if it's not already here) that we'll start seeing shielding for the memory lanes so they can be packed closer together and more can be added.
It wouldn't shock me if this became a part of DDR6.
It also wouldn't shock me to start seeing more complex memory controllers doing things like lz4 compression to eck out even higher bandwidths (particularly when the memory is highly uniform). We are sort of seeing the precursor of this in DDR5 with commands to set whole blocks to a given value.
PC desktops have 2 memory channels. EPYC servers have 12 channels per socket. It's not the physics, it's that ~8-core processors generally aren't bounded by memory bandwidth and it's not worth the cost of increasing the number of pins.
The server sockets do it because they have 128-core processors.
If this changes what we may start seeing is CPUs (or iGPUs) with a few GB of HBM as L4 cache.
no physics matter
because it makes it much harder and in turn more costly to the more channels you have to have high quality reliable connections
server RAM is for example also slower to compensate this e.g. recent EPYC processors run at up to 4800MT/s but a Ryzen 9 runs at up to 5200MT/s (without overclocking, with running at 6000MT/s isn't that uncommon)
server also tend to always use EEC memory
so it's a benefit/cost balance calculation
Zen3 EPYC has 8 channels per socket and uses the same 3200MT/s DDR4 as Zen3 desktops, so this little explains why desktops couldn't have e.g. 4 channels if there was a need for it. They already commonly have 4 DIMM slots.
The rated speed for DDR5 is 8% higher for desktops, but the additional channels don't seem to be why. Intel uses 8 channels of DDR5 rather than 12 and it's still 4800MT/s.
> so it's a benefit/cost balance calculation
But that was my point?
I'm not sure what that would add for latency and cost, but I'm guessing that with optical wave guides, crosstalk isn't a concern. At least for the optical part of the data path.
It really is insane the level at which current hardware works.
Isn't that kind of what those APUs sold by AMD are? And SoCs? I could be misunderstanding you, though.
An extreme example, the phone I am using has RAM running at 1.8Ghz but on a 16bit bus. Anything to keep costs down.
If I'm correct, this is the difference between DDR RAM and GDDR RAM - the latter has higher bandwidth but also higher latency, which is great for graphics with its high bandwidth requirements and very predictable memory access patterns (except when you do ray tracing) which allows you to hide the latency, but you don't want to use GDDR for system RAM if you value performance.
Here’s an unusual motherboard salvaged from Xbox Series X which combines AMD Zen2 CPU cores, GDDR6 memory, where the integrated GPU is disabled due to defects: https://www.tomshardware.com/news/4800s-xbox-chip-shows-us-w... The performance is not great.
GPUs are good at hiding the latency due to high degree of parallelism. GPU cores switch to another thread instead of waiting for the data. Each CPU core only runs 2 threads, it can’t quite do the same thing.
GPU memory access patterns are only great for well written compute shaders (or CUDA kernels) when the developer who wrote that shader managed to do fully coalesced loads and stores. Memory access patterns for pixel shaders are not that great, especially when doing trilinear or anisotropic sampling in these shaders.
But you can optimize around the limitation (within reason) and don't have to content with a general non-specified OS.
This is why I do wonder about the path not taken with the 1T-SRAM that powered the Gamecube/Wii. It was designed purely with low latency in mind but I don't think it scaled well.
But, more memory channels means more pins and larger sockets, and that adds cost and layout issues for motherboards. HBM ram that attaches on top of the cpu addresses the layout issue (and the bandwidth issue), but interferes with heat removal and removes expandability. Apple makes it work, but they limit clock speeds and were never big on expandability. Intel and AMD have a hard time in the market when their top consumer chips don't clock to the moon, even if the last 10% of clock speed takes 50% of the power and delivers 5% of performance (percentages made up). Server chips don't need to clock so high, so AMD can release clock limited compact cores that have about the same IPC in a bit more than half the area, and the difference in clocks for a max-cores regular zen4 and a max-cores zen4c isn't too much (L3/core is half for zen4c though, so may not be a benefit depending on your working set)
Eh. RAM generally consumes a single digit number of watts and will do so regardless of where it's physically located in the system.
If you put HBM on the CPU socket then you have to replace them together, but you can still replace them. It could also make sense to do both: If you have 16GB of HBM and 16GB of DDR5, you have 32GB of RAM and half of it is faster. Or 80GB of RAM and 16GB of it is faster. Cache hierarchies are a good way to strike a balance between performance and cost, and then the latter can still be expanded independent of the CPU.
Yes. That is why you only see more memory channels for higher CPU count.
Tradeoffs are different for server boards (many cores in the CPU) and on platforms with integrated GPUs (becoming more common) so none of this is static. I suspect in-package RAM will become more common even for platforms that still have a couple channels of slotted memory, that would create a pool of faster memory available for graphics etc while still leaving room for more if you need to have a lot of Chrome tabs open or whatever.
People who have many threaded workloads which can actually use all that memory bandwidth tend to get workstation or server platforms.
Edit: Of course, it depends on the benchmark. Latency is critical for programs that access many memory addresses randomly, not for data processing applications that scan memory linearly (and GPU use-cases often fall into that category).
given there use case of
- premium consumer only products
- which are mostly portable and have a low thermal budget
- which are by most users hardly ever run on full throttle
- a well documented tendency to proclaim any hardware failure as user fault
makes this quite viable for them
but if you run systems with 4TiB of RAM and 12k+$ CPU which is most time running full throttle potential for days at a time with a higher thermal budged and customers which might have docents or even thousands of such CPUs and will be very pushy about reclaiming any failure they think isn't their fault then having to replace over 20-50k$ just isn't viable (instead of ~1-3k...)
Or to put it another way, if I had infinite memory bandwidth, how much faster would browsing/gaming/compiling become?
I don't know about typical consumer workloads, but e.g. for physics simulations, any time you need to solve a system where the value in a point depends on a lot of other points you are bandwidth limited. Like solving the pressure equation in fluid simulations. Also I believe a lot of traditional database workloads fall into this category.
You can measure memory bandwidth with the STREAMS benchmark.
I'm not sure any of the standard Linux tools can show you memory bandwidth usage easily (maybe perf), I know we use Intel PCM (https://github.com/intel/pcm) and AMDuProfPCM (https://www.amd.com/en/developer/uprof.html)
then while probably all applications will profit
for most consumer applications you might not notice much of a difference
through you can measure some difference in benchmarks
one exceptions would be various graphic editing software if you do stuff like editing 8k multi channel video content
For compiling you might feel a noticeable difference. It won't be 8x but if you pair it with enough cores and things which have tons of parallel compilation parts it will really feel noticeable faster.
But for most gaming you instead of getting 8x the bandwidth you probably benefit more from a speed improvement e.g. running 6000MT/s DDR5.
Anyway you can get it today ;)
The M2 Ultra has a pretty insane memory bandwidth (and so had the M1 Ultra).
I had a GUS soundcard for a while and it also had an EDO slot I think.
e.g. if we speak about how many memory channels a x86 desktop, prosumer or server CPU have they count them as a single channel
main reason for this is that DDR5 didn't split the channel instead of adding a new channel
More memory bandwidth increases the price/size of the IO module on the CPU but I don't think that's a major reason.
One of the major reason AFIK is to keep the number of channels low is that it directly maps to the number of pins a socket need to has and the number of high quality lanes which need to be routed independently equally long and chided from inference from e.g. power lines to the CPU.
So especially e.g. for server design chip vendors balance the benefit of more memory channels with the cost of it increasing socket size/pins, complexity of mother boards etc.
And while on the paper doubling the memory channels doubles the memory bandwidth it doesn't at all means you RAM is twice as fast. It's similar to the difference between duplicating the number of cores vs. duplication the per-core IPC. While all applications profit from more bus width the degree might differ hugely. Oversimplified super highly parallelizeable tasks (e.g. much of GPU tasks and many AI tasks) tend to also be better at taking advantage of more bandwidth through more channels but many other taks less so. This means that the degree to which more channels will yield benefits highly depend on the task the CPU vendor expect is majorly run on it. And the tasks which most benefit from it already tend to be run on GPUs anyway.
Apples M1,M2 CPUs one the other hand have a much much smaller cost problem wrt. more memory channels due to how they stack ram on top of the CPU (I think) and they happen to also have much more memory bandwidth by having more bus width.
But stacking RAM on top of CPUs has it's own issues. Like for example if the RAM dies the whole CPU is dead and you not being able to upgrade the RAM. Doesn't matter that much for apple, but matters a lot for AMD/Intel which core focus is the server center, not desktop CPUs. Having to replace a 10k CPU and multiple TiB of RAM because a single ram segment fails just isn't acceptable. Additionally for some use cases instead of stacking RAM on the CPU you get as much or more benefit by stacking tons of L3 cache on it (AMD 3D V-cache). There also seems to be in general be more issues with thermal budges, not an issue for apple of which most devices are mostly passive cooled. Issues with manufacturing cost making it harder to provide many CPU spec+RAM amount variants (again doesn't matter for apple but does for Intel/AMD). Potential cost issues making it only suitable for "premium" products (again not an issue for Apple but for Intel/AMD). And maybe more.
So basically:
- on server (or pro-sumer,i.e. Threadripper) more channels are already used, but it's always a cost benefit balance where there could be even more channels
- on desktop, at least currently, it's not seen to be worth the price/complexity increase
Through Apples is putting a bit of pressure on the market, but outside of the high end segment it still makes little sense and it's probably not worth it to have it just for the "high end but not pro-sumer" use-case, that market it too small.
I would predict prosumer devices like Threadrippers to grow the number of memory channels with every now socket they have.
And I do predict a major technology change for "normal desktop computers" 3-6 years down the line which probably will bring more channels to desktop computers. But only then. And it's very speculative.
The new Intel Xeon Max processors have 64GiBytes of HBM in the package, as well as offboard DDR5.