AMD’s 7950X3D: Zen 4 Gets VCache
chipsandcheese.com
chipsandcheese.com
I don't find this unfortunate. Engineering is compromise and being able to make things that do a particular thing well can get you more performance per $ and per watt than you might otherwise see. The whole GPU thing that kneecapped Intel[1] is an example I use of how a compute element optimized for one thing can boost overall system performance.
I have worked on a number of bits of code for "big/little" ARM chips and while it does make scheduling more complex, overall we've been able to deliver more capability per $ and per watt. That is perhaps more important in portable systems but it works in data centers too.
I had an interesting discussion inside Google about storage and whether or not using a full up motherboard for GFS nodes was ideal. The prevailing argument at the time was uniformity of "nodes" meant everything was software, nodes could be swapped at will and you could write software to do what ever you needed. But when it comes to efficiency, which is to say what percentage of available resource is used to deliver the services, you found a lost of wasted watts/cpu cycles. All depends on what part of the engineering design space you are trying to optimize.
[1] The "kneecapped" situation is that since the PC/AT (80286) on, Intel generally made the most margin of all the chips that went into a complete system. Now that it is often NVIDIA.
[1] "Jim: Here it is. The concept is that the speed of light is finite and a nanosecond is a foot. So if you buy a gigahertz processor, it’s doing something every nanosecond. That’s the event horizon. But that’s in a vacuum, the processor is not a vacuum, and signals don’t go in a straight line and the processor is running at 3 gigahertz. So you don’t have a foot. You’ve got four inches. And the speed of light in a solid is less than that. So this is the event horizon. If something happens on one side of this thing, the clock is going to tick before to the signal gets to the other side. That’s why processors of the future have to be small and in fact golf ball– size. Why are they smoking? Well, because they have to run on a lot of electricity. The way you get things to go fast is you put a lot of power into them. So heat dissipation is a big problem. Now it’s astonishing to me that Intel has decided that this is a big problem only recently, because people knew that we were headed towards this heat cliff a long time ago. And why is it hairy? Because you’ve got to get signals in and out of it, so this thing is going to be wrapped in pins." -- https://amturing.acm.org/pdf/GrayTuringTranscript.pdf
We are now definitely near the end of the S curve. May be Another 10 years of node advance and small incremental IPC progress. I often wonder, what's next. Or are we... nearly done and its all software?
With silicon based physical chips, I imagine this is true. I would guess, like some speculative science fiction, that the next leap in performance would be an entirely new concept. Who knows what is possible?
The more we go 3d on compute we have to cut ghz even more to reduce power, and that puts more burden on the programmer. Which we'll probably do some day.
Can someone who knows better than me please comment on the Linux server scheduling issues with a CPU like this.
At this stage I’m assuming I’d be better with a CPU with all cores the same.
Its a bit of an unnecessary hassle and annoyance, though. I'm pretty sure all the V-Cache EPYCs will have cache on all their dies so there won't be this issue.
Evidence?
For example if task previously running on core is waiting for something, the choice is to allow something else to run there (and hope it finished before that other thread comes back from waiting), or not use it at all and waste CPU capacity.
Obviously scheduler will choose to use that now-idle core, but that might cause the previous process (which scheduler have no idea when it will come back) to be resheduled somewhere else, usually on its sibling (as in closest in memory hierarchy) core.
Or power saving. If say core was idle (thread was waiting on something), it might actually be quicker to run it on other core (especially one close in cache hierarchy) than to wake up the core and then ramp up the clock speed of it.
Here is some interesting if a bit outdated info about the nitty gritty: https://www.usenix.org/system/files/login/articles/login_win...
There's definitely a lot of iteration & growing in here we could do to get better here. Yet... it feels like searching for problems to worry about some tasks running faster than they would have before (but not all tasks). In most deployment models, it should be fine. I think, for now, this slightly favors two use cases: one big monolith running on all cores, which will let work get consumed as it's ready. Or divide your workers onto the different chiplets, and use a good load balancing strategy that is somehow utilization aware.
Personally I'm lacking in Fear, Uncertainty or Doubt over this making things worse.
The advantage of the dual CCD design is that this only/mostly hurts in highly threaded workloads at least.
Side note: Most of those high end audio interfaces are now external devices connected via USB (formerly also via Firewire).
And since a DAW typically doesn’t need high end video, I didn’t bother with a separate GPU on my latest DAW build. I will only add a GPU card, if I ever want to do higher end gaming, video production or machine learning on that computer.
Pairing a 4090 + a CPU on the same chip/side by side would be a stretch, but it has nothing to do with physical size - more the motherboard needing to handle power delivery, and the separate memory not being fast enough. Something like a 3060 is probably more reasonable.
This already exists too - the PS5 and Xbox Series X/S both use APUs with much more powerful GPUs than are available as standalone purchases.
4090 have die area of ~600mm2
Ryzen 7 have 71mm2 CPU die and 122mm2 IO die
It's not even fucking comparable
> CPUs would be the same size if they had all that stuff on their own discrete board too
Oh, oh, I know, we can call the board that has all the power and IO CPU needs a "motherboard" cos it cradles the CPU in its hold (socket). Oh, wait, that's exactly how it is.
> Epyc 9654 has 12xCCD
That's $7k CPU you're comparing to $1600 GPU
The 9354P is $2,730 and still has 8 CCD. That's >60% more than the 4090, but it's also >60% bigger, because Epyc has a ~400mm2 I/O die on top of the CCDs.
You're comparing a ~$350 CPU to a ~$1600 GPU.
Intel and Ryzens consumer chips both follow the same marketing formula:
> Celeron/Athlon: ultra low end
> Ryzen 3/i3: low end
> Ryzen 5/i5: mid range
> Ryzen 7/i7: high end
> Ryzen 9/i9: I have money coming out of my ears and want a bigger number than everyone else.
CPUs in general are cheaper than GPUs - spending the same amount on the CPU as the GPU has almost never been the right move.
The fact that you have to move out of the consumer product range and into the workstation stuff to even find CPUs that cost as much as high end consumer GPUs should make that clear - the GPU equivalent in that space would be stuff like Quadros or RTX AXXX, which just like threadrippers and epycs are worse for gaming because they focus on memory and stability rather than raw power.
The pricing for Ryzen 7 and Ryzen 9 overlap. The Ryzen 9 7900 costs less than the Ryzen 7 7800X3D. And Ryzen 9 has better performance per dollar than Ryzen 7 on threaded workloads. The demarcation for "I have money coming out of my ears and want a bigger number than everyone else" is where dollars start going up faster than performance, which is unambiguously Threadripper/HEDT.
But Threadripper actually is significantly faster for the kind AI workload that would otherwise be done on a GPU, or for which a fast iGPU would make sense, because it has traditinally had twice as many memory channels. If Zen 4 Threadripper ends up on Socket SP6, it will have three times as many memory channels as Socket AM5.
Zen 4-based Ryzen 3 doesn't exist and there hasn't been a new Zen-based Athlon for desktop since Zen+ three years ago.
I'm not really sure what you're arguing so aggressively about - are you saying it's impossible to pair a CPU + a high end GPU on a single board? Because as I mentioned this already exists in the PS5 and XBSX, they could sell this in the PC space if they wanted to.
The problem is the existing iGPUs are unnecessarily anemic. Okay, the Radeon RX 6950 XT has 80 CU at 335W. You can't fit that in Socket AM5. But the Radeon RX 6400 has 12 CU at 53W. You could add that to a Ryzen 7 7700X (105W) and still have a lower TDP than a Ryzen 9 7900X (170W). You could double that and add it to a Ryzen 7 7700 (65W) and still be under 170W. You could even add that to the 7950X with the caveat that it has the base clock of the 7945HX when the iGPU is under load.
But the Ryzen 7000 series iGPU doesn't have 24 CU, or 12, it has 2. Intel's have been even slower.
The question is, why bother? 2 CU is plenty to run a monitor if you're not doing anything that benefits from a fast GPU. If you are, 12 CU is still on the low side.
But AI could change things, because then even people who don't play games will have a use for that kind of hardware. Professionals will still pay thousands of dollars for a big hot discrete GPU but there is now a case for having ten times as many CU in every business PC.
It's also starting to make more sense to put the CPU and GPU together because of unified memory. The full version of LLaMA requires ~122GB of RAM. That's petty cash in PC memory but a capital expenditure for GPUs. If AMD wanted to make a big splash, they'd drop a new Threadripper with a high TDP socket, a ton of memory channels, support for RDIMMs and a fast iGPU.
This is the thing that may finally unseat Nvidia on AI, because if every CPU from Intel or AMD starts to include an iGPU that can do AI as fast as a $300+ Nvidia discrete GPU, people are going to make the code run on them.
There's no point going much wider on iGPUs, because they're all throttled by DDR4/DDR5 anyway. The exception: PS5 and XBox both have GDDR (graphics RAM) which is 10x more bandwidth than your typical DDR4/DDR5. Since they are no longer memory-bandwidth constrained, they can go much bigger and remain practical.
DDR5 is something like 50% of the bandwidth of GDDR6. DDR5 up to 8 GT/s, GDDR6 in the range of 14-21 GT/s. For example, the Radeon RX 6400 has GDDR6 at 16 GT/s. But it also has a 64-bit memory bus, compared to 128-bit for Socket AM5. So they have the same memory bandwidth, given the fastest available DDR5.
Higher end GPUs have a wider memory bus than that, but those don't fit in the CPU socket's TDP anyway. It wouldn't prevent the iGPU from having 12+ CU instead of 2.
Moreover, higher end CPUs have a wider memory bus than that too. Socket SP5 has 12 memory channels. And a max TDP of 360W. That's as much memory bandwidth as many midrange discrete GPUs, especially anything that would fit in its TDP with a reasonable power budget left to the CPU.
It's mostly bus width, not the memory technology that makes a big difference.
So basically this is why Intel/AMD iGPUs stink, and why Apple's do not. Apple give you a choice of 100, 200, and 400GB/sec in a laptop and 400 or 800GB/sec in a desktop.
And there are GPUs, like the RX 6900 XT, that have more CUs per GB/s than the RX 6400. At the same ratio you could have 20 CU with the bandwidth of dual channel DDR5-8000, which will no doubt become more common and less expensive as socket AM5 matures. So the problem isn't the memory bandwidth, it's that the iGPU could be bigger.
And nothing stops them from making arbitrarily much memory bandwidth available even in existing sockets by including a few GB of HBM on the CPU package. That's essentially what Apple is doing to the exclusion of even having separate memory, but that's not optimal either. Now you can get a Macbook Pro with 96GB of very fast memory (if you want to pay $4000+), but then it can't run the full ~122GB LLaMA model which anyone can do with ~$200 worth of DDR4 in an ordinary desktop PC.
The right answer is to provide a few GB of HBM on the CPU package which can be used as an L4 cache and to provide the bandwidth needed for a fast iGPU, then have memory slots to add an arbitrary amount of ordinary memory so that anything that won't fit there can run at lower speed, instead of not at all. That isn't on offer yet, but it's better, and it's possible, so give it a minute.
Frustratingly the PS5 and XboxX both had pretty nice memory systems for CPU+IGPU, but during the entire GPU shortage AMD never brought it to market for PCs/ laptops. Although there is hope. The AMD Strix Halo, due in 2nd half 2024 does claim to have a 256 bit wide memory bus.
Sure a fancy hybrid that uses a few fast stacks of HBM and some slower dimm slots is possible, but it's far from sure if such a design would hit the size, power, performance, and cost constraints of the market. Clearly the Apple 400GB/sec apple solution fits in a thin/light laptop with pretty good battery life.
> Sure a fancy hybrid that uses a few fast stacks of HBM and some slower dimm slots is possible, but it's far from sure if such a design would hit the size, power, performance, and cost constraints of the market.
Slower memory costs less and uses less power than higher bandwidth configurations. Moreover, if you offered various CPUs with e.g. 8-32GB of HBM, systems that needed no more memory than that could leave their DIMM slots empty. But then systems with e.g. 16GB of HBM and 64GB of ordinary RAM would be possible, resulting in 80GB of usable memory the most active 16GB of which has quadruple the bandwidth or more. For a much lower price than 80GB of HBM and similar performance for anything with a <16GB working set.
High bandwidth, High capacity, and cheap ... pick 2.
If you need memory bandwidth for more then 64GB Apple is unmatched at anywhere close to the price. Apple has a unique combination of GPU like memory bandwidth, without the GPU like memory limits.
> You could add a $1000 discrete GPU to that and have money left over.
Yes, but you'd also have 1/8th the CPU memory bandwidth and a max GPU ram of 16-20GB.
> [ proposed hybrid HBM + dram]
Possible, but it's a large complex solution that will significantly increase hardware + development costs for a niche market, even if you leave the dimms empty. Trick is will the HBM actually improve performance? Say you have a 128GB LLM, how much will adding 16GB of HBM improve performance? If the fast memory (HBM) is 16GB it's going to be very tempting to just use a GPU.
> 128GB of RAM, it isn't available
M1 studio ultra has a 128GB config for $4,800. It hasn't been refreshed for the M2 yet, but the gen2 is 24GB (m2), 48GB (pro), and 96gb (max). Stands to reason the m2 ultra will have twice the ram of the m2 max, just like the m1 ultra allows double the m1 max. So likely 192GB real soon. Rumors claim a release at WWDC in June, recent OS updates mention 3 new desktop models.
800GB/sec to 192GB ram is quite unique in today's market. There are some similar solutions like the Arm base A64FX which has multiple stacks of HBM memory, but a max ram of 32GB. As you might imagine they aren't cheap either.
Eh, it's wasted silicon for most that will be buying those CPUs.
I'd rather see separate chipset that integrates GPU for those that need it and drop few bucks off the CPU price.
Scheduling for this kind of chip is not super, but it might improve. Meanwhile, you can enjoy almost all of the performance for the specific workloads where it matters by doing this.
The scheduler was more or less designed around symmetric multi processing. BIG/little asymmetric systems will still have obviously preferred cores; if you're optimizing for throughput, add one task to each fast core first, then to each slow core, then maybe move tasks around to fit policies, etc.
With the 7900X3D and the 7950X3D, it's trickier, because one chiplet has a lower clock speed but more cache. Tasks that fit into the smaller cache will do better on the less cache chiplet, and tasks that fit into the larger cache but not the smaller cache will do better on the larger cache chiplet, and tasks that don't fit into either task will probably do better on the faster chiplet, but it kind of depends. In order to make good decisions, the scheduler would need more information about the task's memory access patterns, and I don't think that's something schedulers tend to keep track of; but if this type of chip is common in the future, it will need to happen.
For Epyc, AMD's server processor line, I believe the plan for X3D is to add cache to all the chiplets, keeping them roughly symmetric; there's still the modern situation that some cores will boost higher than others, and moving tasks to different cores can be very expensive if the process has memory still resident in cache on the old core, and the new core doesn't share that cache, etc.
... but overall it would be far easier to just buy 5 faster machines and 10 slower ones instead of having 15 "half fast half slow" and fuck around with scheduling
The aforementioned 7950X3D is a very weird proposition. 7800X3D (8 cores all with spicy cache) have game performance that's near the same in most games so I guess value proposition is "you can just run other workload" (streaming) on other ones but that's not many people's use case...
It'd be interesting to learn if / how the results differ depending on compression implementation.
I'd expect some kind of predictive AI that analyzes thread behavior to be able to help. But not sure if anyone tried making a scheduler like that.
You can get pretty good heuristics by looking at graphics API usage.
On Linux you could just put the appropriate taskset or numactl command in your game shortcut, it's pretty easy.
I guess the hard part would be distinguishing between "a video game getting inputs" and "discord app getting push to talk key", but at the moment you moved all of the noninteractive system stuff to slow cores and only "things using keyboard" on fast cores it's already pretty good approximation.
> While the build times were similar between the 7950X and 7950X3D, with the latter the CPU power consumption was significantly lower.
Just pretend it’s 2013 and you got a chip with some nice overclocking potential you don’t really want to use because you don’t like heat or the noise it brings.
L1 cache is the fastest (and closest to the CPU), but there’s very few of it. There’s also L2, L3 and L4, each being slightly slower but larger than the previous one.
VCache is a new kind of mechanism where the caching layer is placed vertically “on top” of the CPU. This unlocks new possibilities, and possibly more cache available at lower access times, but there are some technical challenges to make it work.
It will enable your database to perform computations faster, as more data can be stored closer to the CPU.
SELECT * FROM t1 WHERE (SELECT * FROM t1 WHERE (SELECT * FROM t1....
Databases usually do a few (say a compare, maybe bit spicy compare) on big set of data