Comment by formerly_proven | Hacker News Reader

Comment by formerly_proven | Hacker News Reader

formerly_proven

The buried lede of these is that they have asymmetric L3 cache. One CCD gets the 3D V-cache stacked on top, the other doesn't. That's almost certainly how AMD managed to solve the thermal and clock issues of the 5800X3D - they didn't. The 3D CCD will run hot (at low power) with lower clocks and the non-3D CCD is responsible for reaching the advertised boost clocks.

Note how the 7800X3D, a single CCD SKU, has an advertised maximum clock of 5 GHz, compared to the 5.6/5.7 of the two dual CCD SKUs (the 7700X below it advertises 5.4 GHz). Don't expect the 3D cached-cores to exceed 5 GHz on those other parts.

This means that the performance profile of the two CCDs is very different. Not "P vs E core" different, but still significant. If you run (most) games, you want to put all their threads on the first CCD. If you run (most other) lightly threaded workloads, you'd often want the other, non-3D CCD.

This seems like a rather significant scheduling nightmare to me.

> This seems like a rather significant scheduling nightmare to me.

It's way easier then big.LITTLE or current Intel CPUs with E and P cores, so it should be a non issue.

It's only a relevant significance when you want to maximize performance but most desktop tasks, including many games, do already perform to a degree where you are unlikely to notice the difference without profiling or being very very sensitive.

But then for gaming having more then 8 performance cores is currently useless for most games (assuming you don't do anything "costly" in parallel). It's to a point that for many gaming benchmarks disabling one CCD can lead to slightly better results on a 7950X (due to more thermal and power budged for the other CCD).

Or in other words if you only care about gaming don't go for a 2 CCD CPU it's not worth the money (similar for Intel go for 8 P-cores but going for more E-cores has most times dimishing returns AFIK)

And if you do other things:

On windows AMD will work with windows to make games run nicely.

On Linux it anyway can be a good idea to pin you game to one CCD manually (even for the 7950X; using cgroups, e.g. through systemd). As this should implicitly lead to other services being run on the other CCD (simple due to the first already being utilized highly).

Now highly scalar applications which use both CCDs fully could have some interesting dynamics I guess. But again it should be much more predictable then doing similar across P & E cores.

But there is one fun use-case where this looks awesome for: Embedding a gaming VM in you system. (Dedicate all of CCD1 to the VM, use IOMMU for GPU and NVME pass through).

Good insights here! I wonder how this would translate to a dual-KVM setup, where one KVM is Windows (for gaming) and another is Linux (all supervised by e.g. Proxmox).

If we don't do core pinning, would the "whole CPU passthrough" be sufficient to preserve the scheduler logic in Windows (and any such logic in Linux, if it is or will be available in the near future)?

Alternatively, as you suggested, one could feasibly allocate one CCD entirely to Windows and another to Linux, to optimize each one for performance in gaming and productivity correspondingly. Though for heavy multi-core tasks, passing the whole CPU to Linux would still be preferable, I think; same for Windows if playing strategy games. So this approach might not suit all cases equally well.

400Mhz off a 5.4Ghz clock is a little less than a 10% drop. While it'd be a shame if the scheduler used the "wrong" core, if that's the kind of price for a misjudgement, I don't expect it to be all that impactful in practice. IIRC published results from the 5800x3d also showed that quite a few workloads did have a small benefit from the larger l3 - not enough to make it worth it financially, but enough to roughly compensate for the lost frequency. The set of workloads where you'll see the full proportional slowdown due to the clockfrequency, with zero compensation from increased L3 is likely to be fairly small.

All in all, this sounds more like a technically really interesting challenge, rather than a high-stakes bet. Even a "bad" scheduler probably won't do much harm.

Realistic worst case, quite a few workloads that might have benefited from the new cache don't... but probably aren't hurt much either.

Its interesting to note it will make benchmarking harder. Any benchmark could be 10% faster or slower based on which core your code is running on.

On linux you can use 'taskset 0x1 ./bench' to force a process onto a specific mask of cores. Sounds like this sort of thing will be increasingly necessary if you want to compare code variants on these CPUs with precision.

Good benchmarks are often concerned with core to core latency due to the physical topology of the cores. I imagine we’ll see some expansion into which core complex is in use for a given test and comparing both of them.

If the change causes the scheduler to put it on the cache heavy cores but it turns out the program runs better on the clock heavy cores then this will cause your benchmark to report things are better than ever when the change really did make it perform worse. If pinning really is the best way to get the application to perform well and you want to implement that then the application or environment should be doing the pinning so the benchmark runner can report what's actually happening not what could happen.

Regardless though if you want reliable numbers, even on a CPU with uniform cores, you should be doing multiple runs and comparing average and variance. This will always tell the actual story regardless how the platform underneath behaves that day.

taskset -c to be pedantic

AMD said their drivers/software should handle it. Basically working with Microsoft to get the Windows scheduler make the correct decision. No clue how it would work with Linux though.

Intel did similar work with E/P cores.

> Intel did similar work with E/P cores.

Indeed, that's Lakefield as seen in the Intel Core i5-L16G7.

The Scheduler changes (in Windows 11 at least) make it extremely pleasant to use on the Lenovo X1 Fold, even if there're some drivers problems (fortunately easy to fix: https://csdvrx.github.io/ )

> (in Windows 11 at least)

That seems to answer my (sibling to yours) comments question. Perhaps the 7800x3d is the only sku unbothered by older OSs, then.

I was thinking of buying a 5800X3D for my gaming rig (as it is already AM4). Am I reading correctly that I might need to upgrade to Windows 11 to get the full benefits of it?

5800x3d should work fine on everything. The 7800x3d should also work fine. It’s the 7900/7950 that may not run well cos of the 3d stacking only on 1 ccd. So one has it and one doesn’t and that may cause issues.

No, only the 7950x3d and 7900x3d from AMD have this property of asymmetrical cores. This is a new thing for AMD. For the 5800x3d, or the 7800x3d, it's only 1 CCD so there's no asymmetry, and no scheduler accomodations are needed. Likewise, any non-x3d chips are symmetrical.

Any idea if this is a win 11 thing? I've got a 7700x currently in part because I didn't want to run win 11 in order to support the big.little arch of Intel's latest. When the 12900k and friends came out I recall reading that just win 11 was getting the scheduling updates to support it well.

Even now I don’t think it’s supported well. I still get weird issues with a 12700 but non big.little works more consistently.

>This seems like a rather significant scheduling nightmare to me.

I look at this as Zen 5 test bed. Microsoft already said scheduler tweaks will come in Win 11 to better accommodate Zen 4 X3D, and I assume Linux kernel devs have been prepping for a bit.

I agree that Intel is already having to play with optimizing BIG.little with Alder Lake and Raptor Lake. Functionally this is kind of the same for AMD, where you want that cache hungry application all on the same core complex. Not exactly apples to apples, but it's at least still fruit.

This is a good opportunity to get a year or more of battle testing this stuff in the field to ensure its at least up to snuff, as well as plan future iterative improvements.

I imagine Linus himself wanting the 7950X3D, given how he raved about 5950X earlier.

Having a 5950X myself I am very impressed at its power efficiency. I've tweaked mine a fair bit (plus aggressive memory with manually adjusted timings) and it's been great. Up to 6-cores it can sit at 5GHz all day long (on air!), and for full core workloads with the aggressive memory it's basically on par with a stock CPU if it was 18.5 cores, and is silent in all but the goofiest long-running workloads. I'd kill for more PCIe lanes, but for a budget data processing workstation (40Gbps networking, GPU as a budget DPU), it's been great.

I believe the 7950X3D will have its voltage locked, but in terms of perf/W should still improve over the 5950X. And the 5950X's power efficiency is fantastic.

The Ryzen Pro processors are also fantastic. The 5750GE was a decent clocked 8-core that peaked out of the box at only around 39W, supports ECC and DASH, other nice features like Memory Guard, Shadow Stack, etc. Those are great in 1L form factors where you can build capable physical clusters where nodes can idle at only 9W total power draw.

Yeah, impressive all around!

Modern schedulers like Window's multilevel feedback queue and Linux's CFS (Completely Fair Scheduler) have advanced a lot in the last decade.

It would be annoying for kernal developers to make an exception for individual processors like this, but it's totally doable to make the scheduler handle this case.

Unlike P/E cores you can't handle this generically, because the performance delta is workload-dependent. For E cores you can more or less just punt "background tasks" on them and pull tasks that are using a significant amount of CPU to the P cores.

With X3D vs non-X3D cores which one is going to be faster depends on the workload. Most games benefit (sometimes drastically so), a lot of other stuff doesn't / is slower roughly in proportion to the clock reduction - just look at the 5800X and 5800X3D of the prior generation to see how split the benchmarks are; virtually every synthetic benchmark and most intensive things - compiling, encoding, running JS and other interpeters - suffer, games win.

I suppose a good heuristic for desktop users would be to pin processes using the GPU to the X3D cores.

I don't think it's quite as bad as you're saying, because reaching to the other chiplet for L3 isn't terrible in latency compared to hitting "close" L3. So probably the higher boost cores win overall in either case.

This is a good point. If the cores in the non-stacked CCD miss in local L3, it's still quicker to ask the stacked CCD instead of going to RAM. But GP's question is valid, how do you prefer the non-stacked CCD for certain high-intensity tasks without hardcoding their names/IDs?

I'd love to see a benchmark with the 3d vcache ccd cores disabled and benched against the non 3d part it is based on with one ccd disabled.

Would be interesting to see at comparable clocks what the performance uplift of hitting the huge cache more often on the other ccd vs hitting main memory.

Can it actually hit the cache on the other chiplet? I thought that Zen's L3 caches were somewhat private to the local CCX/CCD, such that threads running on one chiplet have no way to cause new data to be prefetched or spilled in to someone else's L3.

My assumption was the IO die had a directory of cache lines and would route a request to the other CCD if it were present there. You can't evict to a remote L3 but you can snoop it, I think.

Yes, this is correct...

Now, thinking about it... you can't evict to the distant L3, though. So, it really depends on the workload and whether the remote big L3 is "warmed" suitably for you.

Yea... Bummer about the hybrid layout, voltage control will be disbled anyway to prevent killing the 3D die so I don't really care for high boost CCD. If it weren't for the EDC VID bug on Zen 3 taming PBO, I'd be running a manual OC to keep high load voltage and temps in check. Might as well have two tamed 3D CCDs.

So one might look at the stacked cores as extravagant "L4" controllers serving their faster peers that might occasionally throw in some compute of their own for particularly parallel workloads? Seems rather wasteful, but perhaps it's actually less bad than the performance per watt tradeoffs at high clock rates.

> faster depends on the workload.

but also for most daily desktop workloads the difference is irrelevant tbh.

games already should be pinned to one CCD for max performance

compilation and similar will optimally spawn across both CCDs making it not matter (through you want to only split parallel computation units across CCDs but parallel code gen for the same unit should be on the same CCD)

So I don't thinks it matters outside of artificial "challenges" to try to maximize performance beyond what is normally reasonable for a desktop system (due to work/time involved in doing so).

Can the CPU cores in a CCD access the L3 cache of another CCD with higher latency? If so the CCD without extra cache may still get a performance boost.

I know there has been such designs in the past but I don't know how it works in the Ryzen CPUs.

Speed of cache between CCDs has always been much worse than within one CCD.

At the same time, that latency is still peanuts compared to hitting main RAM.

The die with the cache probably has better latency (provided the cache doesn’t connect through the IO die), but lower clocks making it better with memory limited workloads.

The other die will be better at non memory bound work, but should still be much better than normal at memory bound tasks too. I suppose it remains to be seen if lower latency and lower clocks beats higher latency and higher clocks, but I suspect 10% higher clocks won’t compensate enough for cache hits being several times faster.

That doesn’t make sense. On the 5000 series, we saw 3D was worth ~20% performance uplift, clock for clock, in the right workloads.

A 5GHz CCD with +20% would then be only barely faster than a 5.7GHz CCD with no 3D, at which point the pairing you suggest makes little sense.

Besides, 5800X3D attained 5.6GHz so there’s no big reason to think 7000X3D would be capped at 5GHz.

If your workload is highly sensitive to added cache, then it's likely that increasing frequency will bring diminished returns.

I believe it is both a scheduling and memory allocation/loading issue. It is interesting that caches are NUMA but main memory is not.

amd cache has been numa for a long time now

Reply on news.ycombinator.com