Hacker News Reader

Comment by bhouston | Hacker News Reader

64 cores with 128 threads. Hope that comes to threadripper. I love and for bringing back competition to the CPU market. We've bought a ton of amd machines in the last year, great bargains in our space.

View on HN

I really think this is necessary for exascale computing. The core floodgate would have never opened without Intel getting several kicks in the pants.

You can do the math but my estimates are that the energy costs would still be too high for exascale with pure-CPU compute. Accelerators are the way anyone actually planning to build an exascale machine is getting there.

I think it’s doable with tightly integrated compute nodes, ie specially engineered.

Benchmarks on the 32-core TR are... disappointing, to say the least.[1] If you’re purely compute bound, it can be a win over the 16-core version, but if memory access is a factor, it’s a wash due to the extra hops to memory. And to my mind, there are very few pure-compute applications that wouldn’t benefit more from AVX2 and the like... in which case a cheaper Intel CPU would still wipe the floor with the 32-core chip.

I’m a huge fan of the Threadripper concept, just wish they hadn’t neutered the 32-core chip compared to its Epyc counterpart.

[1]: https://www.anandtech.com/show/13124/the-amd-threadripper-29...

Threadripper’s 4 die variant had to be “neutered” to remain compatible with the TR4 socket without splitting memory bandwidth. The new I/O chiplet would solve that issue entirely, assuming it comes to Threadripper 3.

I was just thinking this. Yes it'll have half the lanes of EPYC still, okay, but now the lanes will be connected to all the dies equally rather than to just 1/2 the dies. This hopefully means Threadripper 3 can perform almost as well as EPYC with just 1/2 the memory bandwidth - that isn't insignificant but it's a big improvement on Threadripper 2.

An interesting point is that TR owners are much more likely to run their RAM at higher, out of JEDEC spec speeds, so the difference can be a lot smaller than 1/2.

Those (Anandtech) benchmarks were performed on Windows. All threadripper benchmarks on Linux show that it is nowhere near as awful a performer as on Windows and most compute workloads do scale okay. Seen multiple ideas thrown around like Windows not being NUMA aware with this processor or just plain bad core scheduling

They did a follow up changing the scheduling policy for thread 0 (again, still on Windows) and it didn’t make a difference for almost all their workloads: https://www.anandtech.com/show/13446/the-quiz-on-cpu-0-playi...

See the phoronix benchmarks, linux performs much better on same workloads.

AnandTech really needs to hire a Linux-focused editor to do some benchmarks there too, especially for these large systems that are unlikely to be running Windows anyways.

The Phoronix benchmarks are quite clear,[0] I don't know why you keep linking to AnandTech's Windows benchmarks. I say this as someone who reads tons of AnandTech reviews because they're great, but Windows just doesn't do well with high core count hardware at all.

[0]: https://www.phoronix.com/scan.php?page=article&item=2990wx-l...

This seems like a familiar issue I've run into with workstations I've used in the past running Xeons. Not sure how NTOSKRNL handles scheduling of parallel tasks. I'd venture a guess and say it's hybrid (M:N threads), where multiple userland application threads are mapped to some "virtual processor" in kernelmode. That leads to priority inversion between the userland and kernelmode threads, which could explain why Windows benchmarks are terrible when dealing with multiple physical cores.

As far as I know Win NT threads are 1:1.

Not even sure how it would work or even make any sense to have N:M handled by the kernel. N:M is usually a mainly a userspace thing. And Windows is even less likely to use that kind of convolution, because IIRC it can call back from kernel to userspace (that design I would not recommend, btw, but oh well). You have fibers, of course, but that's a different thing.

Windows does not scale probably simply because the kernel is full of "big" locks (at least not small enough...) everywhere, and they have far less fancy structures and algo than Linux (is there any equivalent of RCU that is widely used in there? - not sure). Cf the classic posts of the builder of Chrome who every now and then encounter a ridiculous slowdown of his builds on moderately big computers, sometimes because of mutexes badly placed.

> Not even sure how it would work or even make any sense to have N:M handled by the kernel. N:M is usually a mainly a userspace thing.

Correct, I meant that the benchmarking program itself probably used that implementation. Not the Win NT kernel’s implementation of OS threads.

For reference, here is a link to the Phoronix Windows vs Linux on the 2990WX article:

https://www.phoronix.com/scan.php?page=article&item=2990wx-l...

It is not just about 32-core compute performance, if you compare a $385 threadripper motherboard + $650 CPU to the equivalent Intel competition, for single socket, the Threadripper is far ahead in terms of high bandwidth I/O. The threadripper has 64 PCI-Express 3.0 lanes direct to the CPU, for use with things like 25/40/100GbE network interfaces, or very large numbers of 10GbE interfaces. Or low-latency cluster interconnect fabric cards.

They are direct to 1/2 of the CPUs, actually.

This is part of what the GP was complaining about. The 2 dies which do not have memory also do not have PCIe wired out of the socket, so they are one hop away from I/O as well. If you are trying to max out I/O, you'd probably be better off with a low-end Epyc that had fewer cores enabled per die.

It is interesting to note that the memoryless dies do have PCIe root ports, for built-in things like the PSP crypto engines. That surprised me somewhat when I first noticed it, but it makes sense in retrospect, since the issue is that the TR socket was designed for 2 dies.

The interconnect latencies give you something like an extra 50 nanoseconds[1]. That matters when accessing memory. Not so much for PCIe, where your base latency is most of a microsecond[2][3]. There seems to be plenty of bandwidth to handle it, too.

[1] https://www.servethehome.com/amd-epyc-infinity-fabric-latenc...

[2] https://forum.stanford.edu/events/posterslides/LowLatencyNet...

[3] https://gianniantichi.github.io/files/papers/pciebench.pdf

My comment was addressing the parent comment about bandwidth, not latency.

Because you only have 1/2 the memory and I/O bandwidth on threadripper, if maxing out I/O is your concern, you would likely be better off with a low-end epyc that still had all the memory controllers and pcie lanes wired up.

> My comment was addressing the parent comment about bandwidth, not latency.

Oh, it is? It really seemed to be about direct vs. non-direct, and nothing else. And that pretty much only affects latency, because it only loads the infinity fabric by about 20-25% to route 16 PCIe lanes to each die.

The neutering isn't that bad, you can still get an improvement by adding more cores. Here's some of my results on a task doing a lot of string comparisons (plus some other stuff): https://image.ibb.co/ecH9VL/coretest.png

Predicted = Theoretical improvement (2 cores are 2x faster than 1, etc)

10k/00k = number of entries being searched (there's actually a variable number of strings per entry)

I suspect the main problem I tackle at work, robotic grasp planning, could make use of all those threads despite the bandwidth constraint. Now if only someone would put an Epyc in an industrial PC...

Reply on news.ycombinator.com