Hacker News Reader

Comment by pmjordan | Hacker News Reader

The Mac pro only has one CPU because that's all they could fit in.

I suspect the real reason is that Apple still {can't be bothered|can't figure out how} to add NUMA support to OS X. The existing 2-socket Mac Pros have been crippled by the interleaved-mode NUMA. (which, interestingly enough, is even hardcoded in the firmware, so when running OSes that support NUMA on a Mac Pro, you can't benefit from that support, presumably so the benchmarks don't shine a bad light on OS X - of course it's blindingly obvious when compared to a same-spec workstation from HP, Dell, etc.)

In fairness, adding NUMA to xnu is probably quite an undertaking. I wouldn't be surprised if they've previously tried and got in a terrible mess. The task/thread management is scattered across the Mach and BSD subsystems, and the virtual memory system handles some conditions (such as low memory) terribly, and seems to scale badly to large amounts of RAM and 64 bits.

View on HN

Getting rid of user-accessible PCI has a similar benefit for Apple - no more hassle having to support a zoo of PCI cards.

Except there are Thunderbolt-to-PCIe enclosures, which come with their own set of problems (latency, hotplugging, I/O memory range allocation, bandwidth, not to mention cost). But I guess Apple can just say "we don't support those, take it up with the enclosure manufacturer".

Drivers for Thunderbolt devices still are necessarily kexts and can thus take down the system without an obvious culprit. For this reason alone I'm surprised they went with it rather than supporting USB3 early on or that they didn't come up with something similar but altogether new, like they did with Lightning. They seem to be trying to cut down third party kext development, and aren't opening the sources for most of their own new kexts. Of course, with IOMMUs available in all Thunderbolt systems, they could have brought back the microkernel aspect of Mach for device drivers, but that would have been even more work.

What does NUMA have to do with SMP setups? NUMA would be about CPUs with separate physical memories acting as one weird processing resource, not a Mac Pro with multiple processes connected to the same memory. What am I missing here?

All modern CPUs have memory controllers built in as opposed to using a distinct "northbridge" chip. The RAM attaches directly to the CPU socket (or soldered BGA). In multi-socket systems, this means there are memory buses coming off each CPU, and for one CPU to reach memory attached to the other CPU, it needs to go through that CPU via the interconnect. Here's some ASCII art of a system with 3 memory channels from each CPU (old Mac Pro):

  +------+                +------+
  | CPU0 +==interconnect==+ CPU1 |
  +-+-+-++                ++-+-+-+
    |  \ \                / /  |
  MEM0 |  \              /  |  MEM5
     MEM1 MEM2        MEM3 MEM4

Going via the interconnect introduces extra latency and consumes its bandwidth. Therefore, it's more efficient to pin threads and/or processes to specific CPU sockets, and to allocate memory for it from that socket's associated physical memory. This requires kernel support, and for absolute top performance you also want a user space API, so you don't need to rely on the in-kernel heuristics.

Lacking software support means you have to fall back to whatever the hardware does automatically. Current multi-socket x86 CPUs all offer an "interlaced" mode for this (sort of like RAID0), where they dispatch to physical socket based on some physical address bits, e.g.:

  0x00000000-0x0000003f: CPU0's RAM
  0x00000040-0x0000007f: CPU1's RAM
  0x00000080-0x000000bf: CPU0's RAM
  0x000000c0-0x000000ff: CPU1's RAM

and so on. This spreads the extra latency evenly. Makes for consistent but poor memory performance. The mode is selected during system initialisation, i.e. is the responsibility of the Firmware (BIOS/EFI). Non-Apple workstations and servers typically let you select this as part of the pre-boot system setup UI.

It depends on the motherboard layout, but you'll find that there are groups of DIMM slots located near each CPU socket in multi-socket systems. Physical proximity helps keep speeds high.

The other thing to mention here is with PCIe 3, the controller for the bus is now in the CPU, which means that if you have two GPUs, you'll start to see GPU affinity problems as well. In previous systems you'd have one bus attached to a southbridge, which is then shared by both CPUs*

I've seen this with linux and nvidia cards, with two CPUs we would get a maximum CPU->GPU transfer of around 7-8gigs, take out a CPU and it'd bump up to 10-13. The z620 has the second CPU on a daughter card, so its simple to do.

*my memory is a bit hazy, I'm open to corrections

You should at least be able to control that with regular CPU affinity from software.

I get it, I haven't studied NUMA much since 10 years ago, when we were looking in it in terms of Sparc interconnects and expensive workstation stuff.

From what I've heard from my colleagues who do HPC, NUMA just isn't that effective. The sweet spot right now is either with GPUs or MPI; the shared memory abstraction that NUMA provides is just not very tenable. Where is NUMA being used as a best (rather than legacy) solution right now?

The question is not so much "where is NUMA effective," but, "given that for highest speeds memory needs to be attached directly to the CPU, how do we deal effectively with the case of multiple CPUs?" If you want multi-socket SMP, these days you need to deal with its inherent NUMA architecture. Or sweep it under the rug as Apple did with the old Mac Pro and sacrifice performance.

Am I the only one curious as to what NUMA stands for?

Non-uniform memory access http://en.m.wikipedia.org/wiki/Non-Uniform_Memory_Access

Reply on news.ycombinator.com