Measuring CPU core-to-core latency

Very nice

The phrase for this is NUMA, and NUMA scheduling is the term for scheduling that accounts for it. Mostly, it avoids moving threads between different NUMA domains.

You might just be going down a deep rabbit hole with these questions :)

You can definitely do things like that, including pinning an application to a certain set of cores for performance. When you have NUMA it's pretty much mandatory for low latency applications, since each CPU gets its own memory slots, significantly lowering latency cost. It's physically closer, for one.

It gets really fun when you get into network traffic, since you can bind a NIC to specific cores alongside kernel bypass stuffs and interrupt handling. Sometimes it even makes sense to have one NIC per CPU!

It's a lot of setup work though. Have a look into CPU shielding as well. The Linux scheduler cares more about fairness than it does latency, so you have to be very intentional in your setup.

I went to your homepage. Your gif "Programming in C++" made me really laugh, thanks for that! 8-)

https://rigtorp.se/

This works much better on my POWER9 system. I have two CPUs in my machine with the same core count but different cache configurations according to hwloc and this showed that they really are different.

You need to add a Makefile there. It's just two lines.

See also https://github.com/google/multichase

So what did you do to get the most performance out of the system? pin the threads to specific CPUs?

It'd be great if you could provide the output.csv actually :D

I think there's a CLI tool for pastebin

Would love to see what the results are for an m2.

M1 on Macbook air on macOS Ventura beta 7:

min/max/mean - 34.6ns/57.7ns/39.1ns

       0       1       2       3       4       5       6       7   
  0
  1   58±6 
  2   48±6    46±6 
  3   39±6    37±6    36±6 
  4   41±6    38±6    42±6    38±6 
  5   35±6    37±6    37±6    36±6    36±6 
  6   38±6    39±6    36±6    40±6    40±6    38±6 
  7   39±6    38±6    36±6    37±6    38±6    36±6    38±6

Okay, I've added a file

Just look at the processor architecture diagram.

But TL;DR modern big processors are not one big piece of silicon but basically "SMP in a box", a bunch of smaller chiplets interconnected with eachother. That helps with yield ("bad" chiplet costs you just 8 cores, not whole 16/24/48/64 core chip). Those also usually come with their own memory controllers.

And so you basically have NUMA on a single processor with all of the optimization challenges for it

Some of it is simple distance. Some of it is architectural choices because of the distance. A sharing domain that spans a large distance performs poorly because of the latency. Therefore domains are kept modest, but the consequence is crossing domains has an extra penalty.

Most of this cross-core overhead diversity is gone on skylake and newer chips because Intel moved from a ring topology to mesh design for their l3 caches.

Rings are great for latency on low core count situations. The LCC Intel chips all have a massive 512-bit ring bus (2x256-bit in each direction) internally which delivers crazy fast core to core latency. However, this quickly starts to break down under higher core counts. Intel gets around this to some extent with its P and E cores with 4 E cores occupying the same slot on the ring bus as a P core. However, once you start getting over 14-16 slots on the ring bus it starts to get overwhelmed.

So when you switch to mesh buses the interconnect takes up way more space. So one has to compromise between bus width and the amount of area one is using for the interconnects. Typically this means running reduced width buses around the mesh which limits core to core bandwidth. Not so much a big deal if you're running a server, more a problem though if you're trying to run interactively with a user. Unless of course you're Apple and just devote a truckload of die space to dump a fucking mammoth amount of interconnect between your dies.

There's also ancillary concerns as well like fabrication yield. For instance AMD runs chiplets probably because they can mix and match yields and they naturally segment the market. Get a CCX with 3 working cores? Pair it with another and you have a 6C/12T CPU. Get a CCX with 2 working cores? Pair it with another and you get a 4C/8T. Intel either gets a working die or they don't.

The problem here is the interconnect between the CCXs is relatively slow. Dog slow compared to the ring bus. Even running the Infinity Fabric's fclock at 1.8GHz only nets you 57.6GB/sec between CCXs and five times the latency of the ring bus. When you look at a Ryzen 3300 (2x2 CCX) and a Ryzen 3300X (1x4 CCX) the difference in performance is non-trivial and that's the Infinity Fabric dragging performance down. In comparison an Intel core's L3 cache on a 3GHz ring bus (i.e. non-turbo) pulls down at 96GB/sec. Sure you're still ultimately limited by DRAM but if stuff is staying in LLC it's a hell of a performance boost. In Zen 3 AMD even went to 8 core CCXs which gave the whole thing a huge performance boost. Part of that was because the smaller lithography gave them more area to play with so they could fit everything plus the interconnects onto the chiplet size they needed.

So yeah, I hope that little greatly oversimplified, surface level look was helpful.

Please see my reply/response ref: "ffwd: delegation is (much) faster than you think" in this HN post. It's a scheme to leverage high core-to-core bandwidth to make coordinating shared data structures easier. I'd definitely be interested in your 2 cents.

Could it be because it's the newest CPU included in the test suite.

Possibility related to what fab generation the CPU is on vs. other chips tested?

EDIT: why the downvotes?

No idea. There are 8 performance cores and 8 efficiency cores, but that doesn't explain why only one core is faster. There's a die shot here: https://cxblobs.blob.core.windows.net/images/Alder_Lake_revi...

It would be nice to have another bench from the Alder Lake, so we can validate the results

I suspect what you are seeing is the preferred cores. With TVB/OCTVB (thermal velocity boost/overclocking thermal velocity boost) those can boost higher than the other cores.

From my 12900ks (its a boosted 12900k) Core 6-7 is rated for 5.5ghz.

Freq MHz DTS C VID mv PC Eff Fr UCcode 0x1f VR Volt Limit 2500 ---------------------------------------------------- 800 24 1307 52 517 Uncore 3600 IA AC LL 0.5999 800 20 1282 52 47 Power 23.465 IA DC LL 1.0996 800 20 1314 52 70 Current Limit 0.0000 SA AC LL 0.0000 800 20 1308 52 69 iccmax 1023 SA DC LL 0.0000 800 20 1303 52 88 PL1 32760 iccmax dis True 800 18 1293 52 69 PL2 32760 TAU 33 800 20 1286 55 39 Memory 5200 PPP_OVR True 800 22 730 55 37 EE_Turbo_Dis True RTH_Dis True 800 22 1377 40 60 Dis_Ring_EE False HWGuidedSch True 800 22 1172 40 73 IA_CEP_Dis True Dynamic_Mem True Full_Range_Multi False SA_Freq_OVR False TSC_Dis_HW False Banding_Ratio 0 PVD_Ratio_thresh 0 SA_CEP_Dis True FLL OC Mode 3

---------------------------------------------------------------------------------------------------------------------------------

Core Voltage Adapt 0 Ring Voltage Adapt 0 Core PLL V 900 Core Voltage Offset 0 Ring Voltage Offset 0 Ring PLL V 900 L2 Voltage Adapt 0 L2 Voltage Offset 0 AVX512 Offset 0 SA Voltage Offset 0 Ring VID 0 MC PLL V 900 AVX Offset 0 AVX2 V Guardband 0 AVX512 V Guardband 0 SA Voltage Manual 1150

---------------------------------------------------------------------------------------------------------------------------------

Turbo Ratio Limit 55, 55, 52, 52, 52, 52, 52, 52 Core OCMB Max Ratio 0 Ring Min Ratio 8 Turbo Limit Cores 1, 2, 3, 4, 5, 6, 7, 8 Ring OCMB Max Ratio 0 Ring Max Ratio 47 Atom Ratio Limit 40, 40, 40, 40, 40, 40, 40, 40 Atom Limit Cores 255, 255, 255, 255, 255, 255, 255, 255Atom OCMB Max Ratio 0 OS Max Ratio 34 HWP Min Ratio 43 HWP Max Ratio 255 Max Possible Core 40 Max Possible Ring 47 UCLK 2600

Num cores: 24 Using RDTSC to measure time: false Num round trips per samples: 5000 Num samples: 300 Showing latency=round-trip-time/2 in nanoseconds:

       0       1       2       3       4       5       6       7       8       9      10      11      12      13      14      15      16      17      18      19      20      21      22      23
  0
  1    0±0
  2   37±6    36±6
  3   31±6    37±6     5±5
  4   26±6    26±6    26±6    26±6
  5   26±6    31±6    26±6    31±6     5±5
  6   31±6    31±6    36±6    31±6    26±6    21±6
  7   31±6    31±6    31±6    31±6    26±6    26±6     5±5
  8   31±6    31±6    31±6    31±6    26±6    26±6    31±6    26±6
  9   36±6    31±6    31±6    31±6    26±6    26±6    26±6    26±6     5±5
 10   31±6    36±6    31±6    26±6    26±6    21±6    26±6    31±6    26±6    31±6
 11   31±6    36±6    31±6    36±6    26±6    26±6    31±6    31±6    36±6    31±6     5±5
 12   31±6    36±6    31±6    42±6    26±6    31±6    31±6    37±6    31±6    37±6    31±6    36±6
 13   36±6    37±6    42±6    36±6    26±6    26±6    31±6    31±6    31±6    31±6    31±6    31±6     5±5
 14   31±6    36±6    37±6    36±6    31±6    26±6    36±6    31±6    36±6    31±6    36±6    31±6    31±6    36±6
 15   31±6    31±6    31±6    31±6    26±6    26±6    36±6    31±6    31±6    31±6    31±6    31±6    31±6    31±6     0±0
 16   42±6    42±6    42±6    42±6    31±6    31±6    37±6    36±6    36±6    42±6    37±6    36±6    36±6    36±6    37±6    37±6
 17   42±6    42±6    36±6    42±6    31±6    36±6    37±6    36±6    36±6    37±6    36±6    37±6    36±6    36±6    36±6    37±6    47±6
 18   42±6    42±6    42±6    42±6    31±6    31±6    37±6    36±6    37±6    36±6    42±6    36±6    37±6    36±6    36±6    36±6    47±6    52±6
 19   42±6    42±6    42±6    37±6    31±6    31±6    36±6    37±6    36±6    36±6    37±6    37±6    36±6    36±6    42±6    37±6    52±6    47±6    47±6
 20   42±6    42±6    42±6    36±6    37±6    31±6    37±6    36±6    37±6    36±6    36±6    42±6    37±6    36±6    36±6    42±6    42±6    42±6    42±6    42±6
 21   42±6    37±6    42±6    42±6    31±6    31±6    37±6    36±6    42±6    36±6    37±6    36±6    37±6    37±6    36±6    36±6    42±6    42±6    42±6    42±6    47±6
 22   42±6    42±6    42±6    36±6    36±6    31±6    36±6    37±6    36±6    37±6    36±6    36±6    36±6    37±6    36±6    42±6    42±6    42±6    42±6    42±6    47±6    52±6
 23   36±6    42±6    42±6    42±6    31±6    31±6    37±6    36±6    37±6    42±6    36±6    36±6    37±6    42±6    36±6    36±6    42±6    42±6    42±6    42±6    52±6    47±6    47±6

Min latency: 0.0ns ±0.0 cores: (1,0) Max latency: 52.3ns ±16.3 cores: (23,20) Mean latency: 34.6ns

It's probably the ring bus topology. As you pointed out E cores are directly connected to L2 and only one stop away from each other on the ring bus. P cores can be up to 6 stops away from each other because there's 4 on each side, an E core cluster on each side, and a system agent (PCIe/DRAM) or iGPU stop in between each side.

Might too-aggressive power saving be an issue? The spikes (100+ns) are all hyper-threads. edit: No, 4 to 6 is not a HT.

The data seems problematic, there's a few 0's here and there and some strange noise in the rest. Increase the number of samples / iterations with `core-to-core-latency 30000 1000 --csv > output.csv`

I only use AF_UNIX sockets when I need to pass open file handles between processes. I generally prefer message queues: https://linux.die.net/man/7/mq_overview

I haven’t measured myself, but other people did, and they found the latency of message queues is substantially lower: https://github.com/goldsborough/ipc-bench

They mention that they measure "cache coherence protocol" throughput. So yes, this happens when two or more cores need to work on the same memory and they need to synchronize/update what is in their L1 and L2 (and probably L3 as well?) caches.

Answering only the latter question:

A Primer on Memory Consistency and Cache Coherence, Second Edition

https://www.morganclaypool.com/doi/10.2200/S00962ED2V01Y2019...

(free online book) would help

In HFT, we typically pin processes to run on a single isolated core (on a multicore machine). That allows the process to avoid a lot of kernel and other interrupts which could cause the process to not operate in a low latency manner.

If we have two of these processes, each on separate cores, and they occasionally need to talk to each other, then knowing the best choice of process/core location can keep the system operating in the lowest latency setup.

So, an app like this could be very helpful for determining where to place pinned processes onto specific cores.

There's also some common rules-of-thumb such as, don't put pinned processes that need to communicate on cores that are separated by the QPI, that just adds latency. Make sure if you're communicating with a NIC to find out which socket has the shortest path on the PCI bus to that NIC and other fun stuff. I never even thought about NUMA until I started to work with folks in HFT. It really makes you dig into the internals of the hardware to squeeze the most out of it.

It's mentioned in the readme - this is measuring the latency of cache coherence. Depending on architecture, some sets of cores will be organized with shared L2/L3 cache. In order to acquire exclusive access to a cache line (memory range of 64-128ish bytes), caches belonging to other sets of cores need to be waited on to release their own exclusive access, or to be informed they need to invalidate their caches. This is observable as a small number of cycles additional memory access latency that is heavily dependent on hardware cache design, which is what is being measured

Cross-cache communication may simply happen by reading or writing to memory touched by another thread that most recently ran on another core

Check out https://en.wikipedia.org/wiki/MOESI_protocol for starters, although I think modern CPUs implement protocols more advanced than this (I think MOESI is decades old at this point)

The example code uses an atomic store instruction in order to write values from threads to a memory location, and then an atomic read to read them. The system guarantees that a read of a previously written location is consistent with a subsequent write, i.e. "you always read the thing you just wrote" (on x86, this guarantee is called "Total Store Ordering.") Reads and writes to a memory location are translated to messages on a memory bus, and that is connected to a memory controller, which the CPUs use to talk to the memory they have available. The memory controller is responsible for ensuring every CPU sees a consistent view of memory according to the respective platform memory ordering rules, and with respect to the incoming read/write requests from various CPUs. (There are also caches between the DRAM and CPU here but they are just another layer in the hierarchy and aren't so material to the high-level view, because you can keep adding layers, and indeed some systems even have L1, L2, L3, and L4 caches!)

A CPU will normally translate atomic instructions like "store this 32-bit value to this address" into special messages on the memory bus. Atomic operations it turns out are already normally implemented in the message protocol between cores and memory fabric, so you just translate the atomic instructions into atomic messages "for free" and let the controller sort it out. But the rules of how instructions flow across the memory bus is complicated because the topology of modern CPUs is complicated. They are divided, partitioned into NUMA domains, have various caches that are shared or not shared between 1-2-or-4-way clusters, et cetera. They must still obey the memory consistency rules defined by the platform, and all the caches and interconnects between them. As a result, there isn't necessarily a uniform measurement of time for any particular write to location X from a core to be visible to another core when it reads X; you have to measure it to see how the system responds, which might include expensive operations like flushing the cache. It turns out two cores that are very far away will just take more time to see a message, since the bus path will likely be longer -- the latency will be higher for a core-to-core memory write where the write will be visible consistently.

So when you're designing high performance algorithms and systems, you want to keep the CPU topology and memory hierarchy in mind. That's the most important takeaway. From that standpoint, these heatmaps are simply useful ways of characterizing the baseline performance of some basic operations between CPUs, so you might get an idea of how topology affects memory latency.

Hm. Use icelake, with an aggregator process sitting in core 11 and have all the others run completely on input alone and then report to core 11. (Core 11 from that heatmap appears to be the only cpu with a sweetheart core having low latency to all other cores.) I wonder how hard is to write a re-writer to map an executable to match cpu architecture characteristics. Something like graph transformations to create clusters (of memory addresses) that are then mapped to a core.

2 hours after your comment, it has been done: https://news.ycombinator.com/item?id=32892164

I think that's one of the Intel chips where the cores are connected on a 2D grid rather than the more hierarchical or single ring interconnects you see in the other systems.

It's an order of magnitude slower.