Non-Uniform Memory Access (NUMA) is reshaping microservice placement
codemia.io
codemia.io
1. In the cloud, it can be difficult to know the NUMA characteristics of your VMs. AWS, Google, etc., do not publish it. I found the ‘lscpu’ command helpful.
2. Tools like https://github.com/SoilRos/cpu-latency plot the core-to-core latency on a 2d grid. There are many example visualisations on that page; maybe you can find the chip you are using.
3. If you get to pick VM sizes, pick ones the same size as a NUMA node on the underlying hardware. Eg prefer 64-core m8g.16xlarge over 96-core m8g.24xlarge which will span two nodes.
It's sad that we have to do this by ourselves
to browse the info. It is getting a bit old though.
There are so many performance quirks, and so much software doesn't account for it yet (in part, I'd bet, because most development environments don't have multiple NUMA domains.)
Here's a fun example we found a few years ago, not sure if work has happened in the upstream kernel since: the Linux page cache wasn't fully NUMA aware, and spans NUMA nodes. Someone at work was specifically looking at NUMA performance, and chose to benchmark databases on different NUMA nodes, trying the client on the same NUMA node, and then cross NUMA node, using numactl to pin. After a bunch of tests it looked like with client and server in NUMA 0 it was appreciably faster than client and server in NUMA 1. After a reboot, and re running tests, it had flipped. NUMA 1 faster than NUMA 0. Eventually they worked out that the fast NUMA was whichever one was benchmarked first after a reboot, and from there figured out that when you ran fresh, the database client library ended up in the page cache in that NUMA domain. So if they benchmarked with server in 0, client in 1, and then benchmarked with server in 0, client in 0, that clients access to the client library ended up reaching across to the page cached version in 1, paying a nice latency penalty over and over. His solution was to run the client in a NUMA pinned docker container so that it was a unique file to the OS.
Here on HN I saw a comment by someone running VM scale sets with hundreds or even thousands of nodes and their trick was to overprovision and then delete the instances that span NUMA nodes.
One thing the writeup didn’t seem to get into is the lack of scalability of this approach (manual pinning). As core counts and chiplets continue to explode, we still need better ways of scaling manual pinning or building more NUMA-aware OSes/applications that can auto-schedule with minimal penalties. Don’t get me wrong, it’s a lot better than ye olden days of dual core, multi-socket servers and stern warnings against fussing with NUMA schedulers from vendors if you wanted to preserve basic functionality, but it’s not a solved problem just yet.
EDIT: aaaand ... I commented before reading the article, which describes this very mechanism.
Really enjoyed this amazing write up on how Kube does use cgroups. Seems like the QoS controls do give some top level cgroups, that pods then nest inside of. That's something. At least! https://martinheinz.dev/blog/91
Most of us are in the realm of the lowest hanging fruit being database queries that could be 100x faster and functions being called a million times a day that only need to be called twice.
In 99% of use cases, there’s other, easier optimizations to be had. You’ll know if you’re in the 1% workload pinning is advantageous to.
For everyone else, it’s an excellent explainer why most guides and documentation will sternly warn you against fussing with the NUMA scheduler.
Cpu pinning can be super easy too. If you have an application that uses the whole machine, you probably already spawn one thread per cpu thread. Pinning those threads is usually pretty easy. Checking if it makes a difference might be harder... For most applications, it won't make a big difference, but some applications will see a big difference. Usually a positive difference, but it depends on the application. If nobody has tried cpu pinning your application lately, it's worth trying.
Of course, doing something efficiently is nice, but not doing it is often a lot faster... Not doing things that don't need to be done has huge potential speedups.
If you want to cpu pin network sockets, that's not as easy, but it can also make a big difference in some circumstances; mostly if you're a load balancer/proxy kind of thing where you don't spend much time processing packets, just receive and forward. In that case, avoiding cross cpu reads and writes can provide huge speedups, but it's not easy. That one, yeah, only do it if you have a good idea it will help, it's kind of invasive and it won't be noticable if you do a lot of work on requests.
Probably another situation is if you're working on a DBMS itself.
Would be interesting to see if something similar appears for cloud workloads.
Last time I was architect of a network chip, 21 years ago, our library did that for the user. For workloads that use threads that consume entire cores, it's a solved problem.
I'd guess that the workload you had in mind doesn't have that property.
* https://www.thomas-krenn.com/en/wiki/Display_Linux_CPU_topol...
you'll see some NUMA nodes with networking I/O attached to them, others with NVMe, and others with no I/O. So if you're really worried about network latency then you'd pin the process to that node, but if you want look at disk numbers (a database?) you'd be potentially looking at that node.
In recent years there's also chiplet-level locality that may need to be considered as well.
Examining this has been a thing in the HPC space for a decade or two now:
https://www.phoronix.com/news/Linux-6.17-NUMA-Locality-Rando... https://www.phoronix.com/news/Linux-6.13-Sched_Ext https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tierin... https://www.phoronix.com/news/Linux-6.14-FUSE
There's some big work I'm missing thats more recent too, again about allocating & scheduling IIRC. Still trying to find it. The third link is in DAMON, which is trying to do a lot to optimize; good thread to tug more on!
I have this pocket belief that eventually we might see post NUMA post coherency architectures, where even a single chip acts more like multiple independent clusters, that use something more like networking (CXL or UltraEthernet or something) to allow RDMA, but without coherency.
Even today, the title here is woefully under-describing the problem. A Epyc chip is actually multiple different compute die, each with their own NUMA zone and their own L3 and other caches. For now yes each socket's memory is all via a single IO die & semi uniform, but whether that holds is in question, and even today, the multiple NUMA zones on one socket already require careful tuning for efficient workload processing.
Even the Raspberry Pi 5 benefits from NUMA emulation because it makes memory use patterns better match the memory controller’s parallelization capabilities.
I also really enjoyed Semantic Streaming Registers paper, which makes load/store implicit in some ops, adds counters that can walk forward and back automatically so that you can loop immediately and start the next element, have the results dropped into the next result slot. This enables near DSP levels of instruction density, to be more ops focused rather than having to spend instructions writing and saving each step. https://www.research-collection.ethz.ch/bitstream/20.500.118...
I still have a bit of a hard time seeing how we bridge CPU and GPU. The whole "single program multiple executor" waves aspect of the GPU is spiritually just launching a bunch of tasks for a job, but I still struggle to see an eventual convergence point. The GPU remains a semi mystical device to me.
Most advanced uses of e.g. AVX-512 are not just doing simple loop-unrolling style parallelism. They are doing non-trivial slicing and dicing of heterogeneous data structures in parallel. There are idioms that allow you to e.g. process unrelated predicates in parallel using vector instructions, effectively MIMD instead of SIMD. It enables use of vector instructions more pervasively than I think people expect but it also means you really need to know where the register boundaries are with respect to your data structures.
History has generally shown that when it comes to optimization, explicitness is king.
I don't understand this take, you can still querry the vector length and have specialized implementations if needed.
But the vast majority of cases can be written in a VLA way, even most advanced ones imo.
E.g. here are a few things that I know to work well in a VLA style: simdutf (upstream), simdjson (I have a POC), sorting (I would still specialize, but you can have a fast generic fallback), jpeg decoding, heapify, ...
NUMA systems now make this even more obvious, when scheduling is not done properly.