The art of high performance computing
theartofhpc.com
theartofhpc.com
I used to work in AWS, but on the software / services side of things. But now and then, we would crash some talks from the datacenter folks.
One key relevation for me was that increasing compute power in DCs is primarily a thermodynamics problem than actual computing. The nodes have become so dense that shipping power in and shipping heat out, with all kinds of redundancies is an extremely hard problem. And it's not like you can perform a software update if you've discovered some inefficiencies.
This was ~10 years ago, so probably some things have changed.
What blows me away is that Amazon, starting out as an internet bookstore is at the cutting edge of solving thermodynamics problems.
Water has a massive amount of thermal capacity and can quickly and in bulk be cooled to optimal temperatures. You'd probably still need fans and AC to dissipate heat of non-liquid cooled parts, but for the big energy items like CPUs and GPUs/compute engines, you could ship out huge amounts of heat fairly quickly and directly.
I guess the complexity and risk of a leak would be a problem, but for amazon sized data centers that doesn't seem like a major concern.
https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...
https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...
https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...
Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints.
There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world.
Commodity DS are designed with the assumption that the average machine will run with a fraction of it's maximum load. HPC systems at the opposite are designed to operate safely at 100% load all the time.
In a previous company where I worked, we attempted to install a medium size HPC cluster in a well-known commerical datacenter and network provider. The commercial of the DS almost felt from his chair when we announced the power requirements.
Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.
[1]https://www.supermicro.com/en/products/system/gpu/5u/sys-521...
Also, this involves a UK university where it would takes forever to upgrade the power delivery to the building/floor/room. There's no planned upgrade whatsoever so people just need to make do with what we have.
The one data point I know well is NERSC, and has been air cooled until relatively recently. Part of the success of air cool in the past was the great Bay Area weather that is typically cooled enough. I forgot the exact reason for upgrading to water cool, but it was before the recent upgrade to a new machine (Perlmutter). It may have to do with our weather getting more extreme so that there are more incidences that air cool only will “throttle” your machine.
The reason I’m skeptical about that claim is that the ambient temperature also play a role. But certainly the density is increasing so may be the biggest supercomputers at the moment are too dense (in power consumption) to be air cooled.
- Frontier: Liquid cooled
- Aurora: liquid cooled
- Eagle: liquid cooled
- Fugaku: liquid cooled
- Lumi: Liquid cooled with heat recycling.
The need for liquid cooled arrived with the usage of accelerators that bumped significantly the power density.
That said, Fugaku which is not accelerator driven is also liquid cooled today.
I am surprised to learn that NERSC was air cooled until recently. Most of the BGQ machines, that were dominating the top500 10 years ago, were already liquid cooled.
But perhaps you're right that the physics is the ultimate factor here: what comes in must goes out. With order of magnitude scale of increase in power coming in, water seems to be the most effective and cheap entity to absorb those heat and be transported outside the floor very efficiently.
You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc.
Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat.
You add extensive leak detection subsystems, the amount of coolant moving in your primary circuit becomes massive.
Currently you can remove 97% of the heat via liquid (including the PSUs), and it’s cheaper to do so than air, but it’s not “rails, screws, cables, power on”. Air cooled systems can be turned on in a week. Liquid cooled ones take a month.
However, using liquid is mandatory after some point. You can’t cool systems that dense and under that load with air. They’ll melt.
But not convect. Hence why it’s much, much harder than removing heat on Earth.
Almost. It’s proportional [0] to T_hot^4 - T_cold^4. For a 100C surface with emissivity 1, that’s about 1kW/m2 if there is no radiation coming back, which really isn’t very high. You cannot cheat this with fancy folded-up radiating surfaces (it’s thermodynamically impossible, and the actual mechanism that kills it is one fin of the heatsink radiating right at the next one).
So cooling in space is hard. You’re not getting GPU-like power densities without a physically immense radiating surface extending way past those GPUs.
[0] Caveat: emissivity can depend on wavelength, and the law holds independently at each wavelength. So this can introduce interesting effects, which is how all the fancy prototype roof-cooling materials work, and it’s also related to how “spectrally selective” windows and window films work.
I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components.
Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe going through a 1U server and up to the next, then for a full 42U rack you have about 1.7kg of water going through the servers. If the flow rate is about 1s per server (so 42 seconds for the full rack) and each 1U server dumps 500W of energy into the water, there should be just a 3 degree C difference in the water temperature between the first and the last server.
> Delta T between two sockets is 5 degrees C
And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after reinstalling it the delta remained. Since I didn't notice any CPU throttling or whatsoever I figured it's "normal" and ignored it.
I mean, water travels from one socket to another, so one processor adds heat equal to 5 degrees C under nominal load. The second socket doesn’t complain much, but this is enormous amounts of heat transferred in a such quick pace.
I guess it could be bad past some %, but there's a probably a point where it's worth it.
[0]: https://www.opencompute.org/projects/cooling-environments
[1]: https://gitlab.com/open19/v2-specification/-/blob/main/syste...
Problem is that’s how you sell 1-3 racks not an entire room of them.
Ethylene lines might be more practical, easier to wire into existing systems.
What is the cutting edge of cooling tech like?
MPI and OpenMP are the primary abstractions from the hardware in HPC, with MPI being an abstracted form of distributed-memory parallel computing and OpenMP being an abstracted form of shared-memory parallel computing. Many researchers write their codes purely using those, often both in the same code. When using those, you really do not need to worry about the architectural details most of the time.
Still, some researchers who like to further optimize things do in fact fiddle with a lot of small architectural details to increase performance further. For example, loop unrolling is pretty common and can get quite confusing in my opinion. I vaguely recall some stuff about trying to vectorize operations by preferring addition over multiplication due to the particular CPU architecture, but I do not think I've seen that in practice.
Preventing cache misses is another major one, where some codes are written so that the most needed information is stored in the CPU's cache rather than memory. Most codes only handle this by ensuring column-major order loops for array operations in Fortran or row-major order loops in C, but the concept can be extended further. If you know the cache size for your processors, you could hypothetically optimize some operations to keep all of the needed information inside the cache to minimize cache misses. I've never seen this in practice but it was actively discussed in the scientific computing course I took in 2013.
The use of particular GPUs depends heavily on the problem being solved, with some being great on GPUs and others being too difficult. I'm not too knowledgeable about that, unfortunately.
I’m not sure how much multiplication vs addition matters on a modern chip. You can have a bazillion instructions in flight after all, as long as they don’t have any dependencies, so I’d go with whichever option shortens the data dependencies on the critical path. The computer will figure out where to park longer instruction if it needs to.
Of course, not every application is optimized to the hilt. But if you want to so optimize an application, exactly things you're talking about are what come into play.
So yes, I would expect every competent HPC practitioner to have a solid (if not necessarily intimate) grasp of hardware architecture.
Knowing more numerical analysis will get probably get you further in HPC than knowledge of specific hardware architectures.
Ideally you want both, of course.
When I wrote code for new HPC hardware, people were always surprised when I asked for the system hardware and architecture docs instead of the programming docs. But if you understood the hardware design, the correct way of designing software for it became obvious from first principles. The programming docs typically contained quite a few half-truths intended to make things seem misleadingly easier for developers than a proper understanding would suggest. In fact, some HPC platforms failed in large part because they consistently misrepresented what was required from developers to achieve maximum performance in order to appear "easy to use", and then failing to deliver the performance the silicon was capable of if you actually wrote software the way the marketing implied would be effective.
You can write HPC code on top of abstractions, and many people do, but the performance and scaling losses are often unavoidably integer factor. As with most software, this was considered an acceptable loss in many cases if it allowed less capable software devs to design the code. HPC is like any other type of software in that most developers that notionally specialize in it struggle to produce consistently good results. Much of the expensive hardware used in HPC is there to mitigate the performance losses of worse software designs.
In HPC there are no shortcuts to actually understanding how the hardware works if you want maximum performance. Which is no different than regular software, in HPC the hardware systems are just bigger and more complex.
Take for instance the so-called workload managers, of which the most popular ones are Slurm, PBS, UGE, LSF. Only Slurm is really open-source, PBS has a community edition, the rest is proprietary stuff executed in the best traditions of enterprise software which locks you into using pathetically bad tools, ancient and backwards tech with crappy / nonexistent documentation and inept tech support.
The interface between WLMs and the user who wants to use some resources is through submitting "jobs". These jobs can be interactive, but most often they are the so-called "batch jobs". A batch job is usually defined as... a Unix Shell script, where the comments are parsed to interpret those as instructions to the WLM. In the world with dozens of configuration formats... they chose to do this: embed configuration into Shell comments.
Debugging job failures is a nightmare, mostly because WLM software has really poor quality of execution. Pathetic error reporting. Idiotic defaults. Everything is so fragile it falls apart if you just as much as look at it in the wrong way. Working with it reminds me the very early days of Linux, when sometimes things just won't build, or would segfault right after you've tried running them, and there wasn't much you could do beside spending days or weeks trying to debug it just to get some basic functionality going.
When I have to deal with it, I feel kind of like in a steam-punk movie. Some stuff is really advanced, and then you find out that this advanced stuff is propped by some DIY retro nonsense you thought have died off decades ago. The advanced stuff is usually more on the side of hardware, while software is not keeping up with it for the most part.
The other cool thing about HPC is it is one of the last areas where multi-user Unix is used! At least, if you're using a university or NSF cluster that is!
Only other place I really see multiple humans using the same machine is SDF or the Tildes
[login1 ~:3] who | cut -d ' ' -f 1 | sort -u | wc -l
41Fist time I saw it, I had a flashback to the times when I worked for HP, and they were making some huge SAP knock-off, and that system was so labor-intensive to deploy that their QA process involved actual patches. As in pre-release QA cycle involved installing the system, validating it (which could take a few weeks) and if it's not considered DoD, then the developers are given the final list of things they need to fix and those fixes would have to be submitted as patches (sometimes, literal diffs that need to be applied to the deployed system with the patch tool).
This is, I guess, how the "patch version component" came to be in SemVer spec. It's kind of funny how lots of tools are using this component today for completely unrelated purposes... but yeah, LSF feels like the time is ticking there at a different pace :)
And what's wrong with shell scripts? It's a lingua franca, generally accepted across scientific disciplines, cluster vendors, workload managers, .... Considering the complexity of some setups (copy data to node-local file systems; run multiple programs, post-process results, ... ) I don't see how you could set up things other than in some scripting language. And then unix shell scripts are not the worst idea.
Debugging failures: yeah. Too many levels where something can go wrong, and it can be a pain to debug. Still, your average cluster processes a few million jobs in its lifetime. If more than a microscopic portion of that would fail, computing centers would need way more personnel than they have.
When used as configuration? Here are some things that are wrong:
* Configuration forced into a single line makes writing long lines inconvenient (for example, if you want Slurm with Pyxis, and you need to specify the image name -- it will most likely not fit on the screen.
* Oh, and since we mentioning Pyxis -- their image names have pound sign in them, and now you also need to figure out how to escape it, because for some reason if used literally it breaks the comments parser.
* No syntax highlighting (because it's all comments).
* No way to create more complex configuration, i.e. no way to have any types other than strings, no way to have variables, no way to have collections of things.
* No way to reuse configuration (you have to copy it from one job file to another). I honestly don't even know what happens if you try to source a job configuration file from another job configuration.
All in all, it's really hard to imagine a worse configuration format. This sounds like a solution from some sort of a code-golfing competition where the goal was to make it as bad as possible, while still retaining some shreds of functionality.
You can launch a job(s) via command-line, config in Bash comments, REST APIs, linking to their library, and I think a few more ways.
I found it pretty easy to setup and admin. Scaling in the cloud was way less developed when I used it, so I just hacked in a simple script that allowed scaling up and down based on the job queue size.
What do you like better and for what use-case? Mine was for a group of researchers training models, and the feature I desired most was an approximately fair distribution of resources (cores, GPU hours, etc.).
> I’m also curious if HPC practitioners have to fiddle a lot of black-box knobs to squeeze out performance?
In my experience with CUDA developers, yes the Shmoo Plot (https://en.wikipedia.org/wiki/Shmoo_plot, sometimes called a ‘wedge’ in some industries) is one of the workhorses of every day optimization. I’m not sure I’d call it black-box, though maybe the net effect is the same. It’s really common to have educated guesses and to know what the knobs do and how they work, and still find big surprises when you measure. The first rule of optimization is measure. I always think of Michael Abrash’s first chapter in the “Black Book”: “The Best Optimizer is Between Your Ears” http://twimgs.com/ddj/abrashblackbook/gpbb1.pdf. This is a fabulous snippet of the philosophy of high performance (even though it’s PC game centric and not about modern HPC.)
Related to your point about abstraction, the heaviest knob-tuning should get done at the end of the optimization process, because as soon as you refactor or change anything, you have to do the knob tuning again. A minor change in register spills or cache access patterns can completely reset any fine-tuning of thread configuration or cache or shared memory size, etc.. Despite this, some healthy amount of knob tuning is still done along the way to check & balance & get an intuitive sense of the local perf space of the code. (Just noticed Abrash talks a little about why this is a good idea.)
Usually though, you have to iterate. Doing shmoos along the way can help with understanding the effects of code changes, help understand how the hardware works, and it can sometimes help identify what code changes you might need to make. A simple abstract example might be I know what my theoretical peak bandwidth is, but my program only gets 30% of peak. I suspect it has to do with how many registers are used, and I have a knob to control it, so I turn the knob and plot all possible register settings, and find out that I can get 45% of peak with a different value. Now I know it was partially registers I was limited by, but I also know to look for something else too. Then I profile, examine the code, maybe refactor or adjust some things, hypothesize, test, and then shmoo again on a different knob or two if I suspect something else is the bottleneck.
You won't be able to use a supercomputer at all without scalability, and it's the one topic that is specific to it. But, of course, those computers time is quite expensive so you'll want to optimize for performance too. It's just secondary.
But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any kind of monitoring running. I set some up, and it doesn't really get used. Once in a while we get questions from management about "how busy is the cluster", to justify budgets and that sort of thing.
Most of my 'optimization' work ends up being things like making sure people aren't (usually unknowingly) requesting 384 CPUs when their script only uses 16, testing software to see what # of CPU's it works with before you see a degradation, etc. I've only had the Intel profiler open twice.
And I've found that most of the job is really just helping researchers and such with their work. Typically running either a commercial or open-source program, troubleshooting it, or getting some code written by another team on another cluster and getting it built and running on yours. Slogging through terrible Python code. Trying to get a C++ project built on a more modern cluster in a CentOS 7 environment.
It can be fun in a way. I've worked with different languages over the years so I enjoy trying to get things working, digging through crashes and stack traces. And working with such large machines, your sense of normal gets twisted when you're on a server with 'only' 128GB of RAM or 20TB of disk.
It's a little scary when you know the results of some of this stuff are being used in the real world, and the people running the simulations aren't even doing things right. Incorrect code, mixed up source code, not using the data they thing they are, I once found a huge bug that had existed for 3 years. Doesn't this invalidate all the work you've done on this subject?
The one drawback I find is that a lot of HPC jobs want you do have a masters degree. Even to just run the cluster. Doesn't make sense to me, I'm not writing the software you're running, we aren't running some state of the art, TOP500 cluster. We're just getting a bunch of machines networked together and running some code.
Researchers/Scientists with their hard earned PhDs should only concentrate on doing cutting-edge "researchy" stuff. It is hard enough that they should not be asked to learn all the intricacies/problems inherent in Software Development. That is the domain of a "Professional Software Engineer".
There is now in fact a new class called "Research Software Engineer" who are Software Developers working in Research developing code specific to their needs - https://www.nature.com/articles/d41586-022-01516-2 and https://en.wikipedia.org/wiki/Research_software_engineering
Is it possible that pretty much any specialization, outside of the most common ones, engages in a lot of gatekeeping? I remember how difficult it appeared to be after I graduated to break into embedded systems (I never did). I persisted until I realized it doesn't even pay very well, comparatively.
It's also cultural. From what I hear, the east cost US cares a lot more about prestige than the west coast that focuses a lot more on performance.
In today's x86_64 hardware, there's no "supercomputer memory subsystem". It's just a glorified NUMA system, and the biggest problem is putting the memory close to your core, i.e. keeping data local in your NUMA node to reduce latencies.
Your resource mapping is handled by your scheduler. It knows your hardware, hence it creates a cgroup which satisfies your needs and as optimized as possible, and stuffs your application into that cgroup and runs it.
Currently king of high performance interconnects is Infiniband, and it accelerates MPI at the fabric level. You can send messages, broadcasts and reduce results like there's no tomorrow. Because when the message arrives you, it's already reduced. When you broadcast, you only send a single message which is broadcasted at fabric layer. Multiple Context IB cards have many queues and more than one MPI job can run on the same node/card with queue/context isolation.
If you're using a framework for GPU work, the architecture & optimization is done at that level automatically (the framework developers do the hard work generally). NVIDIA's drivers are pure black magic, too. They handle some parts of the optimization, too. InterGPU connection is handled by a physical fabric, managed by drivers and its own daemon.
If you're CPU bound, your libraries are generally hand tuned by its vendor (Intel MKL, BLAS, Eigen, etc.). I personally used Eigen, and it has processor specific hints and optimizations baked in.
The things you have to worry is to compile your code for the correct architecture, make sure that the hardware you run on can satisfy your demands (i.e.: do not make too many random memory accesses, keep the prefetcher and branch predictor happy if you're trying to go "all-out fast" on the node, do not abuse disk access, etc.).
On the number crunching side, keeping things independent (so they can be instruction level parallelized/vectorized), making sure you're not doing unnecessary calculations, and not abusing MPI (reducing inter-node talk to only necessary chatter) is the key.
It's way easier said than done, but when you get the hang of it, it becomes like a second nature to think about these things, if these kinds of things are your cup of tea.
> In today's x86_64 hardware, there's no "supercomputer memory subsystem". It's just a glorified NUMA system, and the biggest problem is putting the memory close to your core, i.e. keeping data local in your NUMA node to reduce latencies.
I mean, memory topology varies greatly by uarch (doubly so between vendors). I can't take a routine tuned to Nehalem, run it on Haswell or Skylake and expect it to stay competitive. More generally, different hardware has different bandwidth and latency ratios, which affects software design (e.g. software written for commodity Dell w/ PCIe cards probably won't translate to Cray accelerator grid connected by HPE slingshot). And then there's hardware-specific features like RNICs bypassing DRAM and writing RDMA messages directly into the receiver's cache. So I think that ccNUMA and data locality is not sufficient to reason about memory perf.
You're absolutely right, this is why I said that if you're using libraries, this burden is generally handled by them. Also compilers do this and handle this very well.
If you're writing your own routines, the best way is to read the arch docs, maybe some low-level sites like chips and cheese, do some synthetic benchmarks and write your code in a semi informed way.
After writing the code, a suite of cachegrind, callgrind and perf is on order. See if there are any other bottlenecks, and tune your code accordingly. Add hints for your compiler, if possible.
I was able to reach insane saturation levels with Eigen plus, some hand-tuned code. For the next level, I needed to change my matrix ordering, but it was already fast enough (30 minutes to 45 seconds: 40x speedup), so I left it there.
Sometimes there are no replacement for blood, sweat and tears in this thing.
I have never played with custom interconnects (Slingshot, etc.), yet, so I can't tell much.
I’d bet a significant part of compute work on HPC clusters in academia works the same way. The only thing we paid attention to was number of cores on the node and preferring node local storage over the shared volumes for caching. No MPI.
There are of course problems requiring “genuine” HPC clusters but ours could have run on any pile of workers with a job queue.
I've seen similar things at the intersection of bioinformatics and genomics. Computers are getting bigger but the genomes aren't, and tasks that require distributed computing are getting rare.
While I am not acquainted with him personally, I did my doctoral work at UT Austin the the 1990's and had the privilege of working with the resources (Cray Y-MP, IBM SP/2 Winterhawk, and mostly on Lonestar, a host name which pointed to a Cray T3E at the time) maintained by TACC (one of my Ph.D. committee members is still on staff!) to complete my work (TACC was called HPCC and/or CHPC if I recall the acronyms correctly).
Back then, it was incumbent on the programmer to parallelize their code (in my case, using MPI on the Cray T3E in the UNICOS environment) and have some understanding of the hardware, if only because the field was still emergent and problems were solved by reading the gray Cray ring-binder and whichever copies of Gropp et al. we had on-hand. That and having a very knowledgeable contact as mentioned above :) of course helped...
Lonestar5 was a Cray again. Currently Lonestar6 is an oil-immersion AMD Milan cluster with A100 GPUs. The times, they never stand still.
I haven't found many good sources for this kind of information. If you are aware of any, please cite them in a comment below.
Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.
[0] https://www.youtube.com/playlist?list=PLBnLThDtSXOw_kePWy3CS...
[1]. https://techcommunity.microsoft.com/t5/microsoft-mechanics-b...
Solving these problems has basically been by job for the past 6 years, for Google's TPU systems and some of the GPU systems (the non-Cloud ones). It is a pity that, after the pandemic, it has been impossible to give similar conference talks at my company (at least if you are employed in Europe, due to restrictions on travel outside one's country).
The Meta sessions are very interesting. I wasn't aware of their work on Arcadia; Nvidia has also their own systems (Nvidia Air / Omniverse) in this area.
Queuing theory seems trivial and easy how it is introduced, but it has many open questions.
Performance metrics for a system with random arrival times, independent service times, with k servers (M/G/k) is still an open question as an example.
https://www.sciencedirect.com/science/article/pii/S089571770...
There are actually lots of open problems in queuing theory that one wouldn't expect.
The real world is way more complicated...
You can think about each host as its own markov chain: they may be serving, have hidden or diagnosed problems at various confidence levels, be on route to various remedial processes (reboot/reinstall/repairs I cited above, to simplify), require software/firmware changes, scheduled or opportunistic diagnostics (e.g. periodic deep-screening for otherwise silent problems).
Repair workflows are even more complicated and depend on the specific of the fault, and connection of components (e.g. a system diagnosed with a missing GPU may be because a cable or an interposer card is not seated correctly). Also, repair time is modulated by work shifts and some details of the logistics in datacenters.
Parts can become bottlenecks too. I remember one time in the early TPUv3 days: we had delays in recovering from a large incident because of fault-positive diagnoses of a $4 fan, that was widely used in all systems.
Add that nowadays systems are not one host and some attached cards, but have multiple nodes: e.g. the simplest combination is a main compute node and the smart nic / IPU, but some systems can be a lot more complicated.
So these alone are a hierarchical markov chains, with inhomogeneous arrival times and service times that are themselves time-dependent functions. There are also a lot of long-term memory effects, ensemble average is not necessarily the the time average. Chaotic behavior, in the mathematical sense, is common.
Systems are built with field replaceable units (FRU): e.g. in some generations, you can't swap one GPU in a server that has 8, you have the swap a whole block of them. You can choose when to repair, to maximize TCO/$ based on usage patterns (how many users want all GPUs vs. a smaller number).
Some systems (e.g. TPU pods) have links between accelerator trays, both within a physical rack and across them. So the usefulness of the neighbor system is reduced while a host/tray is being services, and you can have the equivalent of deadlock/livelock in repair dependencies.
Cluster scheduling and management is designed to maximize service levels, minimizing disruptions. This also implies that some disruptions (e.g. when you do some repairs) is driven by what workloads you run. Workloads are power-law distributed in size, so you have small-world network dynamics and the potential for supercritical behavior: both in disruptions (picture jobs preempting other jobs as a graph) but also in risks (add to the previous picture, one job that trigger an hardware problem).
Multiply these by several thousands to get the size of a datacenter cluster. Add supporting compute+storage, networking, power and thermal constraints. Multiply these by the number of clusters (Some hyperscalers have global scheduling systems, that make it possible to see the whole ML fleet as one). Add rare events, because at this scale you start to think about utility and electrical grid failures.
Figuring out what to do in control systems for the current ML fleet is one problem. Simulating what kind of service the future systems a few generations of hardware, software, datacenter design down the line should provide, so you can define what you can offer, influence the designs and make the right investments... it's more complicated. Both of these two problems are my current job.
Meta also publishes a number of papers/blogs/OSS projects on their engineering site [2]
James Hamilton of AWS gives a talk most years on their infrastructure. Worth watching multiple years [3].
[1] https://youtu.be/69PrhWQorEM?si=u7vh_Um6SQNoyeFH
[2] https://engineering.fb.com/category/data-center-engineering/
I haven't looked at what courses are being offered in several years, but when I was a graduate student, I really would have benefited from a dedicated semester-long course on parallel computing, especially going into the weeds about particular algorithms and data structures in parallel and distributed computing. Those were handled in a super cursory manner in the scientific computing course I took, as if somehow you'd know precisely how to parallelize things the first time you try. I've since learned a lot of this stuff on my own and from colleagues over the years, as many people do in HPC, but books like these would have been invaluable as part of a dedicated semester-long course.
Related: Jorg Arndt's "Matters Computational" book and FXT library - https://www.jjj.de/fxt/
I'm a very experienced Python programmer with some C, C++ and CUDA doing application level research in HPC environments (ML/DL). I'd really like to level up my C++ skills and looking through book 3 it seems aimed exactly at the right level for me - doesn't move too slowly and teaches best practices (per the author) rather than trying to be comprehensive.
I was looking for range-based for loop, std::array and std::span and happy to see that they are all there.
Because this book relates to HPC, I'd add a few things: Return Value Optimization, move semantics, and in the recursive function section a note about Tail Call Optimization.
As a beginner level material I can highly recommend it.
1) Discovering Modern C++: An Intensive Course for Scientists, Engineers, and Programmers by Peter Gottschling - Not too thick and focuses on how to program in the language.
2) Software Architecture with C++: Design modern systems using effective architecture concepts, design patterns, and techniques with C++20 by Adrian Ostrowski et al. - Shows how to use C++ in the modern way/ecosystems i.e. with CI/CD, Microservices etc.
Optional but highly recommended;
a) Scientific and Engineering C++: An Introduction with Advanced Techniques and Examples by Barton & Nackman - Old pre-Modern C++ book which pioneered many of the techniques which have now become common. One of the best for learning C++ design.
- Modern object initialization using {} and ().
- std::string_view
- std::map
- std::stack
- Emplace addition of objects to containers like vector and map.
- Smart pointers (std::unique_ptr, std::shared_ptr and their ilk).
- Ranges library.
- Concurrency support library (std::async, std::future, std::thread, locks and the whole deal).
"The Art of HPC", volume 2 > "Parallel Programming for Science Engineering" https://theartofhpc.com/pcse/index.html
FWIW, MPI is only one way to Python for HPC.
ipyparallel will run MPI jobs over tunnels you create yourself IIRC.
A chapter on dask-scheduler, CuDF, CuGraph (NetworkX), DaskML, and CuPy, and dask-labextension would be more current.
Dask doesn't handle data storage for you, so it's your responsibility to make sure that the data store(s) before each barrier are not the performance bottleneck.
Dask docs > High Performance Computers: https://docs.dask.org/en/stable/deploying-hpc.html
Sources of random may be the bottleneck. You don't know until you profile the job across the cluster.
Re: eBPF-based tracing tools: https://news.ycombinator.com/item?id=31688180
And then something about GitOps (and ChatOps), code review and revision, and project resource quotas
After a cursory glance, I can honestly say that if this book were available then, I'd have taken the opportunity.
The combination of what I perceive to be Knuth's framing of art, along with carpentry and the need to be a better devops person than your devops person is compelling.
Kudos to the author for such an achievement. UT Austin seems to have achieved in computer science what North Texas State did in music.
I'm not aware of any other significant lapack-related developments, but I might just not know about them.
Field et al, recent winners of the James H. Wilkinson Prize for Numerical Software.
Field and Goto both collaborated with Robert van de Geijn. Lots of TACC interaction in that broader team.
I would have thought, that after some 30 years of high performance distributed computing, the requirements were well known and at least the protocol for command and data exchange could be fixed. Apparently not so.
There have been some attempts to standardize basic job management APIs in the past - DRMAA being one noteworthy example. Although DRMAA v2 was only ever implemented by Grid Engine, and is effectively an lightly-abstracted version of their internal APIs, that has never really seen first-class adoption by Slurm/PBS/LSF.
For Slurm, the REST API is meant to be the way forward. It punts the authentication problem to, potentially, anything the admins may care to wire up through an Apache / NGINX proxy. And the basic job submission and status APIs have stablizied to the point that a client application should be able to consume nearly any version going forward.
https://github.com/VictorEijkhout/TheArtofHPC_pdfs/blob/main...
And with volume 3, such a contrast: the author teaches C++17 and... Fortran2008.