Free MIT Course: Performance Engineering of Software Systems
ocw.mit.edu
ocw.mit.edu
In large software systems it's not obvious at all where your hotspots actually are and what you actually should do to take care of them. Further, once you do an optimization, you need to be able to make the right measurements to prove that your optimization actually worked. A lot of optimization also involves parameter tuning (eg how many threads to do what), and that is incredibly dependent on having good measurements.
Once you know what to do, the actual optimizations you need to do - even assembly-level tricks - are usually somewhat straightforward in comparison to the measurements you do to get there. If you don't do the measuring, you are going to end up doing a lot of high-effort, low-return work.
See the following paper for an example of real life performance engineering, where the engineers involved slowed down their own code to create a significant improvement in general application performance: https://www.usenix.org/system/files/osdi21-hunter.pdf
As another example, performance engineering in trading systems often involves figuring out how to do non-invasive measurement of events that cause systemic tail latency so you can find the bottlenecks and slow parts. If you do the hamfisted things that most engineers think of ("let me chuck performance counter checks everywhere"), you will prevent your system from making money and often destroy the usable signal.
Your trading system example sounds like a great topic for a masters thesis or similar graduate level work (or better yet - industry), not a core component of an introductory performance engineering class.
This is a very simplistic view of how software measurement works that is pretty pervasive in academia, but doesn't actually translate all that well. If you're sticking to only typical profiling methods (which tells you very little about things like latency and doesn't tell you at all about many sources of slowness), you still need to find a way to do it in a low-impact way if you actually want to measure a software system of any complexity. As an example of another technique, tracing (and even just reading logs) can give you a lot of interesting signal that profilers don't generally capture.
Most commercial software systems have a number of lines of code (and a code profile, by the way) comparable to the Linux kernel, so if you're just going to apply a simple profiling methodology to it, you're going to get a lot of crap data and you're going to slow things down a lot. Performance engineering is about extracting signal from the crap.
> The course involves assembly because it’s taught in C, but the core concepts apply to any language
I never said I wrote assembly, I said assembly-level code, which you can write in most languages. You can write it in Python if you are skilled enough at Python. Many people do it in Java and Go. I usually do it in C. Most of this kind of code these days is not actually written in assembly.
> Your trading system example sounds like a great topic for a masters thesis or similar graduate level work (or better yet - industry), not a core component of an introductory performance engineering class.
The primary point I am trying to make is that a class called "micro-optimization" should be teaching you how to micro-optimize code. A class called "performance engineering" should nominally be teaching you about how to actually do performance engineering, which is actually not all that related to micro-optimization.
The class is not about micro-optimization. The first couple lectures are. But people in this thread love to read their titles and stop there. I don’t get it. Here are the lecture notes for measurement and timing, you’ll notice they include what you say about separating signal from noise. I’m sure you, with tons of industry experience behind you, know more than this introductory undergraduate’s lecture in the topic provides (it’d be quite sad if you didn’t!), but that does not mean the class does not indeed provide an introduction to the topics. One that, again, I have personally built upon to dramatically speed up countless medium-to-large scale systems in a wide variety of contexts. https://ocw.mit.edu/courses/6-172-performance-engineering-of...
(Though I’m currently backpacking through the Andes so you may need to be patient)
Is the end of semester Leisserchess competition still going? I believe I heard that the year you TA'd it (might have been a year before or after) a group finally compiled an opening playbook and beat everyone in the class
I’m not sure what you mean by “finally complied an opening book”, by the time I was involved with the class opening books were table stakes and we even included code to generate them as part of the starting distribution. They’re somewhat useful, but effective culling of the search space in multithreaded contexts is far more important. The opening book can only ever help for the first few ply, after that the engine which can consistently think a move ahead will likely win. (Indeed the staff designed the rules to optimize for that very property). A quick and informative heuristic function is of course also critical.
Here are some other resources for people interested in improving their understanding of performance:
- "Computer Architecture: A Quantitative Approach" by Hennessy and Patterson. The "Computer Organization" book by the same authors is also good. Both of these resources were extra useful when building deeper intuitions about GPU performance for ML models at work and in graduate school.
- CMU's "Deep Learning Systems" Course is hosted online and has YouTube lectures online. While not generally relevant to software performance, it is especially useful for engineers interested in building strong fundamentals that will serve them well when taking ML models into production environments: https://dlsyscourse.org/
- Compiler Explorer is a tool that allows you easily input some code in and check how the assembly output maps to the source. I think this is exceptionally useful for beginner/intermediate programmers who are familiar with one compiled high-level language and have not been exposed to reading lots of assembly. It is also great for testing how different compiler flags affect assembly output. Many people used to coding in C and C++ probably know about this, but I still run into people who haven't so I share it whenever performance comes up: https://godbolt.org/
The statistical parts are very good: self contained and with good explanations. Perhaps a weakness is there isn't much on distributed systems.
He did some Youtube videos as well.
If you want to improve the performance of some random software system you encounter at work, there is normally much lower-hanging fruit available.
First profile the application on as representative a workload as possible. Then fix the accidentally-quadratic loops, stop it from hitting the disk every iteration, add indexes for the worst database queries. If that doesn't at least double your throughput, you are working on an unusually-good application.
If you're lucky then it's CPU bound and everything runs on the same box and you can just look in htop.
Yeap, usually it's more about properly profiling your application on the CPU and memory dimensions, occasionally instrumenting if necessary, experimenting different measurements and tools to have a good picture of your software bottlenecks and where there are gains.
Knowing the full picture and where the issues are you can then proceed to these specifics of optimizing your code for performance in these spots, or using a specific library that solves better that problem or what is usually an approach that gives better gain, rethinking knowing the limitations and finding a better design for the problem that part of the code is trying to solve.
But it’s an 18 unit course, it’d be a disservice to the students (and the world at-large, considering the potential impact of performance engineering) if we stopped there.
(I TA’d this class)
Introduction and Matrix Multiplication
Bentley Rules for Optimizing Work
Bit HacksBut yes, I do agree profiling is step 0 to performance engineering.
I 100% get why you're saying it: the fear is that with all of these contraptions in mind, we send a generation of engineers out who immediately reach for their performance bazooka when all they needed was a pocketknife. But SOMEbody is doing actual high quality engineering and needs to reach for this stuff. It's MIT: there's a chance those people are in the room.