Drgn: How the Linux kernel team at Facebook debugs the kernel at scale

I say this as someone who uses and has used linux as a daily driver for years now but honestly I don’t feel good about the future of the platform in the medium term at all.

If anyone has some time to throw at understanding why this is probably the single most sober and rational set of arguments I’ve found of the topic yet https://m.youtube.com/watch?v=WtJ9T_IJOPE

The kernel panic one started when I got pinged by a downstream team that we somehow delivered garbage data (valid-looking but entirely the wrong schema) to them, and that broke everything. Tracing things back up the chain, I found out that the source machine had had a kernel panic. Turns out that ended up with file metadata flushed to disk, but not file data. The unwritten sectors happened to contain data from a different process that was in the right file format, but had the wrong payloads, and since the file format was garbage-tolerant and self-synchronizing, the processing from that point on cleaned it up into data that looked completely valid... but was just from entirely the wrong place. All that happened automatically upon reboot from the panic. I was able to quite conclusively prove that this is what had happened by observing the data offsets logged to debug logs, and noting that the funkiness happened on filesystem block boundaries, and also aligned time-wise with the kernel panic, and a process that matched the schema of the data that ended up delivered had been running simultaneously on the same machine. This was all from just one instance of the fault, forensically debugged; we never reproduced it. The conclusion was basically "working as intended"... that is, this is apparently a thing that can happen given the filesystem mount mode in use (which was chosen for performance), so I think in the end they introduced some sanity checks on the data shape as a stop-gap. I think this was all soon deprecated and replaced with a system that didn't use local disk at all anyway, so there wasn't much point in introducing a big refactor to eliminate this rare failure mode at that point, but it's interesting that it did have security implications (it was just rare and not directly triggerable). It was very satisfying being able to trace down exactly what had happened (and the team lead sent this one off to me specifically due to my reputation for working this kind of stuff out... :) ).

The other one had to do with Google's NIHed gzip implementation, because of course they have one. We had found data mismatches between output computed over identical input in two different data centers (for redundancy/cross-checking), but it always went away if we tried again. This happened once a petabyte or so, give or take (I'll let you guess how often that was in terms of days, at Google scale). Digging through the error logs, I found that the mismatch was an offset mismatch, not a data mismatch. These files had gzip-compressed data blocks. Investigating the data was tricky because we weren't allowed to copy parts of it to workstations due to privacy reasons, but doing some contortions to work in the cloud I found out that the logical data was equivalent, but the compressed data was one or two bytes larger in one of the files, hence shifting the offset of the next block. Eventually I managed to grab one of the problem-causing uncompressed data blocks, build a test case for the gzip compressor with exactly the same flags used in production, and I had the idea of running it under valgrind... and that's when it screamed. Turns out the gzip compressor used a ring buffer of history to perform searches for compression, and for performance reasons the lookups weren't always wrapped, but rather there was a chunk of the ring buffer duplicated at the end. This was calculated to never exceed how much the pointer could fall off the end of the buffer. But, on top of that, the reads used unaligned 32-bit reads, again for performance reasons. And so you could end up with the pointer less than 4 bytes from the end of the buffer padding area, reading 1 to 3 bytes of random RAM after that. The way the compressor worked, if the data was bad it wouldn't break the compression, but it could compress a bit less. And so, the chances of both the read falling off the end of the buffer and happening to be one that would've improved compression were tiny, and that's how we ended up with the rarity. We never saw crashes due to the OOB read, but I think I heard some other team had been having actual segfaults (I guess the page after the buffer was unmapped for them) and this fix resolved them. I think we had the problem going on sporadically for a few months before I got sick of it (and I'd managed to rule out the obvious "it's a RAM bit flip / bad CPU / something" things you tend blame rare random failures on at that kind of scale) and spent a couple evenings figuring it out.

This isn't unique to big production systems in any way; while the numbers may have a few extra zeroes and the pieces of code may be more disparate and numerous, it's exactly the same process I go through today when debugging a kernel issue (these days I work on Linux support for reverse engineered Apple Silicon platforms). You start at the point where the problem happens, then add debugging statements and dig through the call stack, gaining an understanding of the system as you go, until you find the root cause. The good thing about Google, at least in the field I worked in (production SRE), is that usually there was enough logging already, in different forms and at different levels, that you could work things out without resorting to patching code to add more. But digging through the stack like this is certainly a skill, and it applies regardless of the scale of the system.

This skill is the crux of a fairly classic Google interview question: "You type www.google.com into your web browser and hit enter, what happens?" The goal there isn't to test your knowledge of any specific part of the puzzle, but rather to see if you have an understanding of how the puzzle fits together at some level, and hopefully enough of an interest in some part of it to go into significant detail. If you're a networking person, you might start talking about DNS requests, the TCP 3-way handshake, TLS handshakes, HTTP headers... If you're a web person, you might start describing how HTML is progressively loaded and resources are fetched in the background, how CSS lookups happen, how the DOM works... If you're a systems person, you might start talking about input events from the window manager, system calls to initiate network operations, caching to local filesystem... And if you're a smartass like me, you might start "At first, nothing happens; the microcontroller inside your keyboard is likely sleeping waiting for an interrupt that will have it advance to the next column in the scan matrix. Eventually, once the timer expires and the interrupt fires enough times to reach the column corresponding to the Enter key, it will bring that column to a logic high level, and the voltage will travel through the Enter key switch into the corresponding row line. The microcontroller will sample this line a short time later, and detect that a key has been pressed. It will then build a USB HID keyboard report packet containing the scan code, which will be placed in a buffer. The USB device hardware will then wait until it receives an INTERRUPT IN packet from the USB host, addressed to the HID interface endpoint responsible for delivering input reports..." (if I really wanted to frustrate the interviewer, I could start by describing a generic clock oscillator circuit and how the timer countdown logic might be implemented in terms of logic gates and transistors... but that might be going a bit too far, and we probably wouldn't get to the USB packet before the interview was over :-) )

All of those are valid answers, and the idea is that hopefully you can dive through some subset of the stack. So if you're interested in this kind of thing, that's a good place to start - pick something you're at least vaguely familiar with, and start digging through the stack to see how much of an understanding you can get of the different bits and pieces. Doing this with an open source OS is invaluable, since all the code is right there at your fingertips - there is nothing you can't figure out given enough time and patience. If you find interactive environments with a REPL more intuitive, you might use a debugger, set a breakpoint at the top level of the code, and single-step your way down the stack. If you're more of a reading code and printf() debugging person like me, you might do it that way instead. If you're interesting in digging downward towards the metal, perhaps try working with microcontrollers (e.g. Arduino, but make sure you dig through the libraries, don't stop at the interface - it's not that hard to learn how to program 8-bit microcontrollers in assembly!), and if you want to look at larger distributed systems, perhaps take a look at something like Ceph or some kind of Kubernetes deployment (both of which can be tested in a single machine with containers or VMs).

https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

This one isn't bad either:

https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/

The rule of thumb is that the vast, overwhelming majority of developers are working on business apps, especially Line Of Business (LOB) internal apps nobody ever sees.

It's hard to get exact numbers for this stuff but I wouldn't be surprised if the ratio of base infrastructure developers (kernel, shell, drivers, compilers, runtimes, frameworks, libraries etc.) to app developers is 1:100.