Core dumps, feedback loops, and a game of telephone

The rumor at the time was that they actually printed them out and went through them with a black marker.

But now that I think of it, it couldn't have been true. With Courier 10pt you can fit about 4KB per page. So a 1GB core dump would would 256K pages!

Maybe they weren't examining anything, and it just took that long for them to sign off and allow the core dumps to be sent.

Author here. Unfortunately, I don't actually know what happened to Ray...and it was long enough ago that I've forgotten his name!

It's quite common to use crash dumps. I run a game server, we compile our release builds with debug info and deploy them to production. Sometimes the server runs for hours before crashing so we can end up with a pretty big dump. When the server crashes, we grab the dump and the correct debug symbols and we at least get a call stack to investigate and (hopefully) come up with a reproducible case.

At a previous job, we developed a desktop application. When the application crashed it would allow the user to submit the core dump to our crash system. We would then have a crash dump + application version and we could investigate customer issues to see if there was a bug or something like out-of-memory issues.

Fairly standard practice for a lot of shops. I haven't seen it for web applications since it's mostly dynamic languages with exception handlers that ultimately trap the exceptions (e.g. sentry).

We are currently building a core dump based scheme to monitor crashes on our deployed devices. Devices use a different architecture than our development environments, and the deployed devices are often accessible only with limited bandwidth, so a live shell session can be more painful than waiting for the core dump to come back.

I don't think dealing with core dumps is that uncommon if you work close to the infrastructure level and the environment comprises many different technologies and high machine count.

In my case of high performance Java, core dumps were immensely useful for investigating issues like JVM bugs (C2 compiler bugs, GC bugs, classloader bugs) but also kernel bugs and hardware issues as well.

> What kind of jobs require such experience aside from compilers?

I work on an SD-WAN router (though on the "systems" side of things like configuration and upgrades, not routing or anything like that). The vast majority of the code we write in is C++, so when we get a crash, basically all we have are logs and core dumps; there's no stack traces in there for us!