Introduction to Sentry Symbolicator
getsentry.github.io
getsentry.github.io
It's written in Rust, originally in actix, today in axum.
Are there any interesting bugs and edge cases that you could share? Always good to hear good war stories from production :)
Too many to count, particularly the PDB format has so many absurdities in practice that I'm quite impressed they are so well supported even outside of Microsoft's ecosystem. For me one of the most fun aspects is that Microsoft's kernel tools instead of fixing up PDBs when performing certain optimizations on the object files, chose to instead attach additional maps in the PDB (OMAP) to remap everything. That decision makes handling of PDBs much more tricky than they otherwise would have to be.
In terms of actual bugs, there are too many to count. One of the more fun one in recent history is that Apple's tools are running up against limitations of some remaining 32bit offsets even in 64bit macho formats and now at times just overflows and you're stuck.
AFAIK there's still no good solution to getting a useful Android stack trace that starts in the ART and crashes inside native code other than the unwinding on the device itself. But even that is a crapshoot.
The thing that worked the best for us was pulling the stacks that Android dumps into logcat just before terminating the process.
I really don't understand why Apple and Google both make it so hard to get a useful stack trace. Especially when the OS itself is typically already capturing it out of process.
Generally it seems like most companies have no interest in making it easy/possible to make stack traces in production. Google clearly is running into that themselves, and they are having some support in Android for it, but they don't like to share that. Even today we ship multiple different unwinders on Android to cover all bases.
For iOS (I know I'm telling you stuff you already know) you used to have to plug an Apple device into a Mac running Xcode, let Xcode copy over the symbols, then you could upload those to your internal symbol service. But at some point Apple at least stopped encrypting the firmware so you could skip the Xcode step and extract the symbols directly from the firmware. So various folks have just been caching copies of them. e.g.
It's a bit of a cat and mouse game still.
> how its supposed to be used by engineers in their day-to-day?
Most people would not use tools like this day-to-day unless they build tools themselves that need to process this at scale. If you are building a profiler, debugger or else for instance you might find this useful.