Hacker News Reader

Comment by catiopatio | Hacker News Reader

DWARF is not just the better alternative, it’s the only correct solution.

Not only do you get more complete and correct backtraces with DWARF, you also get preservation of stack-saved registers and the ability to reconstitute a subset of local state in a given frame.

libunwind is MIT licensed, not GPL.

I’ve also written a full DWARF stack unwinder myself, including a full DWARF expression interpreter — it’s nowhere near 10k lines of code.

View on HN

It's "correct" in the sense that if it works perfectly it gives you all the information and is strictly better than frame pointers in the amount of information it gives. In practice, the thing a profiler really wants is basically just a backtrace, and fast. DWARF isn't particularly great at producing these reliably or quickly, because compilers will mess up and the format is complicated.

Isn't correct unwind information a precondition for handling C++ exceptions? It seems like a compiler that produces incorrect unwind information would display incorrect behavior when a C++ exception is being caught.

My assumption is, having not looked very closely at it, is that in cases where correct unwinding matters for program correctness compilers will make sure things work, and everything outside of that rots into not being reliable because it’s not tested.

TFA actually gives an example where DWARF alone doesn't work.

It doesn't give any insight as to why, though, which makes it very weak evidence toward whether DWARF is the real problem.

It's not so much DWARF is the problem as it is that the information is just lost in omit-framepointer mode. You can't reconstruct it from nothing.

There's a related problem of compilers generating imperfect debuginfo that fails to reconstruct local variables from registers/memory due to an incomplete DWARF representation, but I don't think that plays a roll in unwinding. (But maybe something similar does.)

What information is lost? The call stack can't be lost or you'd crash immediately.

It's lost in the translation to DWARF. Of course the program knows how to unwind its own stack correctly, but explaining how to do this for arbitrary programs is untenable without DWARF to help, and it's annoying to parse and compilers don't emit it correctly.

If the information is lost by DWARF then I'd say that DWARF is clearly the problem.

Unless DWARF can handle it fine and compilers are thoroughly broken, in which case it's a tossup between "DWARF is fundamentally flawed" and "compiler designers are lazy for some unknown reason"?

DWARF can handle it fine, it’s that compilers don’t generate correct information or the tools to parse DWARF don’t do it properly.

The problem is that with perf the stack trace is collected by the kernel (when the interrupt from the performance counters fires), and the kernel doesn’t implement dwarf unwinding of user space code.

It _could_, but the kernel folks have made it pretty clear they don’t want to put any code related to dwarf in the kernel.

There are tools which do dwarf unwinding with eBPF but I have not used them myself.

They give an example of where perf was unable to decipher the stack using DWARF; it only begs the question of whether the issue is perf or DWARF. It does show that frame pointers make things easier if all you care about are named call traces, but nobody disputes that. (It could also be the case that enabling frame pointers had the side effect of preventing certain optimizations. I wouldn't be surprised if flame graph generators use frame pointers (e.g. libunwind) preferentially, and using DWARF on that same binary would have resulted in the same graph.)

DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place.

Is DWARF too complex? There are many layers to DWARF, and there are compromises everybody could make so we can get at an optimal balance for which DWARF info we should ensure is always there. And we could also spend more time fixing tooling so it works properly with that minimal debug data.

> it only begs the question of whether the issue is perf or DWARF

AFAIK there are two issues at play here.

1. There's a bug in the Linux kernel where it sometimes return an invalid RBP register through `perf_event_open`.

2. The compiler doesn't generate the necessary DWARF info to support unwinding asynchronously, so depending on where exactly you land in a function you might not be able to unwind the stack. (This is why unwinding from *within* the program always works correctly but unwinding from *outside* the program with perf doesn't.)

Source: I wrote my own sampling profiler.

I would be interested in taking / motivating a deeper dive into the failure modes of -fasychronous-unwind-tables, would you happen to recall a scenario where the undwinding problems you discuss reproduce ~consistentlyish?

asynchronous-unwind-tables is not the default. Ever since I enabled it, I have never seen gdb fail to unwind.

It was quite a long time ago, so unfortunately at this point I don't remember what exact problems I encountered with `-fasychronous-unwind-tables`. (I think it might have been completely broken and did nothing on 32-bit ARM? But I might be misremembering.)

And there's of course the issue that you'd need everything to be compiled with it (because the program you're profiling is going to call into external dynamically linked libraries), which on an average system you most likely won't have.

I suspect the dwarf issue from the post is just missing debug info for some library, or too deep stacks.

But leaving that aside, for profiling, dwarf is just very expensive (perf, where the data volume explodes, due to copying stacks) or not available (bpf stacks), because the unwinding has to happen in the kernel. Even if the kernel had a dwarf unwinder, it's vastly more expensive to do that, compared to unwinding via frame pointers.

Try running a system wide profile on larger and busy box. Dwarf based call graphs are basically not usable. The profile quickly is ginormous, and viewing profiles is extremely slow.

> DWARF is useful for so much more than flame graphs. Flame graphs are cool and useful, but they're over hyped. Anybody who has ever spent any significant portion of their career debugging compiled binaries knows that proper debug information is infinitely more useful than flame graphs could ever be. Flame graphs are also useful... very useful. But that's my point: if they advocated even half as much for ensuring everybody made debug info more readily accessible by default as they did crying for frame pointers, the world would be a much better place.

I want both really badly but the fact is that flamegraphs drive performance optimizations which save companies millions of dollars, while symbols sometimes fix bugs and sometimes get blamed for making their software easy to reverse engineer. You can understand why one gets more focus than the other.

Seems like there's a strong difference of opinion on this, the article's author also says

> The thing is that DWARF unwinding (the only userspace option that doesn’t use frame pointers) is really unreliable. In fact it has such serious problems that it’s not that usable at all.

I'm not that familiar with the problem space - can you elaborate on why DWARF is better? or speculate on why the author is experiencing (apparent) reliability issues with it?

Reply on news.ycombinator.com