Hacker News Reader

Comment by ho_schi | Hacker News Reader

GDBs built-in reverse debugging: https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial

I assume rr provides more features and flexibility. Anyway I want to mention that GDB itself can already reverse debug for some time now.

View on HN

If you want to mention this, then you very clearly haven't actually tried it. The implementation in GDB is more convenient than rr (you can start/stop recording at will), but it is also orders of magnitude less efficient. It's only usable for very small code snippets. Otherwise it takes effectively forever and/or runs out of resources.

    > runs out of resources

RAM? What kind of dev box runs out of RAM in 2024? I built a 64GB RAM dev box during COVID-19 crisis. I have never once come close to using all that RAM, even with a squillion Chrome tabs open.

Still, thank you to share your first-hand experience. Did you ask the GDB Dev team for any feedback on the slow performance?

Uuh, any kind of dev box that requires more RAM than available? I promise you that storing that much data about the runtime can be really memory consumptive.

umm, embedded devices without active cooling f.e. those node-b’s sitting on cellphone towers come to mind here, there can be quite a few other similar examples i can think of.

gdb record and replay will absolutely eat up your piddly 64 gigs of ram in a few minutes if you just let it loose. it will eat your 12-terabyte hard disk too, it just takes a little longer. the gdb manual has many helpful tips for how to allocate your ram carefully to record and replay, as well as using a disk buffer instead of recording to ram, so the gdb dev team is already well aware of the problem

I have successfully used GDB's build in reverse debugging once, on a platform that rr didn't yet support at the time.

It worked, it helped me track down the bug, but it was painfully slow, I had to do things to limit the size of the input to make it possible to use at all (and thankfully was luckily able to still repro the problem after doing so).

gdb's built-in replay implementation imposes a slowdown of about 10000× on your program, so if you can binary-search to the desired program state in less than 10000 restarts of the program, that will take less machine time than using reverse execution. in fact, the slowdown is large enough that even interactively navigating the debugger close to the right state repeatedly and then restarting the program is often enough

i have been able to use gdb's replay functionality usefully because i had an input file which crashed the program within a fraction of a second after startup. this meant that i could navigate backward from "this variable is wrong" to "how did this variable get set to that wrong value?" in only several minutes of waiting on the computer

rr predates the one in gdb if I am not mistaken

Actually the gdb implementation predates rr, but (as an rr maintainer) I have to say that it is vastly inferior to rr. It's about 1000x slower than rr, and can't record across system calls or multiple threads or processes. It's so limited it's really a different feature.

Thanks. Can you explain why rr is so much more efficient?

The two approaches are completely different. gdb singlesteps the program and, before each instruction, records the state of registers and memory that will be changed by that instruction --- an undo log. Then you can reverse-execute an instruction by restoring the state from the undo log. This is incredibly slow because singlestepping requires a full context switch for each instruction executed.

rr, on the other hand, intercepts all system calls and other sources of nondeterminism but regular CPU instructions execute normally with no overhead. The details about rr are here: https://arxiv.org/abs/1705.05937

rr was introduced in 2014 [1].

gdb reverse debugging was introduced in 2009 [2].

You can see a fairly comprehensive history of time travel debugging here [3].

Not to say the built-in gdb reverse debugging was any good. It had (has?) like 1,000,000% overhead which is basically unusable. At least some implementations in the history that were introduced earlier only had ~1,000% overhead or less in general. Yes, a literal 1,000x overhead difference.

[1] https://robert.ocallahan.org/2014/03/introducing-rr.html

[2] https://www.sourceware.org/gdb/wiki/ReverseDebug

[3] https://jakob.engbloms.se/archives/1564

Thanks for the info and links :)

Reply on news.ycombinator.com