Hacker News Reader

Comment by menaerus | Hacker News Reader

Has anyone used TTD, or rr for that purpose in Linux world, to debug a complex multi-threaded application? What is the main use-case of these tools?

For example, limitations of rr seem to suggest that it is almost of no use for multi-threaded programs so I have never actually tried it. I don't know about the TTD though.

  > rr limitations

  > ...

  > emulates a single-core machine. So, parallel programs incur the slowdown of running on a single core. This is an inherent feature of the design.

View on HN

It can still debug multi-threaded programs, but if your bug relies on multi-core interactions it won't appear under rr (you can still catch a lot of race conditions though, and it has a 'chaos mode' which generate irregular scheduling intended to increase the likelyhood of such bugs appearing). It's been used to debug race conditions as well as other bugs in firefox, for example.

That's what I thought as well, limited to the single-core race conditions era. Thanks for the evidence.

It's still much better than the alternative, which is not having any kind of way to reproduce race conditions :-) QEMU's a pretty complex multithreaded program and my experience with chaos mode has been that it's quite good at tripping up the races (and you only need to catch the race once).

It's not limited to any era - it's a general-purpose tool that can perfectly reproduce bugs, even most race conditions. The lack of multi-threading will mostly just slow things down. Being able to perfectly reproduce a bug is immensely valuable.

I didn't say anything about the usefulness of rr other than in the context of concurrency bugs so I don't quite understand a defending attitude of yours.

> limitations of rr seem to suggest that it is almost of no use for multi-threaded programs

Only if your main use case is debugging race conditions only possible with multiple cores, which is a tiny subset of all bugs.

I use rr all the time for a complex multi-threaded (although not extremely parallel) application and it works wonders. It frequently saves me hours of debugging. Practically none of the issues I use it for are race conditions (not because it doesn't work well for those, but because I rarely get any).

Even if I had to suffer an 8x slowdown from running it on a single core, it would still be worth it nearly every time.

(Disclaimer: rr maintainer.)

As others have said here, in practice rr works very well for debugging a wide range of bugs in multithreaded programs, including race conditions.

Where it falls down:

* It can only use a single core, so highly parallel programs run very slowly when recorded by rr.

* Some race conditions may be difficult or impossible to reproduce under rr recording (but rr's chaos mode helps a lot with this).

* rr imposes sequential consistency on the recorded program, so bugs due to weak memory models do not show up under rr. Such bugs are pretty rare on x86 (because x86's memory model is pretty strong); this may be more of an issue on ARM.

TTD (the WinDbg one) works very well for complex multithreaded apps. (The caveat being the performance hit you get from emulation). It's one of the big advantages it has over rr.

The main use case I saw for TTD was debugging complex memory corruption issues. Certain types of issues like stack corruption became trivial to debug under TTD. It was also very useful for capturing a repro. If a customer complained about something and I couldn't immediately reproduce it or get a crash dump, I'd ask them to record a TTD trace. More than 75% of the time I'd say it was enough to root cause the bug, without spending tons of time figuring out the repro steps.

Reply on news.ycombinator.com