Mozilla's low-overhead open source replay debugger | Hacker News Reader

Mozilla's low-overhead open source replay debugger

robert.ocallahan.org

272 pointsbzbarsky44 comments

Another gem from Mozilla - I am astounded at how much good stuff is coming out of Mozilla in such a short timespan. Keep it up folks. Between this, rust, servo, asm.js and emscripten - you have my attention.

I don't know about the author but not all of us oppose Gay marriage.

Please don't confuse the personal views of the CEO outside of work with the views of the entire organization. There are a lot of genuinely good people at Mozilla doing good work, and I would bet that the overwhelming majority working there do not share the same views as the CEO. You're comment not only does a disservice to yourself and the HN community, but everyone else at Mozilla working to make the world a better place.

Was submitted yesterday: https://news.ycombinator.com/item?id=7458554

It seems to me that a large portion of cool stuff doesn't make it out from /new any more (not just this case, many others lately)

I was tracking a few of my own submissions - I think it's random. IF you happen to arrive on the frontpage and stay there for a few minutes, a reasonable amount of people will vote that up. However, if you stay on the front page for too short or not at all, the random fluctuation might kick you down. This is e.g. raspberry pi foundation linking to pypy blog vs submission of the pypy blog itself.

I think it would be great to show one random "new" submission at the bottom of the front page on every page load (it could be marked as "just submitted"). Of course spammers would love that, but I think it would make the discovery process much better, and the end result would be positive.

Deleted comment

Not true, it really is fairly random. Usually if it's cool to you it will probably be cool to a bunch of other people. However, the chance that those other people are in a position to upvote it is pure chance. One it gets a bunch of upvotes, then it will stay around in a position that enough similar people will see it and keep voting. Getting those initial upvotes are pretty much pure chance though.

http://rr-project.org/rr.html is a presentation titled "How rr works" from Mozilla Research, with more details on the implementation.

...but that's great.

Wow, this is a wonderful thing.

I would go as far as to say it is the tool every developer wanted and most thought was impossible (the low overhead part at least).

Only recording the nondeterministic parts and replaying the rest is so obvious, once someone else has thought of it.

except me of course cos I don't write bugs

It looks like the presentation is still a work in progress; many of the later slides just have "TODO: example" or similar. But it does contain some interesting content.

I've been very interested in that topic for the past years.

We've implemented an in-kernel linux multicore record/replay engine. See for more: http://velvetpulse.com/2012/11/27/scribe-the-deterministic-t...

We also did a mutable replay engine, to replay on modified program (e.g. replay a bug that happened in a production environment, but this time with the logging verbosity turned into debug mode). Check out Dora: http://viennot.biz/asplos2013_dora.pdf

This is just a bit too awesome, yet simple.

Think of it as the "git" for debugging. Take a snapshot, keep going etc ... and then backtrack back to the point where it all started going wrong.

Just very hard to automate easily, yet this uses gdb and all the other macro/command goodness (well, for power users) that gdb comes with.

An amazing project. Would it be possible to get to a certain point in a trace, then switch the process back into "live mode" so that further inputs could be made into the program? This could open up all sorts of interesting things, like recording interactions with web applications and then allowing a developer to open up high-level JS/DOM debugging tools to see exactly what went wrong without needing to worry about replicating the exact input sequence that led up to the bug.

It's unfortunate that the no-shared-memory limitation is so hard-baked in, so it would seem that things like Chrome rendering processes wouldn't play nicely with rr (I'm assuming they communicate with shared memory, correctly if I'm wrong!).

As I understand, you can't "go live" with the current architecture. In replay, no system call is executed. Write is no-op. Read is done from the trace. This is how PTRACE_SYSEMU works. To "go live", you would need to actually execute system calls.

http://sysemu.sourceforge.net/

Re: Chromium IPC. "Channels are implemented using named pipes on Windows, and socket pairs (or in some special cases unix domain sockets) on POSIX."

http://src.chromium.org/svn/trunk/src/ipc/ipc_channel.h

Hmm, was afraid of that re: SYSEMU. Still, I'm sure people will find ways to do higher-level debugging in rr with the current architecture!

This looks very cool, and will undoubtedly help with most run of the mill crashes. Unfortunately the machine model it uses is single-core, so you're not going to necessarily catch the really subtle concurrency bugs. Good multicore record and replay is hard, and has been the subject of tons of papers in the past 10 years. It would be cool to see some of that research make it out into practice.

This looks really interesting. I hope it doesn't get buried in the Facebook buys Oculus news tsunami.

I can't wait to try rr out.

"rr currently only supports x86 32-bit processes."

While the idea of a replay debugger is really nice, I don't understand where this limitation comes from. I thought, that since years we practically live in a 64 bit world.

There are a lot of low-level intrinsics that need to be called behind the scenes to do the recording, and I'm not surprised that they did their first experiments using the 32-bit versions, since they might be better documented, or there might be more expertise on the team if they've been working on 32-bit low-level stuff for decades. All those intrinsics should have 64-bit versions, though, so it's just a matter of time before 64-bit support lands. I'm sure they'd welcome pull requests!

You can use rr on 64-bit systems. rr requires 32-bit processes at the moment, but does not require 32-bit systems. 32-bit application on 64-bit kernel is a supported configuration.

> I don't understand where this limitation comes from.

Time and money? Such as scheme requires a lot of very low-level ops (including things which are not available on all architectures), x86 is simpler and better-understood than x64, starting the project there seems as good a choice as not.

Yes. This. This sounds like a great project and I would love to integrate this into our debugging workflow but this one limitation basically kills that prospect.

Well, it is version 1.0. If it is that important, maybe you could port it and issues some pull requests.

The latest gdb also contains a reversible mode. Anybody can comment on how does it compare with rr?

I have only tried myself the reversible debugger in ocaml, and it was an eye opening experience.

gdb reversible mode is useless for anything other than a toy example. We never got to the point where you can e.g. start the PyPy process, not to mention do anything with it. We had much more luck with UndoDB, but this one is still ~20x slower than native.

The critical word is "low-overhead". gdb's implementation is orders of magnitude slower.

On the other hand, rr can't do reverse debugging.

The lack of reverse debugging isn't an inherent limitation. One could index the replay and take micro-checkpoints along the way; then when you want to step backward, you rewind to the last checkpoint and play forward until you hit the previous instruction.

Very cool! This looks a lot like something Michael Chastain was working on in the late 90s: http://lwn.net/1999/0121/a/mec.html

It also is similar to GDB's record-and-reply functionality, except that I believe GDB is limited to a single process. I suspect that GDB's approach is much higher overhead also, because I think it actually records state changes for every instruction: http://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-...

Unlike the GDB functionality, this doesn't seem to support reverse-stepping. Hopefully this can be added -- it makes a huge difference when you can reverse-step backwards from the failure to find out where things went wrong: https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution....

In theory, the replayer could be made to rerecord a full memory trace. Running the replayer in gdb would be enough for a single process.

> In theory, the replayer could be made to rerecord a full memory trace.

Yes exactly, or if it's cheaper, create periodic compressed snapshots and replay from the previous one to recreate intermediate states. In my experience GDB's recording is very expensive and very rapidly runs into resource limits. Hopefully a snapshot-based approach would be more scalable.

Link to the project: http://rr-project.org/

And the github: https://github.com/mozilla/rr

(On a side note, I think the gradient background is worse than a plain white background, when removing it, it become a lot more readable)

This has some similarities with a project I worked on that provided very advanced analysis and debugging capabilities including instruction-level "bidirectional execution". I can't say much more about that, other than the fact that it was for a whole-machine environment, and was a combination of special hardware support, but it's good to see some of the techniques becoming more widely available to the public.

This reminds me of a similar idea I'd like to see. A smart tracer that keeps track of function call args and results and possibly local variables in a cache. Then when your program crashes, it can show you not just the backtrace but also recently called functions that may have influenced the crash. With such a tool you may be able to debug some crashes without a debugger at all.

Replay tool is also for Java. It is called Chronon, is Eclipse plugin and was recently integrated into Idea Ultimate.

Is this something similar to Intellitrace in Visual Studio?

No. IntelliTrace records, but does not replay.

No one's going to rag on the post's author for labeling himself "Christian" right at the top? He might have donated to Prop. 8 after all.

Edit: That's sarcasm, people.

Sarcasm fully understood. So, to address the point you're making underneath the snark:

No one (that I've seen, anyway) is ragging on Brendan Eich for being a Christian. I don't even know whether he is, although it's a fair guess given his decision to support Proposition 8.

Many people are saying that Eich's support of Proposition 8 is a bad thing, makes him a bad choice for CEO of Mozilla, discourages them from using Mozilla's products, etc.

Attacking someone for being a Christian: religious bigotry based on disapproving of someone simply on the basis of their identity. Generally a bad thing.

Attacking someone for supporting Proposition 8: disapproval based on what the person has actually done and its likely consequences. No reason why it shouldn't be a reasonable thing.

This doesn't seem to me to be a very subtle distinction.

(Aside: I do find it curious that of the four people who first come to mind when I think of Mozilla -- Mitchell Baker, Gervaise Markham, Robert O'Callahan, Brendan Eich -- two are rather vocally Christian (gerv's blog is called "Hacking for Christ"!) and a third has become infamous for an action that's probably the result of a similar religious position. Just coincidence?)

The question I have is why it's those four people who come to mind for you. Why Mitchell and Brendan is very clear, since they are the ultimate project leadership. But why specifically Robert and Gerv, as opposed to David Baron or Asa Dotzler or Johnny Stenback or Stuart Parmenter or Mike Shaver or various other people who were involved with the project since close to the beginning?

I have some hypotheses (e.g. Johnny doesn't blog, so isn't as "famous"), but I'm curious whether you have any yourself.

I'm confused, are you saying there's a link between people's beliefs and actions or not?

Of course there's a link. But criticizing an action someone has taken that happens to have been religiously motivated is not the same thing as objecting on principle to their religion.

An example more extreme than Brendan Eich's: Suppose someone is a devout Christian but also rather deranged, and believes that the Bible is full of coded messages telling him to kill (say) school biology teachers. So he goes and kills a bunch of school biology teachers. If I say "That was a terrible thing to do and he shouldn't have done it", that doesn't mean that I am going to hate on everyone else who is a devout Christian -- because, as it happens, plenty of people manage to be devout Christians without murdering biology teachers.

Perhaps (so far as I know this is pure speculation) Brendan Eich is a Christian and the particular variety of Christianity he adheres to tells him to try to stop same-sex couples' relationships being legally recognized. If so, then I think the particular variety of Christianity he adheres to is leading him to do harmful things and I wish he would stop. But that doesn't require me to object on principle to everyone who is a Christian, not least because plenty of people manage to be Christians without trying to stop same-sex couples' relationships being legally recognized.

Now, let us suppose there is a Christian denomination that has opposition to same-sex marriage as a central part of its teaching, and that does not permit anyone to belong to it unless they work to oppose same-sex marriage, including making financial contributions to campaigns like the one for Proposition 8. And suppose it turns out that some particular person is a member in good standing of that denomination. Then I'm quite happy following the following line of thought: This person belongs to that denomination; belonging to it means being a fervent opponent of same-sex marriage and contributing to campaigns against same-sex marriage; therefore this person has almost certainly done things I consider harmful and antisocial; so much the worse for this person. You may, if you please, consider this anti-religious bigotry. If so, I invite you to consider the following two questions. (1) Is it possible for a religion (or some particular version of a religion) to be morally harmful to its practitioners? (2) If you learned that someone was a member in good standing of an Islamic sect that had glorification of terrorism as a central doctrine, would you think worse of them as a result?

It's his personal blog and there's this thing called free speech.

Reply on news.ycombinator.com