Leaving Mozilla | Hacker News Reader

Leaving Mozilla

robert.ocallahan.org

258 pointscurtis103 comments

I've never heard about rr http://rr-project.org/), his tool to record program execution and debug it afterwards. The alleged 1.2x slowdown in execution is negligible. The ability to reverse in time is almost a nirvana. I'm sad I didn't have it when I used to program C++. Hope he succeeds in his new endeavor. We'd all win big.

Does anyone knows of similar tools for Python and in-browser Javascript?

Re: Javascript - If you use the'redux' (http://redux.js.org/) framework you can get something like this (https://github.com/gaearon/redux-devtools-dock-monitor)

I understand you never claimed they are similar. But to be fair (and with due credit to redux for being a great framework), all redux does is save method call arguments in an array. rr is on a different scale altogether.

the python case is interesting ... after all, the interpreter is just another binary executable. perhaps there's a way to write a layer above rr to "step" in Python code rather than the underlying C code. same with JS (Node.js) or any other language run-time.

Python and in-browser Javascript are also programs, in theory rr covers the entirety? But I don't know what the ergonomics are like (e.g., how much will you see program activity in your platform that you don't care about, and how well can you see the activity in the code you are debugging?)

Yes, rr can record python and JS execution just fine, but the debugging experience is hopeless, since right now rr just gives you gdb. With an astounding tolerance for pain you can make progress by calling JS engine functions from gdb, but nobody wants that.

This is fixable, however, and on my agenda.

Sounds very similar Green Hills Software's TimeMachine for embedded applications.

TimeMachine needs a specialized hardware probe connected via JTAG, if I'm not mistaken. The probe is powerful enough to record everything (registers, operations, etc) in its memory without affecting the embedded device. Then you can replay the info however you want, in as much detail as you like.

The feature here is more like software emulation, and it will really allow you to take the instruction pointer in reverse direction. Unlike TimeMachine which records in real time and let's you browse the recorded data as you wish, but after execution has occurred.

My thoughts were very similar to yours. In 1992, when I started out in embedded on a Philips 8051 and debugging it was via an ICE and had direct visibility and control of the 8051.

The ICE would be connected to a bonded out version of the 8051 and I was almost able to capture every single clock cycle of 8051's execution. When I hit a failure, I would just replay the trace and see what went wrong.

It looks like the folks on the big machines have caught up :-)

I had access to a Huntsville Microsystems ICe that used a bog-standard - non-bonded-out - 68008 in harness. I had scripting tools that would capture and replay all the registers to back the machine up.

That HMI was a brilliant machine.

It's really the last debug tool I ever used besides AVR, PIC and one ARM JTAG for doing BSP work.

Mozilla is prototyping a feature called "WebReplay" to support reverse-debugging of JavaScript and DOM mutation:

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/We...

https://bugzilla.mozilla.org/show_bug.cgi?id=1207696

It looks like Microsoft is implementing a similar time-travel feature in Chakra:

https://github.com/Microsoft/ChakraCore/commits/TimeTravelDe...

If I'm understanding this correctly, then Java has had it for ages now via the "Chronon" debugger. It's the only reason why I pay for an IntelliJ Ultimate Edition license, since it comes bundled.

There are other reversible debuggers, but even a mature one like Chronon has more overhead than this. 1.2x is basically nothing.

Not if you have > 1 busy thread.

If you're willing to accept things that compile to in-browser javascript, check out Elm's magical time-traveling debugger: http://debug.elm-lang.org/

For Python my coworker (and friend) Tom has a great project called timetravelpdb: http://everythingsysadmin.com/2014/04/time-travel-pdb.html explains a bit on it, the repo is https://github.com/TomOnTime/timetravelpdb

Basically it's an extension of pdb that allows you to jump back and forth in time.

I'm building something like this for in-browser Javascript, but in a larger context. We're not ready for public consumption yet, but: https://fabric.fm

Interesting! Tried to subscribe but got an error: {"code":-5000}

I was really impressed when finding out about Microsoft's Chakra JS engine Time travelling debugger[1], I think that's what you're looking for, haven't tried it myself though.

[1] https://youtu.be/1bfDB3YPHFI?t=304

No exactly the same thing but Tonic has similar capabilities using CRIU (checkpoint and restore in user space) for Javascript time traveling: http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.j...

It's an interesting different approach because it allows you to rewind the entire machine, so you can interact with interactions between different processes, etc (happy to answer questions, I am a dev on it)

There is something similar in bpython called rewind. Run a few commands in the repl and then try `ctrl+r`.

I wonder how it works. Assuming it must capture state at t=n, and somehow calculate the diff to get to t=n+1. Could theoretically pop diffs to get to any point in the past.

Gdb has supported reverse debugging since 2009.

I've used it, and I've been using rr. gdb's reverse debugging was useful to me a few times, in extremely limited situations.

I often do my initial runs under rr, just in case I might want to debug, because it's useful most of the time and it's not enough overhead to worry about.

gdb's reverse debugging works after you narrow the range of interest down to a very small region, because otherwise its buffer will repeatedly overflow and it'll take ages.

Seems rr is using fork for its implementation. The first time I've seen this approach taken is in Ocaml's debugger.[1]

[1] http://caml.inria.fr/pub/docs/manual-ocaml/debugger.html

Its not free, but there is http://undo-software.com/undodb/

I worked with Robert at Mozilla for many years. He might be the most talented software engineer I've ever met, and he's a strong candidate for nicest person as well. An absolute pleasure to work with. Mozilla will miss him dearly but some other project (apparently rr) just got really lucky.

rr is roc's baby ;-).

From reddit to answer the question some might pose ("who?"):

> One of Mozilla's most senior engineers who has worked on Mozilla projects as a contributor and employee for well over a decade particularly working in the graphics code and helping make native audio and video in the browser a reality.

http://www.reddit.com/r/firefox/comments/48ix55/robert_ocall...

Great! Now how about the next question:

"Why?"

I suppose it's nice to know that he is a big deal but as someone who doesn't follow mozilla religiously a little more background is always appreciated.

Why is he leaving? That's covered in the OP: he wants to devote his full-time attention to working on rr, a reversible debugger for native code that imposes impressively low overhead, and possibly spin it into a startup.

I think roc would give me a sly smile for pointing out that this doesn't really answer the "why". It's what he's going to do after he leaves. Doesn't quite answer why he's leaving.

Because... he wants to do those things? I'm really unsure what you're trying to say.

He just wants to build a startup out of some cool tech and get rich. There's no conspiracy or dirty laundry here.

My, cynical much? Seems to me that 10+ years at a non-profit, and choosing to live in a place outside the traditional startup locations (New Zealand) with fewer networking opportunities, might indicate fairly little interest in that exact trajectory, for those exact reasons.

But what do I know, I'm just another ten-year Mozillian myself. And if or when I leave, it'll also surely be to get filthy rich at a startup, right?

Well, it'd be foolish to assume that someone doesn't have multiple reasons for making a dramatic change in their life.

It'd also be foolish to assume that most of those reasons are any of our business.

Robert is an amazing guy, and we're all lucky to have benefited from his work over the years. Now, he's set off on a new adventure, for reasons of his own... And has hinted that he'll continue sharing some of the fruits of his labor.

That's more than enough ;)

And for those that have no idea what rr is: http://rr-project.org/

So no, he is not leaving for "rest and relaxation"

To be fair, people also think "rest and relaxation" is amazing, and we probably need to capture some of the value of this technology via a for-profit company as well.

I would pay to be able to time travel and replay my rest and relaxation!

Wow, it's the end of an era.

I wish him the best of luck. rr is really awesome and deserves even more attention.

(http://rr-project.org/)

Regarding RR he says:

> we probably need to capture some of the value of this technology via a for-profit company

I hope this doesn't mean making it proprietary. I mean obviously a person gets to decide how to best compensate themselves for the fruits of their labor. But I've seen too many compelling technologies fail to achieve the impact they could have because they were proprietary.

I mean, DTrace languishes despite being open-source, because it wasn't open-source enough

DTrace doesn't languish. It just doesn't have a Linux port, because of people's contrived rewriting of the history of the GPL.

It does, it's part of Oracle Linux. Oracle obviously therefore believe that a CDDL kernel module is fine.

Oracle also owns SUN, who originally licensed it that way...

It wouldn't be distributed as a kernel module (DTrace probes are far more invasive than that). But yes, I posted this before I realised that Oracle Linux has sublicensed it under the GPL in their port to Linux (for their customers), so it's fine for us to do the same as long as we "base it on their port".

Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.

This is a technology that received an innovation award from the Wall Street Journal in 2006. Whatever impact DTrace is having today, it should have been much larger.

You really aught to look at Bryan Cantrill's twitter feed then. It happens frequently enough that he notices blog posts and tweets them.

Then again Cantrill has an axe to grind.

In case others are wondering: https://en.wikipedia.org/wiki/Bryan_Cantrill

I'm no fan of the man, but it doesn't mean that dtrace is not an excellent tool.

Dtrace isn't completely dead; it's used to great effect as the underlying technology behind Instruments (performance analyzer/visualizer) on OSX.

It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.

I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with. The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same:

    /pattern/ { statements }

In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations).

So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned.

This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.

We use it every day at Circonus. One of us is learning something new about how our stuff works every single day.

I use DTrace fairly often to look at the boundaries between whatever interpreted language my app's written in and the kernel.

For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.

I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day.

DTrace is used extensively in FreeBSD development these days.

It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/

My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume.

We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too.

I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.

I use dtrace often, but am too busy to write articles about it, sorry. A couple of times, dtrace has even been THE tool that helped me understand exactly why our big production systems were dying seemingly inexplicable and horrible deaths. This has directly resulted in us still being in business instead of fading into irrelevance. That's plenty impact for me.

> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.

Then you haven't been looking:

http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wi...

http://dtrace.org/blogs/brendan/2014/02/11/another-10-perfor...

https://www.joyent.com/blog/node-js-in-production-runtime-lo...

https://twitter.com/b0rk/status/681536144362848257

https://speakerdeck.com/sartak/dtrace-war-stories

https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/

https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...

Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace

Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)

There's a decent market for selling consulting consulting services on top of open source technology. Being the expert on applying an open source technology to solve specific problems comes with a decent rate card.

You're still selling your labour rather than a product, though. It's possible to be well off doing that, but basically impossible to become ludicrously rich.

He didn't say his goal is to become ludicrously rich. He said he has ideas that Mozilla can't and shouldn't provide resources to realize, so some for profit company would work better for it.

True. There is a potential multiplier when you're selling a product or money, rather than time. It's a matter of priorities and risk appetite.

http://undo-software.com/undodb/ is a rival which i have marketed at me on linkedin frequently

There are multiple ways to fail at spreading technology. One way is to choose a business model that makes it difficult for potential users to actually use the technology. Another way is to choose a business model that doesn't generate enough revenue to bring the technology to its full potential.

Collabora does for-profit work on LibreOffice, but integrates their work into LibreOffice's master. Mostly they even do this fairly openly and don't batch up commits then push them but make their changes small(ish) then push them.

You can definitely be a for-profit company AND work in the open.

Thanks Robert for signing up for StackOverflow just to answer my question http://stackoverflow.com/q/34488237

I'm not familiar with Robert or his work, but I quite liked this post. The brevity and honesty was refreshing. I couldn't help but feel excited at the end!

Reverse debugging has been around for several years, but hasn't seemed to be widely adopted. Which seems kind of unfortunate, as the concept of it looked really useful.

Does rr make reverse debugging much easier than it's previously been in gdb or something? :)

I've tried to use rr on LibreOffice but had no luck whatsoever. :(

Please file an rr issue. I've had it working in the past.

Just saw this - will do!

rr (http://rr-project.org/) is a piece of magic. I've heard lots of people talk about using it in wild and awesome ways.

The second part of a magic trick is to connect it to your audience. Making time-travel style debugging is... not easy, but possible. Making it relevant to people, across languages and other conceptual domains, is hard.

If your next work is tackling this, good luck and godspeed :)

Great work; Great song!

Wonder if that's the end of Mozilla's only southern hemisphere office in Auckland.

I know of at least one other NZ Mozillian (Nick Cameron, on the Rust project) but I'm not sure if he works remotely or at an office.

Pretty sure he lives in Wellington and so must work remotely.

There were about ten NZ Mozilla employees when I worked there a year ago. Not all at the Auckland office but there were plenty there to keep the office utilized at the time.

No. Mozilla's Auckland office has other engineers working on Gecko video playback.

If and only if there is a way to debug network traffic like I am stopping at this router and hop to the next router.

"we probably need to capture some of the value of this technology via a for-profit company"

Why?

Perhaps they don't want to end up having to beg [0, 1] for donations so that they can maintain a high quality application while also being able to afford trivial niceties like "food" and "housing."

[0] https://www.propublica.org/article/the-worlds-email-encrypti...

[1] http://arstechnica.com/information-technology/2014/04/tech-g...

Because there's a ton of work to do and getting it done in a reasonable timespan is going to require some talented people to work on it full-time.

he just committed 7 hours ago. https://github.com/mozilla/rr/commit/ef7691ac8a25ec9767ba744...

Yes, per the blog post he plans to spend more time working on rr than he thinks it's reasonable for Mozilla to have to support:

And now there's rr. People think it's amazing, and it is, but we're only scratching the surface of what could be done. I'm bursting with ideas, and realizing them will require a lot of resources, resources that Mozilla cannot and should not provide --- we probably need to capture some of the value of this technology via a for-profit company. We can change the way people debug software, and in its own way that may be as important as my Web platform work, and it's work I desperately want to do.

Reply on news.ycombinator.com