Postmortem on a beta regression | Hacker News Reader

Postmortem on a beta regression

mail.mozilla.org

40 pointsphilbo11 comments

Being annoyed at how Firefox has handled the introduction of add-on signing, this stuck out like a sore thumb:

""" the problem was discovered, and filed as bug 1240559. This was on January 18th, which is 8 days before the beta gets shipped to release.

A workaround for add-on developers was posted in https://bugzilla.mozilla.org/show_bug.cgi?id=1240559#c10, and the bug was marked as a WONTFIX on the 21st.

On the 22nd, an add-on developer reopened the bug, saying that the workaround wasn’t acceptable, as they wouldn’t have time to fix their add-on and get it approved before 44 went out the door.

A patch was written, and then the decision was made to land the fix directly onto release. The add-on author confirmed the fix, and the bug was closed out. This is what we ended up shipping. """

The add-on developer in question who got this reopened, Ben Bucksch, got it absolutely right:

""" This is a major goof for ext compat. You goofed, you'll have to fix it. You can't leave the dirt on our carpet. You can't just remove APIs without warning and then say "sorry".

This is a must fix. And even if FF4 were to be released tomorrow, you should still be glad to have found it before it hits end users.

> No, there isn't anything left to do.

NewTabURL.jsm was accidentally removed. There's a trivial fix: Re-add the file. """

Mozilla devs appear to be completely out of touch with what their core goal should be: a stable open-source browser with an awesome add-on community. Instead the Process is holy and cannot be altered. In the process they're alienating add-on devs, people who make their product more awesome for free. Come on, guys, who gives a shit about whether about:newtab is remotely hosted?

This is attributing an awful lot of malice to the FF team. They acknowledged they were wrong and got the fix out. Sometimes it takes a bit of pushing to understand the impact of a bug; we've all been there.

The entire post is about recommendations to avoid this sort of thing in the future and make the FF dev process friendlier for add-on developers. Reading your comment it sounds like their attitude towards add-on devs is "go pound sand", but it certainly doesn't seem like it to me based on this post.

I never ment to imply any malice. (A famous quote comes to mind.) I'm just saying that (as Ben pointed out):

* the bug breaks the browsing experience for millions of users

* the fix is well-understood and simple: re-add a single file that was accidentally omitted.

Closing that as a WONTFIX is.. bad.

The affected add-on, or at least the one that prompted the bug report to be filed, has 46,755 users, not millions:

https://addons.mozilla.org/en-US/firefox/addon/new-tab-overr...

The add-on Ben was talking about is a different one (I don't know which).

Thanks. I missed that because the other addon's name was never mentioned.

Reading the bug comment history, it seems they were saying "won't fix for 44 (too late)" not "won't ever fix, amen!"

But installations of 44 will be out there for who knows how long; affected add-ons would have to carry the workaround egg-corn for a long time before declaring no support for 44 and removing it.

Hindsight is 20/20, but when the wontfix decision was made it wasn't clear that there was significant impact beyond the one addon which reported it -- which had already implemented and shipped a simple workaround -- and the missing file had been missing for nearly 2 months on beta. This was just a few days before shipping already-final code to hundreds of millions of people, which isn't a time to be taking changes lightly. (And as a fun hindsight-in-hindsight twist, it now turns out the millions-of-users addons wasn't actually as badly broken as described.)

It all worked out ok in the end, and could have taken a much better path, but it's not as egregious an issue as you make it out to be. A mistake was made, and it got fixed as soon as we realized it was a mistake.

> Mozilla devs appear to be completely out of touch with what their core goal should be . . . Instead the Process is holy and cannot be altered

They made a mistake. They then fixed it and admitted exactly what the mistake was. Because they did so, they're called completely out of touch.

I often advocate for my employer to be more open in our postmortems, but responses like this make it impossible for my side to win the argument against folks who claim, quite reasonably, that admitting any kind of mistake will result in people using it against us later.

They made a mistake, which broke an API used by many third parties. The fix was very simple. Yet they close as WONTFIX until another dev makes repeated comments that point out this is not tolerable. Then they reopen and fix.

It's the closing and insisting "this is not a showstopper, work around the bug in your third party code", that indicates to me they are out of touch. What happened after that was fine.

Keep in mind the perspective of the people trying to ship Firefox:

* The bug does not break the vanilla (and therefore vast majority of userbase) firefox experience

* The bug is easily worked-around (as opposed to completely unfixable)

* The release candidates were already being built (which means you don't want to make changes, because you have to rebuild+resend to manual QA+resign)

* The bug was merged in as part of a security fix which they can't risk touching

They're not necessarily out of touch. It could easily be that people who didn't have all the details (e.g. release managers who didn't realize the fix was re-adding a single trivial file) were worried about the risks of upsetting a security release.

Sometimes it takes time for the severity of a regression to work its way up to a decision maker. And in the end they did exactly the right thing, with 5 separate people all putting in work on a weekend to make it right.

Reply on news.ycombinator.com