Hacker News Reader

Comment by Grollicus | Hacker News Reader

I'm interested in the general writeup what went wrong that they missed this certificate expiring. That's a structural problem.

Also why it took 6 hrs to assign P1 to the bug

View on HN

They closed the trees (stopped merging other code changes to prioritize this) for the bug <22 minutes after it was opened.

I would assume the delay in assigning P1 is really just a result of assigning P1 not being as high priority as fixing the damn problem.

This is one of the things about this whole episode that I find baffling. Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager. This is the right person to run interference and handle comms and deal with things outside of the critical path, like bugzilla updates.

The priority field just plays no practical role here since the bug was immediately escalated.

If people can see it externally it definitely has an effect, "oh moz aren't taking this browser breaking bug seriously". Which is probably why the parent said it's a management issue rather than a directly technical one, per se.

Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager

So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?

At no stage did the parent post state there are no good engineering managers at Mozilla. They just said that adjusting bug priorities is the work of a good engineering manager. There's a world of difference.

If you want to complain about knee-jerk overreactions, I think you might want to look in the mirror first.

I have empathy for people in the middle of hair-on-fire incidents.

If I understand the bug report comments correctly, they didn't close the trees to other code changes to prioritize fixing this, they did it because the cert expiry broke some important tests at the same time as it broke every end user's browser.

> Also why it took 6 hrs to assign P1 to the bug

Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire.

>This was treated as a five-alarm fire.

I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!

> One-alarm, two-alarm, three-alarm fires, etc., are categories of fires indicating the level of response by local authorities. The term multiple-alarm is a quick way of indicating that a fire is severe and is difficult to contain.

https://en.wikipedia.org/wiki/Multiple-alarm_fire

The 5th level or a five-alarm fire, is the top level where you're basically talking all hands on deck.

In the UK they actually sound/flash alarms at locations and in the fire-station, do they not do that in USA? They do in the movies.

Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.

I agree with you, it was more important to do the work than to signal.

However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level.

I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?

You have the book because at one time you didn't have the book.

"The book" is something that needs to change and improve like anything else. Sometimes, that means you'll still need to wing it and add that thing to the book later.

First, because priority is for things like major feature work, so that engineers can find the bugs that are useful to work on. In this case, everyone in the team responsible was already spending 100% of their time addressing the issue.

Second, because we care about solving problems, not being bureaucrats.

Really? Because being the bottleneck (i.e. single point of failure) responsible for approving all addons is exactly what bureaucrats would want to do ;-)

The non-bureaucratic thing to do, as has been pointed out many times of course, would be to give users the power to override the cert signing check as an advanced option.

It is, in fact, an "advanced option", in about:config.

But that option only works in nightly builds, not Firefox release builds. If an option doesn't honour what it claims to, imho it might as well not be there.

The normal practice would be to have someone operating as an incident communication manager who would be taking care of status/things like this.

Saying "we were too busy fixing to communicate" is actually a really bad sign, because it's not just about what you are communicating to the outside world, but also, for example, about making sure people that need to be brought in are getting consistent information.

This is an incredibly uncharitable misreading of what I said. We do have incident communication managers who were working around the clock, of course, communicating through the official channels.

The question is about why someone didn't set the priority setting in the internal Bugzilla sooner. There are a couple of reasons for this. First, that's not really what priority is for: priority is so that engineers on major projects like WebRender know what is most important to work on. It's not an effective tool for emergency responses. P1 doesn't summon on-call people. Second, everyone who was able to get it fixed was already on it. Bugzilla priorities don't make people who wouldn't otherwise be aware aware.

Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there. I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so.

"This is an incredibly uncharitable misreading of what I said"

I'm sorry if i offended you. Really. I don't really think it's that uncharitable, but hey, offending you wasn't my intent, and if you were, that is what matters.

For what it is worth: your response to the parent took them to task for even attempting to state something: "Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire."

This is a pretty rude response (which should never be happening regardless of what they wrote), and if you intended to convey that you had incident managers handling it and they didn't get around to it yet, you did not.

"Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. "

I'm really unsure why you decided to add a completely irrelevant and unnecessary attack like this.

Honestly, it just makes me think less of you and makes me sad. You are usually a very sane and even keeled person. I'm sure you were not having a good time, but i still don't think this was okay.

" One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there."

Which is incredibly ironic, since it is not in fact as private as you seem to believe . Your own assumption here is completely and totally incorrect.

Maybe before you decide to add completely irrelevant and unnecessary attacks, at least verify they are correct?

" I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so."

You seem to have taken my comment incredibly personally, and your response seems very far out of proportion.

If you want to have a discussion, you're gonna have to tone this down a few notches.

FWIW: Right now most of your comments in this story read incredibly defensive (and i'm not just talking about this one)

I would stop and give it a rest. It is not projecting a good image.

Of course, that's just my perspective.

Really hope tomorrow is a better day for you.

I'm also interested in the postmortem to explain the processes that failed to allow the certificate to expire, but let's not overdramatize the situation by nitpicking about filling in form fields on bugzilla. The fact that the tree was closed is equivalent to DEFCON-1, which is all the priority anyone needs to understand the severity of this bug.

> which is all the priority anyone needs to understand the severity of this bug.

Random user: What the fuck is a tree and why is the priority of this not higher yet?

> Random user: What the fuck is a tree and why is the priority of this not higher yet?

Not to be too glib, but any random user who is technically literate enough to know where to seek out Firefox's issue tracker and how to find the issue in question, and who has such a thorough understanding of issue trackers that they understand that such a thing as a priority field exists, is also going to be savvy enough to read the very first comment, and will be well aware of what it means, and will, one hopes, be rational enough to understand that the flurry of activity indicated by the issue in question is more important than a passing field in the bugzilla database.

If anyone expects Mozilla to take power users seriously, then we need to focus our criticism on the things that aren't just imagined trivialites. It makes me frustrated that the people who irrationally fly off the handle at the slightest perceived provocation are also the ones who implicitly encourage Mozilla to write off power users as more trouble than we're worth (and after ten years of watching these incessant whining non-comments on HN, I don't blame them anymore).

I'm also interested in why existing adds-ons are failing to run due to this problem. (There was a similar question in another thread about the issue here at HN.)

I understand why an add-on update or new installation would be prevented from succeeding by a certificate expiration. But why would a certificate expiration prevent an already-installed from running? Any already-installed add-ons were previously validated at installation time and should (IMO) run as-is. It seems unnecessary to continuously check the status of an add-on's certificate if it has not been changed. Am I missing something?

I believe it's set up this way specifically to allow revocation for addons that are initially approved but later found to be malicious.

If it is, without requesting user authorisation, then that's an illegal act under the UK Computer Misuse Act (and the USA's CFAA I think too) - modification of a computer without authorisation.

except you agreed and authorized when you installed the software. Take your position to the logical extreme - software can't make any changes without explicit, interactive approval; and you thought UAC was bad.

I look forward to joining your class-action lawsuit.

When the changes are unexpected, yes, further explicit authorisation is required. Just because you installed a photo-album app doesn't let the distributor delete all your photos, say.

Besides that, this sort of "but we hid something in the t&c-s so now we can shit on you" is the sort of thing I expect from over commercialised companies, not from what was once a paragon of the FOSS community.

FWIW class-actions don't exist in UK.

How do you know if it has not been changed?

Signing just verifies the .xpi file. If the addon has managed to get enough permissions to modify it's .xpi file it can bypass the signing requirement in various ways.

Revocation is, I think, the actual reason.

When a certificate is no longer valid, the authority it represents expires too. Grandfathering trust in various places would make cert management even more difficult to get right, because there'd be no hard deadline when a certificate is no longer in force.

But that represents how people consider trust when choosing addons. It's trusting the code and company at the time of install, not at an arbitrary later time. Sure, if the cert expires and there's an update then the user wants to know.

The main trust check is at installation time, but it's possible for problems to be discovered later, and Mozilla needs to be able to do something about it. Certificate non-renewal is the only robust avenue of revocation.

There's already an extension banlist.

See profile/blocklist-addons.json

Not sure how not renewing a certificate and letting everything get disabled is useful. It's only useful if cert key leaks.

> Mozilla needs to be able to do something about it.

No, they do not need to. They decided that they want to. Remember that there was a time before certificate signing.

And now the decision to be able protect those who install crapware is also harming those who never had those issues.

They should absolutely have asked extra permission to implement a system where they could choose to alter my browser install, in an unexpected way, at their behest without seeking further authorisation, not even a modal???

It's exceedingly poor ethics.

You cannot rely on “check at install time.” An extension could be installed by a crapware installer behind FF’s back. You can’t go and remember the trust state at install time either, because that memory would need to be kept locally and could be modified by a crapware installer. So the only solution that prevents circumventing the check is to check the signature when the extension is loaded.

It's already a mess. I disabled the signing/recompiled Firefox, and all my extensions were still force disabled with no UI to enable them. So there's some memory/extension state there already.

I had to go thorugh profile/extensions.json and set appDisabled to false to make my extensions enableable again.

Looking at the changeset [1], I'm curious why the explicit check for expiry (line 644/646) didn't work. Unfortunately the mentioned bug is rather light on details; presumably they were collaborating on IRC or something instead.

[1] GitHub mirror to not stress their infra: https://github.com/mozilla/gecko-dev/commit/1d1260c7615f1d9a...

Reply on news.ycombinator.com