Yes, it's unfortunate, I'd expect them to meet it head on, push a tested fix in a timely matter, admit a mistake was made, explain publicly how/why and apply learning moving forward. Beyond that, what's your expectation?
There was a chain of bad decisions that led them here though: 1) thinking it's ok to disable software after its installed (using cert expiration -- I'm ok if the cert was revoked but that's a totally different discussion), 2) Taking more control of people's local software than many people are comfortable with, especially considering that their main market is tech savvy people that tend to be more sensitive to this than most 3) Making some of these things opt-out rather than opt-in, giving the perception that they may value data collection and control more than their users privacy.
Here, the mechanism that kicked in was the protection against add-ons that could have been signed with stolen credentials, which would make them clearly malicious.
Of course, it turns out that the problem was an expired cert, so a bug/human error. But generally speaking, I think that 1 is good.
I hate this attitude from security people so much. If for the sake of fighting malicious code you are crippling the software usability or my user experience, you are the malicious code.
If I can't do anything with my hardened computer, I don't care if is eaten alive by malware, it is useless anyways.
At work, as the guy who have to fight on behalf of the sysadmins and the users dozens of clueless security advisors who are hardening everything according to security best-practices written by similarily clueless experts, I'm seriously astonished by the common backward thinking. If you are blocking access to all users pdf files, for an instance, you are the malware, you are causing disturbance to the business operation and annoying everyone.
Go sit in separate corners, both of you. Think really, really hard about how both of your jobs are critical to the long-term success of the business. Don't come back until you've meaningfully internalized that.
Similarily, when "security best-practices" are leading to hundreds of my users losing SSO access to their BI system, or hundreds of printers rendered unusable because of security update that requires administrative permissions for reinstalling the same drivers that were perfectly working so far, I don't care for your security benefits. They suppose to defend against the thing that you are causing. You are already damaging the organization with thousands of working hours lost, and everyone is frustrated, as a bonus.
No, I am apparently not. Microsoft, Apple, and others insist on making it difficult. At least I found out today that I can install unsigned Firefox extensions once I switch to a special "unbranded" build. I'm glad Mozilla, at least, still offers that.
Here's the thing: I disable a lot of the security stuff you're not supposed to disable, when I can. I use a Jailbroken iPhone. My Mac has SIP and Gatekeeper turned off. Windows Defender is turned off on my gaming PC, and I lower Microsoft's driver signing requirements to the greatest extent allowed. I also ran an unpatched day-1 build of Windows 10 for around four years, with the autoupdate system forcibly neutered. (I now run LTSB, instead.)
I have never been bitten by a virus, ever†. I don't know if that's because of all the security measures I'm not able to turn off or because I've been lucky or something else. I suspect it's because I don't run dodgy software. Or maybe my life is a lie and all my devices have been infected for the past decade, and I never noticed.
In the meantime, I'm not seeing the upside to software forcing hardened security.
---
P.S. While I share their frustrations, I don't endorse the GP's attitude. I know that a lot of people really are doing difficult work with the best of intentions.
† Except for a handful of times when I was testing suspicious software in a disposable VM. That doesn't count for obvious reasons.
Not running dodgy software is indeed a very effective way to not get viruses, but that doesn't mean you shouldn't take more security precautions if they are available. What if, for example, malware exploits a zero-day in your browser and successfully installs itself without any interaction from the user? Windows Defender and related antivirus could detect malicious activity from such malware and remove it on windows and Gatekeeper/SIP/driver signing and related systems could severely limit its impact.
Mozilla probably introduced extension signing to prevent less technically inclined users from having adware installed into their browser. X.509 introduces certificate expiration, and X.509 is the most widely used mechanism for signing things. The only reason you have this problem is because the folks at Mozilla forgot to renew that intermediate certificate. While I agree that it should be possible for users to disable extension signing, as users should be able to do whatever they want on their own system, you shouldn't blame them for forgetting to renew a certificate associated with an additional security measure built to protect users from malware.
Well, we agree on everything, in that case. :) I have no problem with sensible defaults that can be adjusted, nor do I take great issue with Mozilla's specific mistake in letting the certs expire.
Edit: Also, just noting that I only lessen security measures when I have a reason, not because I'm rebelling against security practices or something. I use a Jailbroken iPhone because I run Jailbroken software, and I disable driver signing because I use unsigned drivers. It's not that I don't see the risks, so much as I'm very unconvinced the risks are worth the downsides, for me.
Malware that does obvious bad things like cripple the machine or encrypt the drive is minority.
But it's actually not funny, it's dangerous. This behavior endangers others.
Likely your computers have been a zombie in a botnet for many years - I'm always wondering how these large botnets can exist at such scale.
This is why one day we all will not be allowed to use computers without a "driving license".
All that is diffrent is the promise of non-maliciousness. Which often does not hold up. Cause money is corrosive to those little centralized empires of "all-can-fail-but-me".
Security is diversity, as in having a non-centrally controllable ecosystem, that is not a mono-culture. Your updates are the danger, your urge for control is the forrest fire.
Linux is not secure because its updated often. As package maintainer take-overs have show- that is even a vector. Its secure, because its fragmented into a thousand small populations, which offer no real financially interesting attack vector for a large scale take over.
(The situation is admittedly getting better with Flatpak.)
Today, more than half of my open tabs disappeared in an instant, and were not even an option to re-open until either I waited around ("up to six hours...") or manually installed the workaround. All of my in-progress work in any of those tabs? Gone.
That absolutely qualifies as crippled usability. The mere fact of such a thing being possible is a usability defect. On what basis do I trust that my work is not going to disappear on me like that again?
https://bugs.chromium.org/p/chromium/issues/detail?id=952287
Chrome has bugs too.
Firefox will continue to have bugs. All software will continue to have bugs. I'm so sorry that you lost some tabs in your browser but shit really does happen and acting like this is some violation due to overzealous security controls is inane.
If you think this clownshoery hasn't cost Firefox any trust, then you're being as naïve as you accuse me of being "absurdly overdramatic" and "inane".
Bugs are a thing, totally conceded. Sloppy certificate management is, too, but it's an entirely other class of thing. Deliberately conflating them is at least as disingenuous a debating tactic as pointing at Chrome, which is utterly irrelevant to this incident. That's straight-up "whataboutism".
Full stop, this was foreseeable. This was preventable.
EDIT: Phrasing.
EDIT 2: I won't respond further to the same kind of tone.
That said, the part I was referring to is:
> The mere fact of such a thing being possible is a usability defect. On what basis do I trust that my work is not going to disappear on me like that again?
The possibility of a bug happening is hardly a usability defect in my mind. Or if you want to call it one, it seems like a perfectly reasonable one - this was a defense born out of necessity when malicious extensions were more of a problem.
And I think that the "On what basis" question definitely implies a total lack of trust, but sure, maybe not. The basis is that this is a single instance of a failure over the course of the features' lifetime, for a feature that has existed for absolutely ages.
I pointed to Chrome as an example of similar issues cropping up across codebases to show that these sorts of bugs do happen. I don't consider that whataboutism.
All bugs are foreseeable and preventable. Systems are complex. I think you're putting the issue in a very unfair light, even though it's very reasonable to be upset about time and effort that is lost because of the issue.
I understand your perspective, and appreciate your recognition of my own. That said, if you think I'm putting the situation in an unfair light, I think you're downplaying it at least as much.
In my eyes, this is no mere "bug"; it's an abject process failure. As a reply to another of my comments in this discussion suggests, this is more on the level of, "Oops, we forgot to renew our domain name...", than it is, "Gosh, we didn't validate the pointer returned by the frobnitz function, when the whoozle isn't initialized yet..."
Dealing with expiring certificates before they expire is covered in like the second week of Certificate Management 101, as it were. If it's necessary to stick an intermediate cert in there, then it's doubly so to keep it current.
> The basis is that this is a single instance of a failure over the course of the features' lifetime, for a feature that has existed for absolutely ages.
The plural of "anecdote" isn't "data", but an existence proof is an existence proof. That the problem has gone from zero occurrences to one, no matter over what period, literally makes it infinitely more likely to recur, if you want to be that reductive...
Is that so, though? Firefox is still being used by millions of users, and I doubt those are only the tech savvy internet users.
(Then again, this mostly applies to Firefox users using add-ons, which probably has a higher share of technical users.)
it's also entirely predictable that a non-negligable fraction of users -after enabling studies and verifying everything works again- will ... simply go on with their lives and forget about disabling studies...
I also don't understand why the certificate graph is not exposed through a user interface, so that the user can add and remove certificates, or enable and disable certificates at their own discretion. This should have been obvious when the certified add-ons were introduced. Then all they would have to do is host the certificate file on their own domain and everyone could follow the simple steps in the GUI to replace the expired certificate...
CAs can delete certificates from their revocation lists after expiration, which means that you can't tell the difference between a certificate that was never revoked but merely expired and a revoked-and-then-expired certificate.
To me adding a new plugin signing cert through a side loaded plugin is pretty much the definition of exploited.
All this tells me is that their plugin signing solution is utterly worthless.
The only way this should have been fixed was through official update channels.
> Why not just post a link to the fix that can be installed WITHOUT enabling Studies? This sounds like a clever plan to get more people to share their data via Studies…
I definitely don't agree with that guy, and I doubt that's a majority opinion, but asking people to use a workaround that benefits them (Mozilla) after they broke things for a lot of people is bad publicity for sure. For what it's worth I think Mozilla is doing the right thing here, just it's not going to make them look great.
I still prefer Firefox over all the other browsers, and will continue to use it, but the project has lost a lot of trust and goodwill over this.
The optics are indeed awful, and this was fully preventable. Firefox fucked up, full stop.
Based on the timing of initial tweets and blog posts on this fiasco, I'm pretty sure I was in the first 10%, if not first 1%, of people who experienced this. And I was in a plane at 36,000 feet trying to work on a cross country (U.S.) flight when suddenly about 130 tabs in 7 windows disappeared. Really, REALLY bad. Panic, frustration, confusion...
I was more than 50% sure that all was not lost forever, that it was some "glitch" (Extensions all showed the same bloody red status), but I was tweaked. I work in security (embedded systems, not computers/IT) so I have a very good understanding of certificates, TLS, PKI, etc. There are many ways things can get out of whack if the people in charge screw up.
Regardless, this is embarrassing, dare I say shameful (pretty much almost up there with "Ooooppsss... we just lost our domain - it expired and no one thought to renew it)
Come on, guys, get it together. Have a procedure, document it, practice it, stay in front of it.
EDIT: after installing the fixed XPI, I have to sadly report that all data has gone. All my carefully-managed containerized life was wiped clean. Heads should roll.
Complete shambles. And the worst thing is, I suspect it's all a plot to have more people opt-in to the shitty telemetry. Otherwise, why not push an update through the usual channels? Had it been a security-related fix, would have they used "studies"? I bet not.
Given that it has happened, I expect them to provision a new certificate and push a fixed version within an hour or two to all release channels.
What I would emphatically ‘not’ expect, is a hack that might take up to 6 hours to be applied.
Most Firefox users have that checkbox enabled by default, and so most Firefox users received the fix within 0-6 hours of the blog post's publication.
HN readers often take special care to prevent Mozilla from updating Firefox, but that in no way represents the wider population of either all addons users or all Firefox users.
No, it's not. "Studies" is not a security-related mechanism and it didn't exist in the past, when fixes were rolled out very quickly anyway for security reasons. "Studies" should not be relied on to be a fix-delivery mechanism, because it just isn't.
This is not even about privacy, it's simple good engineering sense.
I use FF because I care about principles. Otherwise I might as well just let myself be exploited by Google, MS, Apple and friends.
https://news.ycombinator.com/item?id=19825830
(While of course you’re welcome to continue pursuing your issues with it here, your opinions will receive more views there.)
Presumably the testing burden for a preference update using Normandy is smaller, as (and I’m guessing wildly here) fewer things can be altered with Normandy and therefore testing can be simplified to exclude, for made-up example, “the code-signed binary can be executed on all platforms”.
You can fix this temporarily via setting "xpinstall.signatures.required" to false. Toggle it back to true once update is released and you install it.
Meanwhile I'm hijacking this comment that is to the upper parts of the tree to state this: the way the community treats Mozilla and Firefox is horribly, inexplicably, unacceptably unfair.
This is nothing compared to innumerable other fuckups in software history, and even recent ones like goto fail, heartbleed, or Chrome logging you into Sync w/o notice.
This is a mistake, an easily recoverable one, and is not intentional or malicious. Firefox is developed in the out and open, all the processes are public. And people, with an absurd entitlement and malice, go as far as to call things backdoors or malware. Meanwhile the alternative actually is a backdoor ridden malware.
Please don't be this ungrateful.
While I agree with most of your comment, you're downplaying the severity here, especially since, IIUC, this situation also affected the Tor browser, disabling NoScript. If regimes like China were on the ball, and succeeded in escalating the remote code execution vulnerability into into deanonimization, this debacle may end up having a death toll attached to it.