Add-Ons Outage Post-Mortem Result
hacks.mozilla.org
hacks.mozilla.org
It could be a feature that can only be enabled from the browser UI, with appropriate warnings, but the choice must be offered for those who need it.
Until that happens, those with private extensions must resort to compiling their own browser, use unbranded or developer builds, or patch Firefox at runtime to disable signature checks.
This is how you allow unsigned extensions in Firefox on Arch Linux, the same files can be edited on Windows and macOS, restart the browser after changes:
sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF
pref("general.config.obscure_value", 0);
pref("general.config.filename", "config.js");
pref("general.config.sandbox_enabled", false);
EOF
sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF
// keep this comment
try {
Components.utils
.import('resource://gre/modules/addons/XPIDatabase.jsm', {})
.XPIDatabase['SIGNED_TYPES'].clear();
} catch (ex) {
Components.utils.reportError(ex.message);
}
EOFIs that such a problematic thing?
> Most fundamentally, the full Firefox team does not have a common understanding of the role, function, and operation of cryptographic signatures for Firefox add-ons. For instance, although there are several good reasons for signing add-ons (monitoring add-ons not hosted on AMO, blocklisting malicious add-ons, providing cryptographic assurance by chaining add-ons to the Mozilla root), there is no shared consensus on the fundamental rationale for doing so.
https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technic...
That's not the issue. The problem is: where do you record the fact that the user has seen and acknowledged these warnings? Wherever it is, malware can go there and write "yes, the user has already seen the warnings, no need to ask them, you can run the extension".
Even with signature checks enabled malware can trick users by pointing Firefox shortcuts to a patched browser, changing shortcuts doesn't require root access.
That gives you a trusted place to bootstrap from. The addon database is stored in the user's profile, so _can_ be edited by such malware, but that's where the the trusted code could try to verify integrity in some way. Which is how we got where we are now.
Disclaimer: I work on Gecko, not extensions and the "things injecting into the browser" situation, but have overheard a bunch of discussions about it.
It can however, attach to Firefox as a debugger and modify it's code as needed (ptrace/writeprocessmemory).
Malware can do a lot of things, but I never heard of common software being designed under the assumption that malware has taken over the PC. The software I use is designed under the assumption that the user is intelligent enough to keep the computer malware free. Only the OS or explicit security software is designed to keep me from malware, but not my text editor, office software or browser.
It is a terrible excuse for ignoring user intent.
What's next? Disallowing to change the startpage from google.com because malware can change it? Disallowing downloads because users could download malware? Going that route would essentially take away the entire software in the end. Maybe at one point Firefox is only allowed to operate from within the cloud, where employees make sure it is 100% safe?
Also others have already said that one could add admin privilege to certain settings.
There are indications [1] that not even Mozilla employees have a good understanding of the threat model and the current solution, and that leaves one wondering how could alternative solutions be possibly explored if the topic isn't even properly understood.
I think the post-mortem has shown that the biggest problem of mozilla is fragmentation of decision-making, and the existence of probably >50 small teams that do stuff without communicating.
It is highly likely that the certificate problem isn't the only negative consequence, and that we'll see more evidence of mismanagement in the future.
I have the feeling that for some reason mozilla has established a culture where information does not flow efficiently from top to bottom and vica versa, and it even looks like the management doesn't really exist.
When a small team is formed around a task without central oversight, reporting back to someone, it will tend to justify it's existence, even if it means doing unnecessary work.
I heard the last CEO who wanted to streamline the mozilla hierarchy back to efficiency was Brendan Eich and many people got uncomfortable when he started to demand that people actually work productively.
I think a honest post-mortem would have come to a painful conclusion: That the Mozilla of today is in no way able to compete anymore, and many employees have stopped doing real work. The company only lives on because it lives off it's massive market-share of the past.
I am convinced that, within 2 years, mozilla will be confronted with massive lay-offs, threatening the gecko engine. This incidence shows me that they haven't done anything to address their structural problems, probably because most people in the company are content with the place they have, and comfortable living off the massive google revenue.
The same is true for link handlers, file extension associations, context menu entries, browser plugins. None of these should be able to be changed by applications, only by the user themselves.
We're entering an age where applications running in a user session are no longer user agents but users of your computer unto themselves that need have their own permissions boundary. Applications running as the user != the user anymore.
When there's a will, there is a way...
The answer is why this matters: we’re not in the Windows 95 era any more and a large fraction of users are running without admin privileges, often on a system which does code signing checks. Being able to get a trusted binary to run an add-on is common because the easier direct techniques aren’t as they were a couple decades ago.
If you insist that exploit can't escalate its privileges, then set this addon running privilege from Administrator. That would be a good thing actually.
Exactly, and it’s important in cases where users are being convinced to install dodgy extensions because each additional step increases the chances of someone realizing that it’s a scam.
Thus, I don't really understand your statement:
> those with private extensions must resort to compiling their own browser
I certainly would never recommend such a thing, myself. If I were paranoid about 3rd party access to my extension to the point I felt I needed to compile by own browser, then one thing I'd certainly do is also disable addons and build mine right into the core product...
I'd certainly trust Mozilla far more than I can trust every addon out there that might coexist with my private addon, which is where most of the attack surface would come from (if such were my threat model).
I do trust Mozilla, though I'd prefer if they'd also trust me to be capable of making an informed decision and give me a choice while using their product.
Permanently installing unpacked or unsigned extensions is a feature that all major browsers offer for their users, except Firefox.
That's exactly the point, you shouldn't need to compile your own browser to use a private extension!
I don't mean to be too harsh, but that part of the post-mortem reads like, "Oh, we don't need to renew our intermediate signing certificate because there's a known security vulnerability in Firefox that allows us to get around it."
"it turns out to be more complicated" = It's even worse than you probably imagined. We knew the certificate was going to expire and willingly let it happen because we don't know how our own product works. Also look at this new update mechanism and the other problems we caused! Forget about the fact that none of this makes any sense and please don't fire me for gross incompetence.
This was always a no-win situation for the people responsible. Allowing a certificate to expire has no reasonable justifications possible. It was always just going to be a question of if anyone would actually be held accountable for negligence or if they would excuse and blame-shift their way out of it.
I don't mean to be so negative about it. The silver lining could have been setting an example of doing a real investigation, finding that people were negligent, and holding those negligent people accountable, but I don't see that happening here because the people doing the investigating are the ones responsible.
The "it turns out to be more complicated" quote reminds me of Russ Roberts from Econ Talk. He has a number of interesting mannerisms that clearly signals his personal opinion on things. He doesn't come out and say "That's a terrible idea and I disagree!". Instead he whines "Well I'm not so sure." And when some study or evidence goes against his preconceived ideas or biases he says "Well, it's complicated".
First bullet under Root Causes:
"The server-side teams whose systems (e.g., Autograph) generate signatures knew the certificate was expiring, but did not see a problem because they had reason to believe that signing certificate dates were not checked by the client."
So, if I understand correctly, the assumed process was: Root signs Intermediate --> Intermediate signs Add-on (whether Intermediate is valid or not) --> FF only validates the End-entity is within date range.
Is this the correct procedure? You're telling me that the intended behavior is that FF allows add-ons signed by an expired cert? If that is the case, then what is the point of the signing system in the first place?
Additionally, third paragraph under Recommendations:
"Second, if we are committed to the current PKI and add-on signing approach, [...]"
'_If_ we are committed' is non-absolute. Is the FF team actually set on this approach?
1: https://wiki.mozilla.org/Add-ons/Expired-Certificate-Technic...
Edit: clarifications
No, this was disabled back in 2016. The technical report briefly had this worded wrong, sorry if you read it before it was corrected.
"what is the point of the signing system in the first place?"
Like the technical report describes, things like "monitoring add-ons not hosted on AMO, blocklisting malicious add-ons, providing cryptographic assurance by chaining add-ons to the Mozilla root."
(Disclosure: I work for Mozilla)
The solution for systems like Java and Authenticode is for Certificate Authorities to also offer a 'Trusted Timestamping' service, which certifies that the software existed at a time when the code-signing certificate was valid.
In Java's case, as the CA providing the timestamp is already trusted issue code-signing certificates with arbitrary dates, this doesn't add any new trusted parties.
The leaf certificate is specific to a particular instance of an addon. Firefox has a separate, more flexible capability called the blocklist that can disable individual addons. Since the blocklist exists, enforcing expiration dates on the leaf certificates wouldn't help in any way, but it would require re-signing and re-distributing all addons periodically. So, yes it was a deliberate decision to ignore expiration dates on leaf certificates.
> If that is the case, then what is the point of the signing system in the first place?
I'm not sure if this was meant literally or not but it is outlined here: https://blog.mozilla.org/addons/2015/04/15/the-case-for-exte...
> Now that we understand the importance of add-on support for mobile (ratings plummeted during the incident), we might want to reconsider that strategy/priority for Fenix.
Well that's promising at least. The listed Next Steps is still full of uncertainty and there isn't a firm decision (presumably the people involved in this post-mortem are not the relevant decision makers), but at least somebody is bringing it up.
Given the Mozilla track history, though, I'm not optimistic.
If I sign a binary with Authenticode on Windows and I use a timestamp server, Windows still considers the binary signature valid, even if the certificate expired because it can verify that at the time of signing the certificate was valid.
If you don't use a timestamp server when signing the binary, after certificate expiration the binary is considered invalid.
* All Shield Studies must be designed to answer a specific question - Shield is a tool for evaluating ideas and features in the product. If you are not trying to evaluate a proposed feature or idea there are other means of shipping your code. Shield studies will always respect user privacy in accordance with our data collection policies
* All Shield studies will adhere to the scientific method for answering complex questions - Generate a hypothesis, test, collect and analyze data, validate or refute hypothesis, refine, repeat..
* All Shield studies require a PHD (Product Hypothesis Doc) - A PHD or Product Hypothesis Doc outlines the overarching research question the study is trying to answer. It requires the study author to think critically about the problem and the outcomes long before the study ships.
These guidelines (https://wiki.mozilla.org/Firefox/Shield/Shield_Studies#Guidi...) were created after the beach of trust represented by the Mr Robot cross-promotion as a means to restore confidence. Yet now they have been ignored, and again "Studies" has been used as a convenient backdoor.
One day, some weird data gets into the database somehow, the site's completely unable to handle it, and it crashes and goes totally down. You're faced with two options:
1. Rewrite a major section of the site code to be able to handle the bad data. This is expected to take at least two days to complete.
2. Break the policy, fix the data in the database, and have the site back up in 5 minutes.
Do you truly believe that Option #2 needs to be treated as completely out of the question?
Second, it would only have taken a worst case of 24 hours vs the 6 hours of using Studies to follow that policy. I think the integrity of a company that trades on that very thing is worth 18 hours.
Yes, and adblockers, umatrix/noscript, Tor, and other add-ons were universally broken.
That has an impact on privacy too.
It's literally there:
> Because we had a lot of users who were affected, getting them fixed was a very high priority, which made Studies the best technical choice.
Which makes perfect sense to me. When you have a worldwide critical degradation of service, you fix that as soon as humanly possible, ideological pearl clutching be damned.
I'd have very much preferred them wait just 18 more hours and not have violated their privacy policies.
What exactly is Firefox's selling point if their principals can be traded for speed and convenience? That's just another tech company.
If everyone cared as much as you do about this issue, then they'd have likely waited. The fact that they were jumping through hoops tells me that the users don't care, they want their crap fixed _right now_.
Surely what that tells you is that Mozilla thinks that users don't care, and want their crap fixed; in which case the policy might as well say "These policies will be applied unless their abrogation is necessary to fix your crap", or some appropriately dignified re-wording.
You might think violating them was justified, but that they were in fact violated is inarguable.
You just claimed they had violated their privacy policy, not the shield studies guidelines. Your initial post doesn't say anything about their privacy policy. And as I said, your privacy wasn't affected at all if you didn't enable the studies yourself, so your point about rather waiting doesn't make sense either.
> You might think violating them was justified, but that they were in fact violated is inarguable.
You could argue that testing the fix meets the first two conditions you listed. I don't know if they actually provided that "Product Hypothesis Doc", but at that point you're really just looking for something to complain about.
Imagine your point in other contexts:
"If your company is about to go bankrupt unless it gets some serious revenue, then sure, you start selling the user data you promised not to, ideological pearl-clutching be damned."
"If a priest receives the confession of the killer in an unsolved mystery, he should report it to the police, religious pearl-clutching be damned."
And "critical degradation of service" is overstated; users always had the option to use another browser (like Mozilla seems to want these days). When they shut off everyone's unsigned, whitelisted addons in 2016, that wasn't regarded as a critical degradation.
Confidentiality in such areas (also med/law) is usually restricted to past actions. If you confess you're going to kill someone, that does indeed have to be reported to the police. I doubt your local laws are different here.
Thus to construct a better analogy we need ongoing harm that was initiated in the past but can only be shortened by taking action. Maybe a confession like "Just had an argument with my kid and imprisoned it in the basement without food and water. Will release it after two days". Now should this have to be reported to CPS? I'm not sure, but think I would favor yes.
I'm stuck running an old version of Firefox because I have too much data in Tab Groups, and I haven't be able to migrate it yet.
I'll have to find the time someday, since Google Sheets no longer works.
Yikes! You're running a years-old version of Firefox without security updates? And you're browsing public websites?
I think there are some supported add-ons for tab groups that work with current Firefox. Please, please don't run an insecure browser on the public internet.
There are good reasons for the move to Web Extensions, the deprecation of XUL, more careful checking/validation of extensions, and other such changes.
This seems like a bad faith characterisation. The discussion was about ars's activities, and I think it is reasonable to assume silversconfused was guessing at ars's opinions on these changes, not at "everyone's" opinions.
Firefox has been ignoring that preference since 2016, when it broke my workflow in the middle of the day and I wrote this Hitler parody:
the Firefox QA plan didn’t incorporate
testing for certificate expiration (or
generalized testing of how the browser
will behave at future dates) and
therefore the problem wasn’t detected
Does anyone know the state of the art in integration testing software for future dates?
This doesn't work with everything (for example doesn't work with Go because Go doesn't use libc for such things) but if it does then it's a very simple to use:
% LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' date
Fri Dec 31 23:59:59 PST 1999
% LD_PRELOAD=src/libfaketime.so.1 FAKETIME='1999-12-31 23:59:59' python -c 'import time;print(time.ctime())'
Fri Dec 31 23:59:59 1999
% LD_PRELOAD=src/libfaketime.so.1 FAKETIME='@1999-12-31 23:59:59' python -c 'import time;print(time.ctime());time.sleep(1);print(time.ctime())'
Fri Dec 31 23:59:59 1999
Sat Jan 1 00:00:00 2000