Firefox Addons Unable to Update, Undisclosed AMO Issues | Hacker News Reader

Firefox Addons Unable to Update, Undisclosed AMO Issues

discourse.mozilla.org

184 pointsgilrain107 comments

I have an add-on that I submitted an update for on June 21, 2021. It’s still “Awaiting Review”.

It’s an add-on that only I use. It’s not published broadly. I basically only needed Mozilla to sign it so I can install it.

Very frustrating. After waiting for a long while, I gave up and switched to the Developer Edition so I can use my own add-on.

Here you go https://github.com/mozilla/web-ext

That just calls the API.

https://extensionworkshop.com/documentation/develop/web-ext-...

> After waiting for a long while, I gave up and switched to the Developer Edition so I can use my own add-on.

I find it very frustrating that they now force users into Nightly / Developer Edition if they want to permanently install unsigned add-ons. What's the harm in simply locking that functionality with a config option?

Mozilla is like apple in that regard, users can't be trusted with their own machines and the well-intentioned mothership must at all times be in control since at any moment they could fall to social engineering and then they (apple/mozilla) would get blamed for whatever the malware did.

Installing developer edition is the blessed way to opt out of that.

You can install a locally built and signed extension in the release version of Safari, without disclosing the source code to Apple.

I was referring to apple's general behavior (how they lock down their phones) not their specific browser extension policy.

If apple, one of the most controlling companies on the planet, thinks it's okay for their users to install any extension they want, why does Mozilla feel justified to do the opposite?

What risk are they trying to mitigate, and was it worth pushing addon developers away from their browser too?

Their reasoning? https://blog.mozilla.org/addons/2019/10/31/firefox-to-discon...

No wonder people abandoned them. Mozilla is the digg of browsers.

> What's the harm in simply locking that functionality with a config option?

Nothing, there is nothing wrong with educating and informing users, then letting them use an extension privately. Users should not be forced to use unstable versions of Firefox to install an extension locally, nor should it be Mozilla's business to inspect the source code of that extension.

What's funny is that even in browsers such as Safari and Chrome you can permanently install a local extension after toggling an option, without being forced to disclose the source code to Apple or Google.

Firefox is the only desktop browser that prevents users from installing local extensions, and because Mozilla does not control the platform, malware can trivially bypass their restrictions.

At least Nightly is the superior version of Firefox, so it’s an upgrade.

Firefox Developer Edition is now based on the Beta release channel, since Aurora is no more. It's supposed to be more stable than Nightly.

That’s technically true, but in practice, Nightly is very stable. I’ve been using it for probably five years, and issues are seldom.

Malware can set that config option without consent.

Malware can also install firefox developer's edition, or a modified firefox without consent.

If malware has that level of access on your machine, chances are your browser is already fully compromised.

Configuration and add-ons reside in %AppData%, or an orherwise user-writable profile directory. Compromising the executable, which lives under %ProgramFiles%, or an otherwise protected directory, takes administrator rights.

Beyond this plausible inconvenience, however, Mozilla simply doesn't want regular users messing with unapproved add-ons. Just switch Firefox to Developer Edition for that. It's been very stable, in my experience.

Then why wont they allow users to install their unsigned addons in %ProgramFiles%? I don't think protecting against a compromised %AppData% was their only goal.

The Firefox profile directory also contains sensitive things like its file cache and trusted CA database, so you don't need to plant a malicious extension to achieve significant impact when you only have write access.

Adding to this issue I would like to give some of my opinion on webext.

To be honest Firefox add-ons process is so grotesque. For instance I can't load my extension without signing in Firefox stable version. And their tool especially web-ext has lot of issues like takes lot of time, gives pesky error if your system time is incorrect (my isp has blocked ntp servers and idk why and switching to vpn just to update is painful tbh). And developing addon is also hard for firefox compared to chrome as the dev tools frequently give message unrelated to extension etc.

Sometime I get so angry but I have been using firefox nearly for decades. Its so hard for me :(

Unrelated to your core point but your ISP blocks ntp servers? Wtf? Either some part of the story is missing or that ISP is insane, blocking a piece of core internet infrastructure for which there is, I would guess, no reason.

Not OP , but I have seen type of behaviour , ISPs act like corporate firewalls and block everything by default just allow http on 80 and 443 in their consumer plan and try and sell enterprise/higher plans for literally anything else .

Not just that once a ISP was using a single CG-NATed IP for a ton of users, that the IP was constantly getting rate limited with captcha everywhere including Google Search and Cloudflare. They suggested I buy a plan with dedicated static IP instead of rotating a few more in their routers.

NTP is commonly used for DDoS amplitication.

The right way to do this is to police, not block, and by length.

Some ISPs (e.g. AT&T) block outbound traffic from subscribers with source port 123, to mitigate NTP reflection attacks.

Shouldn't necessarily break your NTP client, right? The client's destination port needs to be 123 but the source port can be anything.

But many NTP clients use port 123 as both the destination and source port.

For a while I had a netfilter POSTROUTING rule that would match outbound packets with source port 123 and force translation of the source port to the 60000-65000 range, which had all my NTP clients working again.

Can you please share the iptables rule?

iptables -t nat -I POSTROUTING -p udp -m udp --sport 123 -j MASQUERADE --to-ports 60000-61000

I don't know why I didn't specify the outbound interface but I probably should have (-o $wan_interface), anyway it did the job for a long time.

The breakthrough I needed to get it to work was discovering that --to-ports option, which I'd never used or seen in the wild before facing this problem.

Do you mean it blocks incoming or outgoing ntp requests?

My old isp blocked port 25 inbound for security but I didn't consider that extreme

I want to give mine too, because I really appreciate how well it works for me. I built a Chrome extension and it runs fine on Firefox with 0 changes since Firefox supports the exact same browser APIs. When I was developing with Chrome, I had to manually upload my extension each time I updated it. web-ext reloads the browser for me automatically - you do have to take care to avoid duplicating the DOM since it doesn't reload the tab. Not sure why system time is so critical but it seems like a reasonable expectation.

I mean, the browser apis are close (and Mozilla still has much better documentation) but there are a LOT of edges cases where behavior diverges.

Frankly - I'm a little peeved that Optional permissions in Firefox are STILL broken - The prompt can only be triggered in response to a user action, and Firefox blows the fuck up if you put a promise anywhere in between the user click and the call to the api. Which is hugely ironic, since Mozilla is the one pushing to move all the webext APIs to be promise based (and provides a nice helpful library for Chrome/Edge/Safari support: https://github.com/mozilla/webextension-polyfill) which... doesn't work on their platform. Doubly ironic, since the result is that most FF extensions just ask for more permissions up front, which is exactly the opposite of what you'd want in the "secure/private" world Mozilla claims they're pushing towards.

I can't weigh in on the rest of your experience but I think the blame for the time synchronization issue lies squarely at your ISP's feet. Requesting that the time be properly synchronized to digitally sign something doesn't seem to be an absurd requirement to me. Blocking NTP on the other hand is quite insane IMO.

If I were in your situation and couldn't change ISP I'd probably buy a cheap GPS USB dongle just to have proper time sync on my network.

Stratux sells a u-blox 8 USB dongle that I've had decent luck with for this timekeeping purpose, although it can be tricky to get a clear signal indoors, and I had to bootstrap the module's first connection outside with usb-otg on a smartphone. But pretty acceptable results for under $20 when feeding it into ntpd. Not exactly plug and play to get it to work, though. But pretty reasonable for most of the diy crowd around here.

I need to tape over the green led due to photosensitivity, though. It's shockingly bright.

It took 2+ months for our extension update to be approved, it was a small incremental update. Getting it done in that time frame required us to email Caitlin and an HN'er who commented on another thread to try and get things done in an expedited manner.

For a time there was a little ticker showing your place in the queue. We graphed that for a while, then they pulled the feature entirely: https://twitter.com/preinheimer/status/1422577415780450311

The queue of course isn't a straight queue. Some people end up traversing the entire queue, very slowly. Other folks manage to jump out of the queue and get listed sooner.

We've seriously considered abandoning our FF extension. FF's market share is shrinking, and we see the difference in the usage stats for our extension specifically.

Some of our updates require coordination on our website and the extension. So holding onto FF means that some updates take months to roll out for everyone.

This would be a lot less of an issue if Mozilla let you install addons that weren't signed on the stable channel. Or at least had a way to add a custom signing CA.

Or, if they did force you to go through their CA, actually make sure they keep it updated.

https://news.ycombinator.com/item?id=19823701

Anything the user can do (like adding a signing CA), malware can do too.

The only safe policy flags a browser can provide/respect for extensions, are ones the user can’t affect from their own computer—e.g. GPOs / MDM profile attributes set by a domain administrator. (And both Firefox and Chrome do have management-domain-level extension policies!) Everything else is just one “trick the user into an elevation” away, and then they’re unknowingly part of a browser-embedded botnet.

A process that is preventing gorhill from testing and updating uBlock Origin is a failed process.

Malware can just replace or patch the Firefox binary if it wanted to, so enforcing add-on signing wouldn't protect against this.

Technically then at least the windows signature failed prompt would show up. Also the firefox binary is usually somewhere you need admin access to write (i.e. admin prompt).

I don't really agree with the method Mozilla is using, but at least the explanation makes sense.

> Also the firefox binary is usually somewhere you need admin access to write (i.e. admin prompt).

No reason custom certs couldn't also be stored somewhere that needs admin access.

Archetypal “malware,” the kind that’s illegal to distribute, doesn’t have to worry about any of this stuff. It can just patch the browser.

The problem is “legitimate” businesses that engage in scummy-but-not-illegal behavior. Stuff like the Ask toolbar being shipped with Java, and five years later my friends wind up with dozens of toolbars and they don’t even know where they came from. Those sorts of companies will not patch Firefox.exe, because that would require violating Mozilla’s registered trademark.

Yet Chrome allows the user to install their own extensions. Where is the malware exploiting this for Chrome?

Malware that has user-level access won't need to bother with addons. Adding user MITM certs strikes me as a far easier way to access browsing traffic.

I believe you can do this on the Developer Edition, install unsigned add-ons. The DE is basically the same as Firefox Beta but with these kinds of tweaks.

But as you said, developer edition is based on beta, not stable.

So first they take complete control of add-ons by deprecating an old (admittedly creaky) system and replacing it with a nerfed one that can't even hide the tab bar; and by requiring add-ons to be signed by them to be installed. Now they'll let what's left die of neglect. Really poor and predictable.

edit: I've found of late that my accuracy goes up when instead of trying to predict the future, I just imagine the most dystopian development of a situation possible. Right now I'm imagining add-on developers having to pay Mozilla for the review of their extensions, and their placement on the site.

Why not paying them to review? It takes time for them, why should they do it for free?

I was just thinking "a way to solve this problem might be if they started charging for reviewing add-ons and updates".

Maybe they could charge more from organizations with lots of money, and less from indie devs with less money

Or they can let people side-load addons like every other desktop browser.

An idea: To strike a balance between flexibility for individuals / small groups, and security for the masses,

Firefox could let people side load add-ons used by less than say 1 000 people.

However once an add-on gets more than 1 000 users, only the first 1 000 can side load it; the others need to wait until it's been approved (reviewed).

gorhill is unable to test and update uBlock Origin:

> This is unfortunate. Development of uBO for Firefox is completely stalled as a result -- the purpose of dev builds is to test code changes before publishing a stable release. It used to take less than five minutes to obtain a signed version of uBO. There are changes in 1.38.0 which the filter list authors are awaiting and this is also stalling proper filter list maintenance.

https://reddit.com/r/firefox/comments/pv15k2/_/he7qm5u/?cont...

I think this is their extension: https://addons.mozilla.org/en-US/firefox/addon/i2p-in-privat...

Updated two months ago. The forum post is from late August. They still haven't been able to update it. Evidently, app stores suck even when maintained by well meaning people.

The latest update in the thread, nine days ago, is pure corporateese: Hi @zephyr, unfortunately we don’t have a lot more we can share at the moment. However, I’ll talk to the team and see if there are any updates for next week.

It's a frustrating state of affairs that this extension review process even exists ...

Of course we'll be called entitled whiners for demanding prompt customer service from an open source foundation which does this completely for free.

But then why does such an organization with such an offering lock it down with a mandatory capricious review process like Apple? What profit is in it for them?

It's to prevent the bad publicity which results from malicious extensions affecting thousands of poor helpless users. How repugnantly negligent would mozilla be considered by news commentators, to allow so many of their users to be violated so badly by their extensions!

But I can install any debian package I want from anywhere? Or download and compile a source tarball from anywhere? Use pip to install from pypi which is a free-for-all ... or from any tarball or git url? Even windows users can install arbitrary things!

I do find it surprising that there is trend for shady companies to try to buy out popular Chrome extensions to slip adware and then malware into them, this has hardly been seen anywhere else (except mobile app stores maybe).

Anyway, I would love a completely different solution: something easy and obvious in settings to disable extension lock-down, with whatever scary warning is needed to let the commoner know they will probably hurt themselves badly, if they can choose what software to run on their computer. Let this option be disabled by GPO for corporate-controlled workstations. Let third-parties provide "trusted extension registries", similar to adblock lists, with hashes of versions of extensions that have been checked by someone who is savvy enough to install debian packages, and can tell when a popular extension has changed ownership. Yeah I know this is unlikely, and again, I'm a whining entitled idiot for demanding anything from a benevolent organization providing open-source software at no monetary cost.

I think we just need an "un-mozilla'd firefoxium" ...

> I do find it surprising that there is trend for shady companies to try to buy out popular Chrome extensions to slip adware and then malware into them, this has hardly been seen anywhere else (except mobile app stores maybe).

Oh, it did happen a lot with Firefox before the extensions were locked-in and a heavy review process was put into place.

But then, I agree that they should add the possibility of using alternative stores. There is no reason for Firefox to be this locked-in.

> I think we just need an "un-mozilla'd firefoxium" ...

Isn't that what Librewolf is?

Seems like it is - I wasn't aware of that project, thanks for the tip!

(I was aware of Pale Moon, which forked from a much older version of firefox ...)

'Hi @idk, we have a few issues on our end that are causing a delay for some add-ons to be signed or released right now. We’re still trying to work through the backlog (and we understand that some developers have been waiting quite awhile for a review) but it still may take some time to get to everyone.'

Mozilla's response. Interesting

My guess:

Some bug caused a whole lot of extensions which normally need no manual review to now need a manual review overloading the manual review team.

And even through the bug has been fixed for some reason or another all the "fallout" still needs to be processed manually.

Maybe?

That's what I'm thinking too. Possibly a security issue.

That... or they fired the reviewers.

That would be stupid.

Why? Firefox has relatively few users and has basically no viable path for growing it's users. At the same time they get a massive amount of money from Google that isn't based on the number of users. Cutting costs seems like the perfect MBA approach to increasing profits in this case. Sure you lose some more users but that's not relevant for profit. By the time it might matter those in charge would have cashed their bonuses and moved onto the next company.

> At the same time they get a massive amount of money from Google that isn't based on the number of users.

Is that really true? Surely, even if the contract isn't directly tied to user numbers, it's relevant whenever the deal is up for renewal...

See the last sentence. The people involved won't be there anymore by then. Short term focus is pretty standard for corporations.

> Short term focus is pretty standard for corporations.

I would rewrite this as "Short term focus is pretty standard for those not making the product."

From everywhere I've worked, the more actual work a person is doing on a product, the better they want to make it, and the less they want to deal with architecture problems down the road. The short term morons are always the ones that aren't actually making anything.

Sure, but the people making long term decisions in corporations are generally not the ones making the product. I would say in a way everyone cares about their own personal objectives and goals. Someone making a product cares about the joy they get from seeing it released and running and people using it. They want to minimize their own annoyance down the line. Those higher up care about the money they get in their bank account from their bonus. Neither particularly cares about the success of the corporation.

> Neither particularly cares about the success of the corporation.

Well written.

And sometimes, let's say for an online betting site, maybe that's a make sense mindset -- I mean, to not care.

Why think about the poker game company's future or the shareholders future profits -- why would they be and more important than oneself.

And maybe the existence of many companies is as meaningless as online poker games

Then who's doing the reviews? Higher paid people with more responsibilities?

Same issue with all our addons. What’s even more frustrating is that unlisted addons are also affected, you cannot even sign an addon since it needs to go through a manual review now.

Ruffle nightly builds broke for a good week because Firefox signing broke. We eventually decided to let nightly builds continue without a working Firefox release, but the end result is that Firefox users can't update Ruffle anyway.

Smells like an unpatched critical vulnerability.

I think that Mozilla are trying to make addons disappear.... they started with Firefox Mobile, only about 10-15 addons are available now (which is why I use an ancient version of Firefox Mobile).

They are getting a lot of money from Google, so it MIGHT be a request from Google.

Um, it's just a delay in add-on approvals. In this thread you can even see one complainant report back that their add-on was approved a day or two later.

Two months is a really long time to be delayed. And they probably prioritized the one person who complained, and everyone else is still delayed.

It's a company with less than a thousand employees, compared to Google's 100k. The author didn't even state what their extension was, so for all we know it could be a legitimately sketchy add-on with tons of bloated JS that requires extensive review.

Is uBlock Origin being unable to update any more concerning to you?

https://reddit.com/r/firefox/comments/pv15k2/_/he7qm5u/?cont...

Their latest version was released on Sept 15th, 10 days ago. Can't imagine this version has been in manual review for even a week.

Edit: Nevermind, it was submitted 10 days ago. The latest on Firefox is from months ago. Concerning, but I've had plenty of apps on the App/Play store take longer for review with the most innocuous updates. We moved to OTA updates because of this.

The issue is the signing of self-hosted dev builds being stalled.

It had always taken only a few minutes before I would get a self-hosted dev build to be signed, allowing for the dev build to be used by volunteers so as to be able to spot regressions.

I haven't been able to get a signed self-hosted dev build for 12 days now. This means I can't move forward with a stable Firefox release -- which is manually reviewed and is expected to take a number of days.

The signed self-hosted dev builds are required steps to keep releasing stable releases, and they are currently stalled for unknown reasons.

Has this been resolved yet? Should we collectively be reading between the lines?

What does number of employees have to do with anything? First of all, most addons don't undergo a manual review process, so this means something is wrong with their automated system. Secondly, Google does a whole lot more than just make a browser so comparing total employees of the two companies is meaningless.

Are there any forks of Firefox with these add on limitations removed?

I'm speaking outside of my bailiwick here, but are browsers now the equivalent of Microsoft Windows? Great big, crufty, no-one-really-knows-how-they-work, security risk laden, feature-fat, clumps of software that everyone uses?

Correct.

Yup!

Worse in a certain way as part of the problem are "feature-fat, clumps or over-complicated" web standards every new browser would have to implement.

And then Mozilla wonders why people don't use Firefox.

Yes, this kind of thing just shouldn't happen after the "sudden global outage of all add-ons [incl. privacy ones]", followed "no, no, don't worry guys, we forced out a fix via a secret backdoor!"

https://news.ycombinator.com/item?id=19823701

> [incl. privacy ones]

I remember when all my privacy extensions were entirely absent and I accidentally surfed the web. I felt so vulnerable after that, as if I surfed the web bareback

The vast majority of users are not extension developers. I'd not heard of this issue until now. It sucks, but it wouldn't make me decide to stop using the browser.

You're not looking at the full picture. Mozilla's pure incompetence to even maintain their addons ecosystem. Remember when all live addons failed due to a cert failure? Besides dumbing down the browser now they can't even seem to get addons running smoothly. Extensions/addons are a core part of a modern browser.

It's `Armagaddon` part deux!

https://bugzilla.mozilla.org/show_bug.cgi?id=1548973

https://hacks.mozilla.org/2019/05/technical-details-on-the-r...

But this is different. The addons still work, they just don't update.

It's a different outcome but it is again a problem caused by their signing requirement bullshit. It wouldn't be that bad if they would just let us developers use our unsigned extensions.

Reply on news.ycombinator.com