Improving Firefox stability on Linux
hacks.mozilla.org
hacks.mozilla.org
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1679430
Unfortunately the specific bugs here are in library packages that Firefox uses and those have no such restrictions.
/opt/firefox-nightly % ls *.so
libfreeblpriv3.so liblgpllibs.so libmozavutil.so libmozsandbox.so libmozwayland.so libnss3.so libnssutil3.so libplc4.so libsmime3.so libssl3.so
libgraphitewasm.so libmozavcodec.so libmozgtk.so libmozsqlite3.so libnspr4.so libnssckbi.so liboggwasm.so libplds4.so libsoftokn3.so libxul.so
As a linux user I appreciate the work, but it's tough to read this blog post and not think about the wasted effort that could be used to fix the bugs in the upstream software itself.
For example, NVIDIA's windows drivers are like 600mb at this point yet they don't even include function name symbols. It's inexcusable.
> Arch Linux defines simplicity as without unnecessary additions or modifications. It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release.
It's one of the core things that has kept me on Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration. I can trust that the software I use is simply vanilla upstream software with little or no modifications, and that's a great advantage when it comes to filing upstream bug reports and working on patches. In addition, it means the Arch Wiki is fairly general in its applicability, and effort spent documenting software for Arch can apply equally well to, for example, Void Linux (which also has this "vanilla software" philosophy).
I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.
* after "BootHole" GRUB vulnerability, I've read that upgrade requires re-installation of the bootloader. And just in case, did run `sudo grub-install …`. After reboot, system didn't boot. Had to use installation USB to restore.
* More recently, during routine upgrade pamac (Manjaro's pacman alternative) GUI showed me some "transaction can't be completed" error message. Shuddered it away - few days later pamac wasn't starting at all. Starting it from terminal showed error message about some missing *.so file. Googling showed that this is a required file for pacman (Arch package manager) to function. Typing `pacman` in terminal showed "command not found" error message. Restored the missing *.so file from snapper snapshot (thanks btrfs!), after that pamac started fine and happily upgraded my system.
I'm not sure what happened in second case and why pamac left system in broken state (looks like it wanted to upgrade pacman by first removing old files and then putting new files in place, but aborted in the middle), but first one might be quite distro-independent.
Also, reading through recent Arch news, I believe this could bite someone:
https://archlinux.org/news/sshd-needs-restarting-after-upgra...
> After upgrading to openssh-8.2p1, the existing SSH daemon will be unable to accept new connections. When upgrading remote hosts, please make sure to restart the SSH daemon right after running pacman -Syu. If you are upgrading to openssh-8.2p1-3 or higher, this restart will happen automatically.
FWIW, as a vanilla Arch user, I have not encountered this issue. I remember a time when pacman updates were kind of iffy (and in fact, pacman itself asked you to update it first before proceeding with the rest of the updates), but since 5.0, all subsequent updates have been completely unremarkable in the best way.
- Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
- I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset
- If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break
- If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once, every app that didn't get upgraded will fail to start due to soname mismatches... and that can mean that pacman, sudo, etc are all busted (this has actually happened to me)
- If you do a full system upgrade and have AUR stuff installed, you need to be sure to upgrade the AUR stuff otherwise it could break due to being incompatible with any library that was upgraded.
- In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
Okay I just wanna point something out here, partial upgrades in Arch are broken by design because they simply can't work. Don't do it.
That being said...
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
... kernel ABI is stable and even ancillary interfaces are usually stable, so usually it's quite A-OK to not immediately upgrade the kernel.
That said, the vast majority of the time only upgrading a particular application/package (and its dependencies) will work just fine. It's just that there are no (official) guarantees.
Debian's way of working is labour intensive, requiring packagers to fork, follow and maintain fixes and security issues in the version of the software they're packaging. They generally do a great job, but this is not a sustainable approach for smaller distributions. Arch Linux on the other hand follows upstream in real time, so you get the latest fixes directly from upstream, but there is no real Arch Linux "version buffer" that allows to freeze the versions of (parts of) your system. You move with the stream, that's sort of the philosophy of rolling distributions like Arch Linux.
The real trick is that Debian packages have sonames in their versions, so ABI compatibility is encoded in package dependencies. So when I try to downgrade a library (to a known older version from snapshot.debian.org), it knows precisely which packages depend on the new version and forces them to be downgraded as well.
This is entirely orthogonal to following upstream vs. backporting patches. Sure, Debian (stable) does backport patches, which makes it more likely that single package downgrades don't downgrade half of the system, but it really is a different thing. Debian testing/unstable follow upstream to a larger extent than Debian stable, and upstream fixes are usually preferred to patch backports. Still, partial upgrades and downgrades almost always work without trouble.
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
I imagine this is mainly a problem if you have an Nvidia card (which would explain why I haven't had the problem).
> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
More often, sure. But for software delivery from the production side, the common viewpoint seems to be that deploying more often leads to less pain in total, because you're making smaller increments so the individual failures are smaller as well.
As such, aside from integration issues like Xorg and the drivers, I would expect Arch to have fewer major breakages than (e.g.) using LTS releases of some distro and upgrading every second year.
This is mostly true for production infra (at least for your own software in production infra) but I'm not sure this logic flies for personal machines and software you don't directly interact with. I have used arch for at least 7 years and I have had upgrades render my system unbootable (or nearly unbootable, like Xorg/GDM/Gnome failing to start). These issues were mostly not due to my configuration needing to change, they were due to breakages between different packages that would end up being fixed in some later minor version.
As an end-user upgrading distros every 2 years, I really doubt I'd hit many major breakages each time as all of those pieces of software will have been in the wild for a bit and major bugs will have likely been fixed. I think the system-level issues that are resolved by frequent deploys are stuff like "systemd deprecated setting X in service unit files, so I need to update my config" or "library xyz changed some API so I need to update my app", etc. With linux distros that have coordinated releases like fedora/ubuntu/debian/etc my software's interaction with the distro may break, but for the most part the major inter-package relationships within the distro get some amount of coordinated testing which doesn't happen with rolling release distros.
Put another way: deploying more often is great for quickly discovering problems introduced in software that you write. Deploying every single dependency in a linux distro more often is going to cause you to hit every bug destined to be fixed in some minor version of the software in pieces of code that you are very far away from and lack the context to quickly debug. So you will hit a much higher sum total of bugs that would have otherwise ended up getting fixed whether you personally hit them or not. However, if you are developing a linux distribution itself, then yes your CI infra should be constantly upgrading dependencies.
I'm not convinced that an arch-paced rolling release at a distro level will ever reach a point where inter-package dependencies do not cause totally unexpected bugs given just how many inter-library dependencies there are. The entire OSS ecosystem would need to write a ton more tests for this to be a reality, and distros would need to run that full suite of tests any time and dependency is upgraded. And even then, there's still so much possibility for breakages given that shared libraries don't do a fantastic job of versioning and the upstream vendors can't possibly ensure their software works against versions of a library they never ran it against. There's a linus torvalds post about this: https://lwn.net/ml/linux-kernel/CAHk-=whs8QZf3YnifdLv57+FhBi...
Despite all of this, Arch is actually amazingly stable, and the entire community is probably better off due to the existence of arch and arch users. We might be the best integration test there is for OSS software.
> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.
You accidentally put a plural in "vendors", but in practice this is just Nvidia.
If you are forced by circumstance to deal with an Nvidia card, use the LTS kernel and most of your problems go away, and you can also pin your X.org version and manually update it. The real solution is to use a GPU vendor that has mainline drivers, the only valid reason nowadays not to do that being CUDA or being unable to acquire hardware with mainline support.
> - I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset
That's a gedit bug. Complain to gedit devs or stop using it.
> - If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break
You should not be using system python for non-system tasks. Use asdf, Nix, Docker, pyenv or a similar tool for projects requiring their own non-system environment.
> - If arch upgrades the major version of[...]
Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.
> - If you do a full system upgrade and have AUR stuff installed
AUR is not Arch, it's up to each AUR maintainer to keep their scripts up to date and you as a user to keep up with those external dependencies. A common approach is to use an AUR helper to handle both system and AUR upgrades.
> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.
And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.
> Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.
Definitely correct, the issue is when you're trying to install a package to get something done, and suddenly your faced with the proposition of either upgrading your whole system while trying to get work done or attempting to upgrade just the required libraries. The latter often carries less risk, but occasionally is very problematic.
> And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.
The issue is that this is true of so many very major pieces of linux software. Upstream vendors don't necessarily test their stuff in a ton of contexts, plenty of bugs occur in the kernel/xorg/wayland/gnome/glibc/python/etc. All of these are very mainstream projects that are difficult to avoid.
I found more problems using e.g. debian stable. Upgrades were totally black box and anything could break spectacularly. This was most commonly caused by mixing up to date software you build yourself and the stable repos. Arch on the other hand updates happen daily, and for me, unattended. If I ever do find issues (rare to never), backups fix it.
For any definition of stable.
I may be wrong, as my default is to -Syu any package I install, but doesn't -S install the version of the package compatible with the last time you updated your repos? It's -Sy that can cause problems.
> Partial upgrades are explicitly unsupported by the distro.
The problem is obvious. Sometimes you might have to pin a package version because the newer version does not work with your hardware driver, as you said, but then again Arch calls this unsupported.
I had the same problem in the past, not with Nvidia but with DisplayLink. For that reason I always leave a USB stick with a bootable Arch ISO in my Thinkpad, so that I can arch-chroot and repair the system. It happens rarely, but it happens to me and others as well.
this was what convinced me to switch to Debian back in the 90ies. It was the only distribution that could do this without throwing a tantrum. I was coming from RedHat and SuSE and both had huge problems with this. Debian presented a higher learning curve (at least that was what people said). Debian really only lagged behind back then with their graphical installer and the overall lack of integrated DE back then (compared to others).
I really like the Arch wiki as a Debian user but never used Arch itself. Not going to change because old dog, new tricks ...
I've also been using Arch on a "for messing around; I don't care if it breaks" laptop for about two years, and haven't had any major breakages.
For me, the most noticable difference has been that Debian will install new kernels as new packages, will suggest removing old kernel packages, but won't suggest removing the currently booted kernel. I like this behavior. By default, Arch will swap your currently booted kernel and modules out from under you. You don't necessarily need to reboot right away, but if you don't, you can run into issues loading modules or recovering from hibernation.
You can work around this once you realize what's happening, but it seems like a needlessly dangerous default.
My favorite documentation is the arch wiki and there has never been an issue due to it not being compatible to the "Debian way"
When upstream devs package for a distribution they usually put some effort into it. Just because a package gets published to Sid/unstable doesn't mean the package is unstable. There are some devs that mostly work on higher layer GUI and user-lamd applications who are perhaps inexperienced or reckless who do sometimes introduce breaking changes. It's rare though since most people understand that packaging for a distro means a potential huge number of issues if they're not careful
My 70+ year old aunt also runs sid since a couple of years with unattended upgrades since I installed it for her (and she has KDE). She never had issues with stability or things not working and she uses her computer daily for writing (libreoffice), printing and research (probably reddit idk, I didn't ask :)).
[1] my install is fairly minimal: not much gui, server systems etc are started with docker so hardly anything actually runs on bare-metal which can cause issues during upgrade jeopardizing the things I work on (e.g. due to config file changes). And I don't have a huge DE like gnome/KDE. My sway/wlroots and i3, and tooling like x-terminal/wofi/Waybar/etc are always compiled from git/source. All my dev tooling is the latest and I can still install older versions of clang/gcc or other environments with my package manager if I have to.
When several of us raised the issue in the forum...I'll just say we didn't get a warm reception. That was more than a decade ago, so things may be completely different now, but it left a bad taste in my mouth.
I've been using Arch for years and I never experienced any breakage. I update it every month or so and it's still stable even after these huge updates. There's nothing for me to do other than merge .pacnew files.
As engineers we have no one but ourselves to blame for this one
This has drawbacks too though, in that it's now up to app makers to take care of keeping supporting libraries up to date and secure. That's not necessarily their top priority though, which is a risk for the end user. Also, unnecessary duplication of libraries increasing memory use, when the distro is able to share most of them. Plus the poor middleman has an incentive to set user-friendly, privacy-preserving defaults that I wouldn't trust as much when the package is provided by a commercial company with different incentives.
It's great to have the option, but overall I would still prefer to use distro packages except in the rare special case.
The nice thing about Arch packages, though, is that they're basically just bash scripts. And if all you need is a simple version bump, it's usually quite easy: download the package tarball from [0] and change [1] to the version you want, then do a `makepkg -s` in the directory of the package. It will build the package in a (usually reproducible) chroot. If it builds without errors, then you'll end up with a tarball that you can `pacman -U ${pkg}.tar.zst` to install the produced files.
If you need help, makepkg documentation on the wiki[2] is pretty great. And don't forget to send a patch to arch-dev-public[3] and CC the maintainer. At the very least, it'll kick off a discussion that will get the package updated.
Rolling your own packages is easy in contrast to, say, Red Hat - where compiling an RPM is easy if you've done it a bunch, but really difficult to get bootstrapped on.
[0]: https://archlinux.org/packages/extra/x86_64/jdk-openjdk/
[1]: https://github.com/archlinux/svntogit-packages/blob/packages...
https://lists.archlinux.org/pipermail/arch-general/2021-May/...
https://lists.archlinux.org/pipermail/arch-general/2021-May/...
This is incorrect. Packages in both core and extra are maintained by the core Arch developers. You're probably thinking of the community repository, which is maintained by a group called Trusted Users. These are still heavily vetted, it's not just anyone from the community. Or maybe you're thinking of the AUR, which is a completely untrusted repository of build scripts, which anyone can submit to.
In this case, the issue is that anthraxx, one of the Arch developers [1], has not updated many of their packages in some time. I don't know if a reason is known, but you might find something in the mailing list links someone else has posted.
For the topic I think is good to have dfsg and to patch any software with the goal to provide better integration with the system and for user's freedom.
My desktop has looked and behaved the same for almost a decade.
I migrate to a new machine by dd'ing from backups and haven't installed fresh in years.
These things are what keep me around.
>For example, at some point, Debian updated the fontconfig package by backporting an upstream fix for a memory leak. Unfortunately, the fix contained a bug that would crash Firefox and possibly other software too. We spotted the new crash only six days after the change landed in Debian sources and only a couple of weeks afterwards the issue had been fixed both upstream and in Debian. We sent reports and fixes to other projects too including Mesa, GTK, glib, PCSC, SQLite and more.
That sounds a lot like Linus' "many eyes make all bugs shallow" idea working as intended.
> the libfontconfig1 package was updated on Debian on the 21st of April and the first crash we have on record was sent on the 22nd. Here's the changelog:
I think that's a good argument for this kind of crash-telemetry, working in conjunction with Linux distributions. And it helped, in this case, that the dependencies could be updated independently from Firefox.
On the other hand, the libdrm hurd patch breaking things on Linux seems like an excellent example of how portability to obscure platforms has a non-zero cost, and it isn't as simple as "just accept portability patches".
Did they? Looks like it's still there:
https://www.mozilla.org/en-US/firefox/all/#product-desktop-e...
It's not like PyPI/rubygems/npm/whatever where any random person can make an account and start pushing packages. The maintainers have to actually trust you.
Subjective, but needless to say I disagree.
I disagree with all of your reasoning too. Inserting a middleman into the software distribution process is needless friction and adds another point of failure. Even Linus has spoken on how much of a pain in the ass it has made it to distribute software: https://www.youtube.com/watch?v=5PmHRSeA2c8&t=340s
But hey, you guys keep doing what you're doing and ignoring what potential users want. I'm sure the Year of The Linux Desktop is just around the corner.
P.S. Yes, I know, it's been your Year of the Linux Desktop since 2001 or whatever. We both know that's moving the goalposts. So is saying "but Android" since it replaced everything above the kernel with a completely new userspace stack and, oh yeah, it doesn't really run on the desktop anyway.
>But hey, you guys keep doing what you're doing and ignoring what potential users want
Both me and the parent users are not "potential" users but actual users. Why we shouldn't count but some "potential" users should? We already have plenty of distributions which don't do patches.
Parent poster didn't mention anything about "Year of the Linux Desktop" so your diatribe about it is completely uncalled for too. Plenty of people are absolutely fine with current trajectory of user base and don't subscribe to prioritizing user growth above serving existing users.
It's a bit lame when a positive well-written article gets the usual negative response about how everyone is doing everything wrong and "Linux desktop is completely broken".
Because at this point I guess I just assume I'm talking to an evangelist about 90% of the time. That seems to be the case with people who insist that the Linux Desktop Experience is "actually pretty great".
> Some users do find value in having different flavor distributions and them exercising some control on the source, you obviously do not.
The fact that Linux Desktop needs hundreds of bespoke distributions to offer those choices is precisely the problem, in my opinion. A well put together system shouldn't require everything to be compiled just-so and all together by an army of volunteers to work properly. You should just be able to use the software you want, as up-to-date or out-of-date as you want, why is that so hard in Linux?
And anyway your app based view is slowly becoming more prominent, with so much development dedicated to various app based solutions(appImage, flatpak etc). You don't see me complaining about army of volunteers wasting their time. People work on stuff they think is useful to them, nothing to see here or be negative about.
I've watched that Q&A before. Linus is of course right but not because of package managers and not because of distribution maintainers. He eventually starts talking about binary interfaces which are the root cause of this issue.
The fact is most open source projects out there don't have stable binary interfaces. So we have important libraries in the system breaking working binaries because they do things like change structure layouts and function signatures. This is a problem of the wider free and open source software ecosystem in general. In that same talk he contrasts this with the kernel which does have a stable binary interface and can therefore effortlessly run binaries compiled in the 90s with no changes at all.
If all software had stable ABIs, you wouldn't even need packages at all. You'd be able to just compile stuff and distribute that binary directly.
> But hey, you guys keep doing what you're doing and ignoring what potential users want.
Sure thing. I don't really care at all about what "potential" users want. I'm a programmer and I want a system suitable for programmers. "Normal" people are more than welcome to keep using whatever it is they're still using. If they don't see the value I'm not gonna convince them.
With Linux I get to trust that if something's made it into the package repositories then it's trusted by the community. That also means maintainers fix idiotic "features" like old and insecure bundled libraries in the case of dynamically linked distributions, telemetry in the case of a privacy-focused distribution or non-free code in the case of Debian.
An issue exacerbated by the package managers and repo model. Developers think "why should I bother with binary compatibility when package maintainers are just going to recompile everything anyway?".
> If all software had stable ABIs, you wouldn't even need packages at all. You'd be able to just compile stuff and distribute that binary directly.
Gee, it's almost like other Desktop OSs did that.
> Sure thing. I don't really care at all about what "potential" users want. I'm a programmer and I want a system suitable for programmers. "Normal" people are more than welcome to keep using whatever it is they're still using. If they don't see the value I'm not gonna convince them.
That's fair, I'm just tired of dealing with evangelists. There are reasons people use other OSs and it isn't because they're stupid or uninformed or whatever nonsense people use to feel superior. In my case, it's because I don't like the way Linux does things.
> With Linux I get to trust that if something's made it into the package repositories then it's trusted by the community. That also means maintainers fix idiotic "features" like old and insecure bundled libraries in the case of dynamically linked distributions, telemetry in the case of a privacy-focused distribution or non-free code in the case of Debian.
They also fix idiot features like actually-working cryptography (Debian), or a "you're using a horrifically out of date version" notification from the developer (also Debian). Oh, and of course the whole model really can't handle simple tasks like:
- Installing software to different disks
- Installing more than one version of the software (unless explicitly packaged separately)
- Choosing to have a mix of old and stable versions of some software and the latest hotness of other software
These are all things that are trivial in other operating systems, but I need to use something like AppImage to get them in Linux software. Sadly not a lot of software is distributed that way because the culture of Linux is that everything must flow through a walled garden repo for your own good, and if you don't like then compile from source.
Because it's good engineering.
> Gee, it's almost like other Desktop OSs did that.
They don't. Every now and then Apple breaks mac applications like they're nothing. Microsoft used to care about this to an almost heroic extent but that's history. I literally can't play some old games on Windows 10 without troubleshooting stupid DirectX DLL errors. Ironically that game just worked on Linux.
> That's fair, I'm just tired of dealing with evangelists.
Wait, so anyone who's happy with Linux is an "evangelist" now? Why even start this conversation if you don't actually want anyone to reply?
No, not everyone who likes the Linux Desktop is an evangelist, but when I come out and say the experience sucks and the very next reply is "actually it is great" it is hard not to read that as evangelism.
Maybe I'm reading too much into it, but what I said was "I find this experience terrible" and what they said was "you are wrong". If they had left out the "actually" it would have read differently, but as it was it seemed like just another evangelist trying to tell me what I do and don't want in an OS.
Note that "Linux" in this case refers specifically to typical userland, the kernel ABI, by contrast, is quite stable.
Let's review. You said that the Linux desktop experience is "awful" and provided no evidence whatsoever for your claim. The reply to you, "It's actually pretty great" at least provided a couple of points in favor of believing that the situation on the Linux desktop is good, at least for some people.
From where I'm standing you're the evangelist, insisting that their own picture of how a Linux desktop should be architected is the only good way with no evidence. Claiming that the way things are currently done is garbage, with no real evidence. I think it's totally unfair for you to claim that you were just giving your opinion and then had other people walk all over you and "tell you what you do and don't want in an OS".
Rather, what people in this thread have been trying to get across (I think) is that we like Linux the way it is. We think it's "pretty great". We don't want to see it change because a bunch of tech "evangelists" (if you'll pardon the expression) think static linking and AppImages are a much better way to do things. Sure at the end of the day that's "subjective" (meaning it's the result of us finding certain things valuable that not everyone else finds valuable), but, well, so are any alternative conceptions of the Linux desktop.
> Rather, what people in this thread have been trying to get across (I think) is that we like Linux the way it is.
And that is fine, I don't want to harass people who use and like Linux any more than I want to be harassed for not using. What I want is to not have to keep having the same arguments over and over about why I feel the way I do about it, because the community has made it clear they don't care.
> We think it's "pretty great".
IF parent had said "I disagree, I think it is pretty great" we wouldn't be having this conversation.
Your opinion about this mythical Linux Desktop was not the reason why I replied to your post.
I replied to your opinion on the packaging model. It is great and in my original post I explained why I think so. It's a highly successful model. Just look at the huge number of platforms with app stores these days. It's fundamentally the same thing. Apple is notorious for enforcing quality control on apps and forcing them to integrate with their wider ecosystem. Linux invented it with software repositories and automatic package managers and it's even more open compared to modern solutions.
I disagree. I do not read this statement as containing an opinion qualifying phrase about whether the Linux Desktop experience is awful, the opinion being qualified here is that the repo/package manager model is the root cause of it.
> As far as I am concerned the whole repo/package manager model is the root cause of most of the reasons the Linux Desktop experience is as awful as it is.
You appear, in this statement at least, to just take "the Linux Desktop experience is ... awful" as an underlying objective fact that everyone should just accept, and we're all debating the reason for it. I don't accept it.
As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.
Earlier discussion: https://news.ycombinator.com/item?id=27009044
This is misleading and in fact not even strictly true. They link three bugs, not two: https://bugzilla.mozilla.org/show_bug.cgi?id=1633459
This third bug was not due to Debian patches. And in fact the two Debian patch cases in the article were included to make the point that they can now catch issues caused by distribution patches, not to claim that most of the issues they find are caused in this way. Assuming on the basis of an article written to make a wholly different point that because 2/3 were the result of distribution patches that this must be a huge problem in general for Linux software development just shows your bias on this issue.
I like and use Arch Linux on my desktops. But I'd defend the choice by Debian to patch. Most of Debian's changes are intended to improve support for certain setups or introduce bug or security fixes that upstream hasn't backported. These are both good things. Furthermore, problems that are created tend to be caught while they're still in unstable or sid: if I'm not mistaken that's exactly what happened in these cases, which is the process working as intended. (Mozilla is certainly free to disregard bug reports from Debian users if they feel that Debian's patching process is simply causing too many problems.)
Perhaps there are just as many escapes being found each quarter in Chrome as the other browsers and they are just being hoarded privately but I don't find that super plausible.
FWIW I think it is probable that the IPC APIs into and out of the sandbox have been more thoroughly fuzzed and otherwise tested in Chrome than in Firefox. I don't know how that translates into actual security though.
And here's Theo de Raadt's opinion of Firefox from back in 2018: https://marc.info/?l=openbsd-misc&m=152872551609819&w=2
> Firefox's sandboxing lacks any site isolation. Site isolation runs every website inside its own sandbox so that an exploit in one website cannot access the data from another.
It does seem that Firefox's site isolation is becoming more ready. From a Mozilla blog post two days ago, with instructions how to manually enable it on Firefox stable, beta, or nightly: https://blog.mozilla.org/security/2021/05/18/introducing-sit...
Also, from the same link, they mention X11:
> One example of such sandbox escape flaws is X11 — X11 doesn't implement any GUI isolation which makes it very easy to escape sandboxes with it.
Definitely true that X11 sucks (sorry NVIDIA users). So we have Wayland becoming more mainstream now. I've been using it for several years already, on GNOME and Sway. Working great... even Electron is native now (Signal, VS Code, etc).
And lastly, they rightly mention Pulseaudio:
> PulseAudio is a common sound server on Linux however, it was not written with isolation in mind, making it possible to escape sandboxes with it.
In the last few months PipeWire became a mature drop-in substitute for Pulseaudio in my experience. It was designed with isolation in mind, and a whole bunch of other things.
Hoping Firefox can bridge the gap in security with Chrome so we can wholeheartedly recommend it to people without caveats! We deserve a fast, full-featured, secure, and open source alternative to proprietary web browsers.
A slightly tangential issue is that the mitigations section is not super compelling to me because I think many mitigations are low-value. Evaluation of mitigations typically does not ask the right questions: How much work is it for an attacker to bypass the mitigation, assuming they're aware of it? Can that work be packaged and reused in multiple exploits? And how many bugs become completely non-exploitable due to the mitigation?
I personally don't even care whether or how much feature parity Firefox has or whether its GPU rendering is 20% slower. Not supporting Firefox comes at our own loss.
Also, there is no such thing as an "ungoogled chromium". Google controls chrome/ium, and as long as people use it, those people remain subject to Google's control.
In that article there's something I disagree with having tried:
> Most of the functionality of the patches are either in the best case minimally beneficial or can be reproduced with either a setting, a flag, or a switch,
Some years ago, on Linux, I tried finding all the command line flags necessary to use Chrome without having it talk back to Google. Unfortunately despite hunting for every option I could find (and there are a surprisingly large number of undocumented options), including with "strings", nothing I tried completely suppressed traffic to Google while using Chrome on sites not connected with Google.
My motivation at the time was to run my own local applications using Chrome as the UI, the way Electron is used now. It didn't work out because I failed to find a way to confidently suppress traffic to Google.
That experience is why I run Ungoogled Chromium now when I need Chrome functionality.
For me Google _is_ a third party. And taken into account its behavior it receives 0 (null) trust from me. What i really don't understand is this blind trust in everything google does.
That said, the "binaries built by anyone" thing is pretty suspect.
If you exclude most hardware acceleration on majority of Linux PCs that currently run FF and if you exclude following standards that good and normal software does[1].
Both issues exist nearly decades now. "Great"? Absolutely not.
What's killing me is the memory leak that just renders the whole computer unusable and almost frozen. Almost because if I can grab a terminal and killall firefox then I get the machine back.
https://bugzilla.redhat.com/show_bug.cgi?id=1597028 something like that, not sure about the root cause. But I don't let imgur tab with running vid opened for too long now.
Memory leaks can be bad indeed and here are two tricks that work fine for me: either run Firefox inside a Docker container... Docker container on which I put CPU and RAM quota. It's IMO way easier than try to put resources quota directly on Firefox. The other one: Firefox has zero issues reopening all my tabs, so I simply kill it from the terminal when I see that it's going wild on memory. Now I've got 16 GB and Firefox rarely eats it all and it's usually not sudden: it's a slow bleed over the course of a few days... So when I get to something like 10 GB of memory used by just one Firefox instances (I typically run several Firefox instances, from different user accounts) , I just kill it and restart it, while choosing to reopening all the tabs (usually tens of tabs).
I'm sure there are other methods too but this works fine in my case so I haven't looked much into it.
What works for me is having the `about:memory` tab open and clicking on "Minimize memory usage" at the end of the day when I suspend the machine. And, I have a lot of tabs and windows open, strewn over various virtual desktops. Though most tabs are not loaded but more expensive bookmarks, but I have no add-on which actively unloads tabs.
What I do know is that the way Linux GUI distros deal with low memory situations is absolutely garbage. The new systemd OOM daemon that's shipping with systemd 248 will hopefully improve this situation, but for now I'm left running nohang[1] on my dev machine, where I'm consistently running out of RAM (IntelliJ + tons of Java frameworks + huge React code base + a web browser all take up way too much space!). Enabling zRAM also seems to work, but I haven't measured the effectiveness of that yet. On my home PC with double the ram (32GiB) everything runs smoothly, but the way Linux on the desktop deals with OOM situations is still atrocious. Besides, none of these tools should be consuming such ungodly amounts of RAM anyway.
For server situations, Linux handles OOMs better than Windows, IMO. On desktop, the same strategy just doesn't work and only makes working on it more painful.
1. It comes preconfigured: https://build.opensuse.org/package/view_file/openSUSE:Factor...
2. Packaging adoption: https://repology.org/project/earlyoom https://repology.org/project/nohang
3. Written in C and unburdened with tangential features like desktop notifications, the daemon's process takes very little amount of memory.
In /etc/rc.local:
cgcreate -t USERNAME -a USERNAME -g memory:firefox
echo 4294967296 > /sys/fs/cgroup/memory/firefox/memory.limit_in_bytes
echo 4294967296 > /sys/fs/cgroup/memory/firefox/memory.soft_limit_in_bytes
cgexec -g memory:firefox /usr/bin/firefoxNow if I could only disable Ctrl+Q and prevent it from asking for a restart after update...
But otherwise, FF has been really stable for years now
It's nothing that would make me stop using it, just an annoyance that I wish I didn't have to deal with on a semi-regular basis.
Good news! =)
Set browser.quitShortcut.disabled in about:config
That's awesome, just did this. But apparently you need to restart Firefox in order to apply the change. (Learned it the hard way)
This isn't just nagging. The API between the main browser context and content processes isn't stable between builds. This means that Firefox can't spin up a new content process if the update has replaced the executable with a different version. Therefore Firefox has the tough choice of trying anyways and risking misbehaviour (including possibly security bugs) or forcing the user to restart the browser process so that it can spawn new content processes.
It also depends on how your distribution updates Firefox. For most distributions they have this problem, however NixOS doesn't because the new version is installed in a new directory.
IIUC Firefox's self-updater also doesn't have this issue, and I think it supports Linux.
Couldn't it spin up another one from the old version by doing execve("/proc/self/exe", ...)? Or by keeping a template process around with all the libraries loaded from which a child process can then be forked.
EDIT: And at the risk of stating the obvious, the fact that it doesn't work this way should be a hint that there are complexities involved that make it more complicated than "just" doing it.
EDIT: Oh, I see. This happens when the distro package manager updates firefox. I don't have a solution, since there isn't an easy one that will satisfy all people. So nevermind. shrug
Maybe Firefox could create a temporary executable copy of itself and run from there, and run that, so when the original is replaced, it doesn't affect the running process. But, that probably comes with a host of complications that I'm unaware of.
You'd have to enable all users to replace files owned by root somehow - which is a bad idea. Alternatively you'd have to somehow extend the packaging to support owning either/or file and being able to verify the installed files that way - which would be a massive change.
As for the template process one major issue is that this means that you only get ASLR once on browser start, instead of for every content process, which somewhat reduces the benefits.
99% of the Firefox code is in libxul.so, so you'd need to get the file handles from the original process or something similar.
A workaround would be to use Firefox directly from upstream instead of the distro packaged version (I understand some people are not happy with this tradeoff).
But I agree with you, I can't remember when was the last time I had a crash before these!
This posed a significant problem when dealing with stability issues on Linux: for the majority of our crash reports, we couldn’t produce high-quality stack traces because we didn’t have the required symbol information. The Firefox builds that submitted the reports weren’t done by us. To make matters worse, Firefox depends on a number of third-party packages (such as GTK, Mesa, FFmpeg, SQLite, etc.). We wouldn’t get good stack traces if a crash occurred in one of these packages instead of Firefox itself because we didn’t have symbols for them either.
To address this issue, we started scraping debug information for Firefox builds and their dependencies from the package repositories of multiple distributions: Arch, Debian, Fedora, OpenSUSE and Ubuntu. Since every distribution does things a little bit differently, we had to write distro-specific scripts that would go through the list of packages in their repositories and find the associated debug information"
This is why I'm glad that a lot of Linux software is moving towards distro agnostic containerized applications maintained by developers. Linux already does not have a large market share and if you have to deal with a dozen distro-specific quirks it seems hardly economical.
To me, it entirely depends on the specific packaging job, not even who is doing it. Entirely too often, they ignore distro convention, and can introduce bugs and other security issues through ignorance, laziness or mistake.
Long term, flatpack and its ilk will devalue distro differences and put pressure on uniformity. Whether or not you consider this is a good thing may depend on your opinion of/relationship to RedHat/IBM.
For me, if it isn't in the Debian repos, I'm probably going to ignore it.
Can you clarify what you mean by this?
I have used both the flatpak and rpm (on Fedora) version of Firefox and don't really notice a difference. Perhaps Firefox is just one of the packages that does distro convention properly?
It was a general comment that is not specific to Flatpak - it will a problem for any cross-distro packaging. (Heck, plenty of developers mess up native packaging for distros they're not familiar with.)
Firefox on Android is a different matter and is unusable on some devices.
[1] https://wiki.mozilla.org/Data/WorkingGroups/CrashReporting [2] https://chat.mozilla.org/#/room/#crashreporting:mozilla.org [3] https://github.com/mozilla/dump_syms/ [4] https://github.com/luser/rust-minidump
It seems Firefox has a script at https://github.com/gabrielesvelto/symbol-scrapers/blob/maste..., but I haven't investigated what it does. In any case, this information should be integrated into the Arch Wiki.
Also how does unwinding info help during debugging? Does gdb use it? Do I need special reverse-engineering tools to extract useful information?
Probably need to try chromium.
I still have to mess with a billion things when I want to do anything more complicated than basic music playback, but for that it sorta works which is already pretty great by Linux audio standards.
More generally I've always been frustrated with Linux audio because in my experience for basic audio playback OSS mostly Just Worked and every single API that came after it was buggier and harder to configure for the simple cases while still usually not good enough for professional use.
One exception was Jack which I like quite a lot, unfortunately it's also not natively supported by a lot of software, and that adds a lot of complications in my experience.
But again, I've been running pulse for about six months now and I haven't had any major issues, so credit where it's due.
I would even be willing to finally learn how to use Ardour correctly!
I do have an issue where Firefox hangs (requiring a kill -9) after the end of any WebRTC call, but I don't think that's related.
I was able to reduce RAM usage by reducing the number of content processes, so that might be helpful to you also.
Yeah, I dropped dom.ipc.processCount by half a while ago, but I think Fission[1] overrides that setting.
Although new tabs starts feeling like they would be quicker to open with 5490 other tabs less. Who knew.
I guess my biggest finding is that tab and history search suck. Half of what I end up searching for is one of my tabs. I wonder why..
Anyone else come across this issue? I'm using Solus.
Haven't had a tab crash in quite some time though.
But that is really a minor nitpick compared to the tons of issues I have with other browsers; Firefox runs insanely stable, and easily handles a large number of tabs.
But it's just another visual bug users of tiling WMs have to deal with. Many programs handle this much worse, either crashing instantly because the WM doesn't let them have their favorite hardcoded window size, or becoming unusable from epilepsy-inducing glitches. Before I went down that path I'd have never guessed UIs(both on desktop and on websites) are so damn flimsy.
I can’t recall an issue with FF behaving badly when tiled. So, it’s unlikely that FF generally doesn’t work with a tiling WM.
And a bunch of others folks (I counted 3 more)
Or are you on Wayland?
There are open bugs I know. Some of which that are getting attention, others we haven't gotten a chance to prioritize. Is there a bug filed about your particular problem?
According to this : https://wiki.mozilla.org/Platform/GFX/WebRender_Where that is not true. Or is it out of date?
https://bugzilla.mozilla.org/show_bug.cgi?id=1701977
Edit: Actually this is a little incomplete. It is still turned off with XWayland in late beta and release. So unless the MOZ_ENABLE_WAYLAND=1 environment variable is set, you won't get WebRender by default. I will need to review why we didn't ship with XWayland yet -- there were a couple of bugs (e.g. bug 1635186) but I don't know the status of them offhand. Expect to see movement here within the next few releases I would say.
I'm glad to hear that. One of my computers is actually using Wayland with MOZ_ENABLE_WAYLAND=1, but I haven't actually checked if it worked yet.
I can't see what benefit you expect to get from the GPU. It is plenty fast with all-CPU rendering.
The Qubes X model is interesting: each VM runs its own X, and renders windows into memory shared with a central VM that copies the pixels to real video memory. Input events get forwarded to a VM's X according to which is the active window frame. The camera can be attached temporarily to a VM, and mike input similarly. So, no VM has physical access to hardware anytime other than when you want it to.
I am hoping a future Qubes will have proxying Vulkan stubs or something, for controlled access to a GPU. But it isn't missed much since FF stopped crashing.
I use mpv when I am serious, moving FF entirely out of the picture.
https://www.youtube.com/channel/UCsn6cjffsvyOZCZxvGoJxGg
https://www.youtube.com/channel/UCHnyfMqiRRG1u-2MsSQLbXA
https://www.youtube.com/channel/UCSpFnDQr88xCZ80N-X7t0nQ
https://www.youtube.com/channel/UCXuqSBlHAE6Xw-yeJA0Tunw
Besides even at 1080p the difference in how much battery it saves is astounding. With acceleration your CPU is just idling while without it has to work quite hard.
There's something wrong with the padding and the up and down arrows simply don't work.
I think it's already fixed on beta/nightly but this can't happen. We can't wait months to have our inputs working again.
Also there's another bug introduced a couple versions ago. If you select some text with you mouse and a the next line, when you open a new tab with the middle button it'll paste the whole thing you selected in the direction bar.
I usually select text or code when I read it and every time I open a new tab with my mouse, boom. I have to delete everything, or select a single word somewhere and close that tab to reopen a new one. And if I'm not looking and start typing I end submitting a long text to duckduckgo.