gsvelto

You're assuming that Firefox is eating all your commit space, but it's not, far from it, I can see it from the crash reports. You've got other applications running and they are also fighting for it, leaving large chunks unused. Without some swap space available Windows is leaving 10+ GiB of physical memory free which it cannot use for anything else.

This is how Windows behaves by design, there's nothing we can do about it. Windows without a swap file sucks.

From the looks of it you have a minuscule page file (16MiB), that's why you're running out of commit space. I suggest setting it to auto-resize (the Windows default) which should provide a much better experience even with other applications, unless you need to keep it small because of storage concerns. Generally speaking Windows behaves very poorly with small swap files because of its inability to overcommit memory.

Could you open a bug on our tracker, and point to those crash reports? I'm very interested in analyzing real-world scenarios where this happens to figure out where those committed areas are coming from. Our best guess is that they come from the graphics stack because we account all the memory we commit in Firefox and there's nothing missing, so wherever this is coming from it's outside of our direct control.

Yes it does, we've disabled tab unloading on Linux because low-memory detection just didn't work. We will re-enabled it sometimes down the line but only to avoid swap, not crashes. The OOM killer does already a good job with this change and we feel no need to do anything further (besides optimizing memory usage, but that's system-agnostic).

But is it? I wanted it to be tongue-in-cheek but the title is actually a rather accurate description of the contents of the article. This is a ~20 lines weird hack that we threw at the wall to see if it would stick... and it massively improved Firefox stability. I can't stress it hard enough, it's the largest, most visible stability improvement in the past decade.

More specifically - on the memory front - we spent lots of engineer/years working on improvements which sometimes would barely register, only for some weird trick make OOM crashes fall off the radar entirely.

Also I'd like to point out that we have no way to tell if this is working because we give Windows time to resize the swap file, if it's because other processes in Firefox die, or it's because other unrelated processes die, or a mix of all the above. It's pure speculation on our part.

Here's the thing: they're not! The reason those users where crashing was because something, somewhere (possibly in the graphics stack) was reserving ton of space without using it. We had crashes on file with 20+ GiB of free physical memory which was the reason why I started looking into why a machine with so much free memory could suffer a crash.

I hope I described how this whole things work because Windows memory management is not well known and some things about it are counter-intuitive; especially if you're coming from Linux.

I'm glad you noticed! We did indeed roll an improvement on Linux too, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1771712

In a nutshell we're directing the OOM killer towards less interesting processes within Firefox so that when the system is low on memory they'll get killed first. This not only makes it more stable overall but it plays nicer with other applications too. Web pages that leak memory in the background are particularly likely to be killed by this mechanism and that alone is a huge improvement in overall stability.

They do and so does jemalloc in Firefox, however that lowers contention, it doesn't let you do away with synchronization. Consider this simple scenario: a thread allocates a chunk of memory from its per-thread pool, but the object is later released by another thread. You can't do that w/o synchronization around the pool. Per-CPU pools are even trickier because a thread might be moved around between different CPUs by the scheduler while it's allocating memory. So you need special facilities to implement those in user-space like restartable sequences: https://lwn.net/Articles/650333/

You're confusing general purpose locks with special-purpose ones used in the memory allocator. Firefox has its own GP locks, you can find them here: https://searchfox.org/mozilla-central/search?q=&path=xpcom%2...*

However those are not suitable for use in the memory allocator, and neither are WebKit's. WebKit uses os_unfair_lock within its memory allocator:

https://github.com/WebKit/WebKit/blob/520379e30f3b2b6d4de995...

And so does Chromium:

https://source.chromium.org/chromium/chromium/src/+/main:bas...

No specific hardware support is needed to implement a lock. The atomic operations are all accessible from user-space - they are platform dependent though. The problem with locks is that they interact with scheduling and that's where the kernel comes in, that's why you need an OS-specific component too.

In a word: no. Efficient locks can only be implemented with significant involvement from the kernel. The best locks that can be implemented entirely in user-space will still suffer from the problems described in the article. Locks have complex interactions with scheduling and the only place where informed decisions can be made about them is within the kernel.

That doesn't mean that good locks don't need a good user-space component - they do - but it's only one side of the coin.

If you could grab a profile of the problem with the Firefox profiler and file a bug it would be greatly appreciated. From the sounds of it it's probably an edge-case we're not aware of where Firefox performs very poorly. The problem with this kind of things in Mozilla is that we're often blind until someone brings up the issue and notifies us.

Post author here, `__ulock_wait2` is used under the hood by `os_unfair_lock()`. I considered it but it would have required more scaffolding, especially to support all the versions of macOS we care about. We might use it in the future though.

Recent Linux kernels allow you to measure memory consumption from a QoS point-of-view via pressure stall information: https://www.kernel.org/doc/html/latest/accounting/psi.html

This isn't the same thing as detecting how much memory is available because Linux overcommits by default. PSI will tell you when the kernel is swapping too much (depending on the thresholds you set) and from that you can infer the user is running low on memory.

Android's lowmemkiller had an explicit free memory threshold userspace could set to be notified. Starting with Android 11 however it's now using PSI too IIRC.

Unfortunately those crashes do not appear to be related to low memory scenarios. They seem to be genuine bugs in the screen reader. I think the right bug tracking them is this one: https://bugzilla.mozilla.org/show_bug.cgi?id=1691928

Detecting low memory scenarios is surprisingly hard on desktop operating systems, but trivially simple on Android/iOS. Firefox for Android already made extensive use of this feature while desktop Firefox' available memory watcher underwent a lot of changes over the years before becoming fairly reliable.

Arch already uploads symbols for their Firefox builds so that's OK, it's the dependencies that are missing. Unwinding information is used to produce accurate stacks. If it's lacking our unwinder has to rely on heuristics to discover the frames on the stack and those might end up producing suboptimal traces.

Only public symbols and unwinding information is extracted from Arch packages because they don't have debuginfo like other distros. It's better than nothing.

That's a really tough problem to solve on Linux. We're actively working on it in order to make Firefox better behaved when memory gets tight but it's not easy. You can follow the work in these two bugs:

https://bugzilla.mozilla.org/show_bug.cgi?id=1532955

https://bugzilla.mozilla.org/show_bug.cgi?id=1587762

Yes, that's one of the benefits. Firefox has a very large user-base compared to other FOSS projects so we will often spot bugs that others haven't noticed simply thanks to the sheer volume of crash reports we get.

You're very welcome! If you're interested in the topic we've got a working group [1] and a very active channel on chat.mozilla.org [2]. Besides the actual stability work we've also been busy either rewriting some of this tooling in Rust (see [3] and [4]) as well as contributing to Sentry's excellent symbolic crate which we leverage during symbol extraction.

[1] https://wiki.mozilla.org/Data/WorkingGroups/CrashReporting [2] https://chat.mozilla.org/#/room/#crashreporting:mozilla.org [3] https://github.com/mozilla/dump_syms/ [4] https://github.com/luser/rust-minidump

I think I started scraping symbols from Fedora 34 builds on Monday, we probably haven't looked at those crashes yet.

We do gather symbol information on Windows too, but it's a lot less detailed than what we get from Linux. It also requires jumping through a lot of hoops. For example we get minimal information from Windows graphics drivers and we have to semi-manually scrape them ourselves.

It's rasdaemon these days: https://www.setphaserstostun.org/posts/monitoring-ecc-memory...

And also a VP

A fairly large number of managers have been laid off too.