Comment by nyanpasu64 | Hacker News Reader

Comment by nyanpasu64 | Hacker News Reader

Parent Full thread

Running Firefox on Windows 10 x64, on a somewhat under-specced machine that tends to run low on RAM.

I'm getting highly inconsistent note timings, with intermittent full 1-second pauses with no new notes being triggered. (FWIW, I also saw something similar in https://news.ycombinator.com/item?id=26870666 , and am getting choppy audio in Zoom that disappears as soon as I enable profiling...)

Is it my browser's fault for having the JS event loop stall, or the page's fault for driving audio off the page's event loop? Or both? (IDK how to fix Firefox, other than increasing the content processes above 3, which will reduce cross-tab lag but possibly increase RAM, or to reduce my extensions or find the one at fault.)

On a related note, how critical is it to follow the "audio on a real-time thread" dogma (http://www.rossbencina.com/code/real-time-audio-programming-...)? I've heard that not using Electron, writing real-time code, avoiding mutexes and allocations, mlock to avoid swapping, tuning OSes and picking processors and audio interfaces based on its low-latency properties, etc. is important for live performances, but not necessary for MIDI instrument input for (casual home users? studio work?).

I can reproduce the audio stutter by having Firefox do anything performance intensive (starting recording a performance profile does the trick nicely) and I seem to be getting severe frame drops because of garbage collection.

I think the flows of React and Redux are just not built for timing-sensitive applications like these. A lot of allocations and deallocations are taking place without any user input. Looking at the source code, the many selectors and other hooks probably has something to do with that.

Redux is also very explicit about treating objects as immutable, causing a lot of duplication as a result. For a simple web application this shouldn't be too much of a problem but again, this application is very sensitive to timing issues where 50ms of GC can cause a noticeable performance degradation.

It's not that the code is bad, it seems quite well memo-ised to get decent performance; I think it's a result of using the wrong too for the job, nothing more. Quite impressive that a framework as heavy as React can be made this responsive in my opinion.

If you run into any memory troubles, check about:performance and about:memory, you may be able to find what's causing the memory leaks.

In the past I've found about:memory to be almost useless for pinpointing leaks. It's segmented into processes and types of allocations, and only the deeply nested sub-entries say which tab the memory belongs to. And even then, I don't know whether Firefox, the site, or an extension is causing memory bloat. I end up resorting to just blindly closing tabs or restarting the browser.

I might try about:performance more, but in the past I recall it being similarly almost useless for pinpointing sites eating CPU.

How many other open tabs and applications? Without a doubt I don't expect excessive overhead from the browser and would encourage users experiencing serious performance issues to wipe out everything else going on. This is still an experiment in just how much we can do in the browser.

It would be a fun challenge for me to programmatically figure out if the user is experiencing issues and disabled features to improve performance...

I work on the Web Audio API at Mozilla, here is an article that I wrote that explains what goes wrong here, distilled to the core: implementing a metronome in literate programming in a way that will _never_ glitch: https://blog.paul.cx/post/metronome/. The result is nowhere as interesting as this app, but this is to get the point across.

https://www.html5rocks.com/en/tutorials/audio/scheduling/ is an article written by my former fellow co-editor of the Web Audio API specification, that talks in length about this.

In 2021, with the AudioWorklet [0] being available in all major browsers (including Safari as of a few weeks ago!), arbitrary computations can happen on the real-time audio thread. It is now possible to send a copy of the sequencer data (which note when, what sample, what velocity, etc.) to the real-time thread, and let it play, unaffected by the load of the machine, be it from the browser or other programs (until everything breaks down because you're also overloading the audio thread).

I think this app is very good and does the best it can under the programming model it's using, considering the environment it's being run in, but if more robustness is needed, there is now a way !

On your last paragraph, it is still _absolutely_ critical. An audio software that doesn't use a real-time thread for rendering cannot render audio without glitches, even under no load, even on a fast machine. You don't need to tune the OS [1] and pick special hardware these days: today's computers (and even computers from 10 years ago or older) are perfectly capable or running reasonably complex audio processing workloads _if the software is written correctly_. This is regardless of the use case. When we disabled the real-time thread on Windows by mistake in Firefox (adjusting priority of the process level, and then that was inherited to all threads, very quickly reverted), we had reports in a matter of _days_ (if not the next day), and the Firefox Nightly population is not that big. This was I think playing regular videos on social media, not even anything complicated.

If the author of the app, or anybody, wants to profile the app with the right tools, I've written a blog post [2] about this as well. I hear this tooling is used by audio professionals, now, and that they're happy with it. Happy to hear about any suggestion though :-).

For your Firefox-specific question, you're right that increasing the content process count will make everything better, and we're constantly improving memory usage so that it's possible to do so even on lower-specs machines, maybe it's worth a try. Please look at the "resident memory" values, and not, say, virtual or shared, because we make heavy use of shared memory and copy on write facilities of the OS to limit the memory usage when using multiple process (which we need to do for origin isolation and avoiding Spectre issues and sandboxing various bits of the browser as hard as we can, for example).

[0]: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle...

[1]: (but you can to improve robustness further, and it helps to have good drivers, such as Jack on Linux, WASAPI with MMCSS on Windows or ASIO, or just the normal stuff on macOS)

[2]: https://blog.paul.cx/post/profiling-firefox-real-time-media-...

Thanks for the post and your blog! Added to my RSS reader.

On the topic of MMCSS, RtAudio is another real-time audio API... except the WASAPI backend had broken real-time thread scheduling (AvSetMmThreadCharacteristicsW, I think this is MMCSS?) on any builds with UNICODE enabled... and somehow literally nobody noticed, until I analyzed RtAudio under Process Monitor and wondered why it was trying to open a random Chinese string as a DLL file. I ended up fixing it at https://github.com/thestk/rtaudio/pull/292/files .

I don't know what that means. Is MMCSS not that important (even though you mentioned missing MMCSS caused severe stuttering in Firefox)? Is RtAudio under-tested/amateurish to the point people didn't notice it was stuttering more than it should be?

I assume it's unacceptable to synthesize (or probably time) audio off the main thread. How bad is it to use mutexes and allocations on the audio thread?

I'm not sure how many Windows users RtAudio has, and the kind of machines people are using with it, and their expectations, but in Firefox, the answer is hundreds of millions and "any kind of machine that has existed", such as single or dual-core machines that are running dozens of modern web apps in parallel, so load is high. That might well explain why it was reported so quickly. It's quite noticeable with a standard load testing procedures (e.g. "how many parallel oscillators can this computer/OS/driver do at this latency and be glitch-less under load").

It is acceptable to do audio work off the main thread (as long as buffering is in place to prevent glitches from scheduling delays, and it's prioritized appropriately), but maybe you mean "on the main thread". In this case, it's not really recommended, but sometimes, it's what you need to do, for example when emulating an old console where everything is synchronous in the render loop.

Locks and allocations are best avoided, but modern locks and allocators are amazing. If the lock is _not contended_, it's very fast, and it's fine. If the allocation comes from a thread-local pool (some allocators do that), it's also very fast and fine, but that needs to be true across OSes, etc. for us. In other case, which is most of the time, it's best to avoid those.

We're not allocation nor lock-free entirely in Firefox though, but it's been thoroughly profiled - when know what we'll optimize when it's time to do so, but for now it's fine under extremely heavy load in a normal situation (we have locks when the audio devices are unplugged for an example of a "non-normal" situation). This is largely due to legacy (the Web Audio API, for example, is about 10 years old).

I experimented 5-6 years ago with a programmable metronome using KnockoutJS and realized quickly how easy it is to cause jitters and unpredictable performance on the main thread. I wish I had come across this post back then!

Although a big contributor for performance issues in my project (and especially OP's) is the amount of UI updates that need to happen in succession (lots of event listeners, css transformations and dom changes)

Reply on news.ycombinator.com