Hacker News Reader

Comment by chrisseaton | Hacker News Reader

They must already be doing a full build, test, and benchmark for each platform for each PR to their codebase in CI, presumably in parallel to make it run reasonably fast. It doesn’t seem like much of an extra step to release those binaries they’re already building hundreds of times a day.

View on HN

We do exactly this for people who want. A reasonably stable build (including full testing, and benchmarking, which takes multiple hundreds hours, parallelized of course) is available twice a day (0:00 and 12:00 UTC). It self updates (either differential or full download depending of what landed on the past day).

I've been using only this for the last 10 years or so (on windows, Linux, Mac and Android), and I can count the number of time things really broke on one hand.

We call these Firefox Nighly, and they are available at https://www.mozilla.org/en-US/firefox/nightly/all/.

Release notes at available at https://www.mozilla.org/en-US/firefox/71.0a1/releasenotes/, but they are not always updated for each version, sometimes batched (its quite some work due to the amount of commits that are pushed per day).

There is a cool blog at https://blog.nightly.mozilla.org/ that talks a bit about the new stuff.

All this is really experimental and probably not for everybody, but helps us to deliver something good at the end. Filing bugs and giving us feedback is of course particularly appreciated, as always!

Are nightlies tested on CI for all platforms that Firefox supports ? I only see Tier-1 in that matrix (Linux, Windows, Mac). No iOS, Android, *BSDs, etc.

They already do that with Firefox Nightly. But most users expect a more stable product than that.

How can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?

It can be more stable by having nightly users test it beforehand for a while and report any new bugs.

What are those users doing that can’t be automated and run in CI?

Test on exotic hardware, for example. Or exotic combinations of libraries in the OS which FF depends on.

Those should be in the CI test matrix then - we shouldn’t be relying on manual testing done specially for releases.

If even Microsoft can't catch these, why should Mozilla be able to? I am admittedly not in the CI business for frontends but I'd imagine that the combinatorics vastly outnumbers the possibilities of any organization.

I'm guessing something the devs haven't thought of. As soon as you hit 10k+ users, chances are people are doing things you didn't expect.

It'd be pretty ambitious to have CI run against the variety of web apps that real users "test" against (including ones they're developing!)

To begin with, it is probably released with a subset of the CI matrix, because I imagine that the full CI matrix takes more than a day. Second, it is nearly impossible* to cover all possible environments+addons+websites and test them correctly. This isn't a normal desktop app, nor a webapp that always is running relatively sandboxed in a more or less known environment. Browsers are platforms, as OS are. Lots of people doing wacky stuff on it. Very hard to realistically automate all that stuff.

*impossible with a limited constraint of time and/or money.

Reply on news.ycombinator.com