Comment by fluffything | Hacker News Reader

Comment by fluffything | Hacker News Reader

Parent Full thread

Firefox ships ~100 binaries for 100 different targets, from mobile phones, to desktops, tablets, ... x86, arm, sparc, ppc, riscv, ... X linux, windows, macos, freebds, openbsd, netbsd, .... X arch linux, debian, ubuntu ..

At some point you need to say this is what we want to ship, and branch, and make sure that not only all tests pass, but that the installers work, run the benchmark suite that might not be run on every PR, make sure the updates work on all systems, appstores, etc. Accept new patches to fix bugs, etc.

I suppose they might have a release team that does nothing but releases, and if creating a full release takes a week (no idea how long it takes honestly), then maybe they could release each week.

But I don't think they can release a new version every day, because probably running their whole CI takes longer.

E.g. before shipping a version of Rust, the RC is used to build and run tests of all packages in the Rust package repository. This takes 4 full days. One could probably throw money at the problem and speed it up, but at least right now, releases just cannot technically happen faster, unless you are willing to compromise on quality.

Right. Desktop software is also much more difficult to QA than server software because of the lack of control over the environment and the difficulty of rollback.

Also, for desktop software, I have found that small releases are a lot less risky.

You experience small issues, one by one, instead of a massive amount of bugs coming all of a sudden.

For example, last time I updated a Linux distribution with 6-month releases this was the case. Whereas with a rolling release one, it's just much more pleasant.

This is true of releasing code in general. Changes that accumulate over time actually increase risk whether it be desktop or server-side.

Not to mention, different builds per language for each OS. [1] This has never really made sense to me, but I'm sure there must be a good reason.

[1] - https://ftp.mozilla.org/pub/firefox/releases/68.1.0esr/linux...

My educated guess is that the localization data for every language is pretty massive, so per-region downloads can include the text for their region without dragging along a few extra megabytes of text/images for other regions. If you swap regions after installing it can probably download on-demand.

Historically reducing your download size by a couple megabytes has had measurable positive impact on conversion rates (from webpage visit -> install) so I expect this tradeoff made a lot of sense in the past even if it's less important now.

EDIT: One other thing worth note is that there's zh-CN and zh-TW, and I can imagine the Chinese government being very irritated about the idea of their release having separate Taiwan loc text/images baked in since according to them Taiwan doesn't exist. Major software like Windows has already had CN-specific releases for this (among other) reason(s).

zh-CN and zh-TW has nothing to do with that, and I'm virtually certain there are no "Taiwan images". in applications which provide zh-CN and zh-TW, including Firefox, the former means Simplified Chinese and the latter means Traditional Chinese. theoretically, it could mean Taiwanese Chinese, but then there would also be zh-HK, zh-MO, zh-SG...

`zh-CN` and `zh-TW` are generally just old-school ways to say "simplified Chinese" and "Traditional Chinese" (two similar writing systems). The confusion comes in as the Chinese language family has multiple dialects and variants, and there are multiple writing systems.

The current practice is really a dimension-reduction, where a N×2 matrix [(CN, TW, HK, SG, MY...) × (simplified, traditional)] collapsed into a 2×1 vector (TW, CN).

The language labels should have been `zh-hans` and `zh-hant` if one means to differentiate the writing systems, and not the underlying linguistic variants.

According to Beijing government, Taiwan absolutely exists, it's just that it is not a sovereign state. Hong Kong has been handed over to China for more than twenty years and people in Hong Kong continue to use zh-HK – Beijing government seems to be OK with that.

They build downloadable binaries every 12 hours, called Firefox nightly, which is to say the bottleneck for full releases isn't building or automated testing. Mind you, the nightlies are experimental, and not for general use.

Building is in general just a small part of the release pipeline.

Mozilla only provides binaries for Linux 32-bit, Linux 64-bit, macOS, win32, and win64:

http://ftp.mozilla.org/pub/firefox/releases/69.0/

All of the various Linux distributions just repackage those binaries. iOS and Android are distributed only through the App Store and Google Play.

The BSDs and other ("tier 3") platforms have to compile from source and distribute their own packages:

https://developer.mozilla.org/en-US/docs/Mozilla/Supported_b...

> All of the various Linux distributions just repackage those binaries

Debian and Ubuntu don’t build from sources?

They do.

You can notice by looking at the build time in the version string.

This makes it impossible to share a profile between two GNU/Linux distributions (or even the official binary and the distribution binary) without hacking with the current versions, even if they are effectively the same version: one will complain that the profile has been run with a more recent version.

(But I sure expect and want my distributions to rebuild Firefox - I trust Debian for making sure no bit of proprietary code is here, and makes tracking opt-in, more than Mozilla)

Most linuces build their own firefoxes from source, same as the BSDs.

it mostly was the case to remove the logo at some point in time :/

> All of the various Linux distributions just repackage those binaries.

Source please.

Never mind, it's simply wrong so don't even worry about it!

Even if you’re right and they’re wrong, the snark was unwarranted IMO. You should either correct the GP or request more information (like a source), not ask rhetorically.

Gentoo does it for firefox-bin:

/usr/portage/www-client/firefox-bin/firefox-bin-69.0.ebuild

    MOZ_HTTP_URI="https://archive.mozilla.org/pub/mozilla.org/${MOZ_PN}/releases/"
    ...
    SRC_URI="${SRC_URI}
        amd64? ( ${MOZ_HTTP_URI%/}/${MOZ_PV}/linux-x86_64/en-US/${MOZ_P}.tar.bz2 -> ${PN}_x86_64-${PV}.tar.bz2 )
        x86? ( ${MOZ_HTTP_URI%/}/${MOZ_PV}/linux-i686/en-US/${MOZ_P}.tar.bz2 -> ${PN}_i686-${PV}.tar.bz2 )"

Doesn’t gentoo essentially do that for (most?) everything?

Only massive packages (firefox, chromium, KDE, Gnome, etc.). The entire promise of gentoo is the opposite: fine-tuning and local ~~heater~~ compilation.

Although this comment was already debunked, I'll add Arch's Firefox PKGBUILD here [1].

And to save you a click, they compile it from a source tarball.

[1] https://git.archlinux.org/svntogit/packages.git/tree/trunk/P...

They even need the Google API key...

https://git.archlinux.org/svntogit/packages.git/tree/trunk/P...

They must already be doing a full build, test, and benchmark for each platform for each PR to their codebase in CI, presumably in parallel to make it run reasonably fast. It doesn’t seem like much of an extra step to release those binaries they’re already building hundreds of times a day.

They already do that with Firefox Nightly. But most users expect a more stable product than that.

How can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?

It can be more stable by having nightly users test it beforehand for a while and report any new bugs.

What are those users doing that can’t be automated and run in CI?

Test on exotic hardware, for example. Or exotic combinations of libraries in the OS which FF depends on.

Those should be in the CI test matrix then - we shouldn’t be relying on manual testing done specially for releases.

If even Microsoft can't catch these, why should Mozilla be able to? I am admittedly not in the CI business for frontends but I'd imagine that the combinatorics vastly outnumbers the possibilities of any organization.

I'm guessing something the devs haven't thought of. As soon as you hit 10k+ users, chances are people are doing things you didn't expect.

It'd be pretty ambitious to have CI run against the variety of web apps that real users "test" against (including ones they're developing!)

To begin with, it is probably released with a subset of the CI matrix, because I imagine that the full CI matrix takes more than a day. Second, it is nearly impossible* to cover all possible environments+addons+websites and test them correctly. This isn't a normal desktop app, nor a webapp that always is running relatively sandboxed in a more or less known environment. Browsers are platforms, as OS are. Lots of people doing wacky stuff on it. Very hard to realistically automate all that stuff.

*impossible with a limited constraint of time and/or money.

We do exactly this for people who want. A reasonably stable build (including full testing, and benchmarking, which takes multiple hundreds hours, parallelized of course) is available twice a day (0:00 and 12:00 UTC). It self updates (either differential or full download depending of what landed on the past day).

I've been using only this for the last 10 years or so (on windows, Linux, Mac and Android), and I can count the number of time things really broke on one hand.

We call these Firefox Nighly, and they are available at https://www.mozilla.org/en-US/firefox/nightly/all/.

Release notes at available at https://www.mozilla.org/en-US/firefox/71.0a1/releasenotes/, but they are not always updated for each version, sometimes batched (its quite some work due to the amount of commits that are pushed per day).

There is a cool blog at https://blog.nightly.mozilla.org/ that talks a bit about the new stuff.

All this is really experimental and probably not for everybody, but helps us to deliver something good at the end. Filing bugs and giving us feedback is of course particularly appreciated, as always!

Are nightlies tested on CI for all platforms that Firefox supports ? I only see Tier-1 in that matrix (Linux, Windows, Mac). No iOS, Android, *BSDs, etc.

Reply on news.ycombinator.com