Hacker News Reader

Comment by csdreamer7 | Hacker News Reader

This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days?

https://developers.redhat.com/blog/2021/01/05/building-red-h...

Think of how much faster their servers would be with one of those Epyc consumer cpus.

I was about to ask people to donate, but they have $80k in their coffers. I realize their budget is only $17,000 a year, but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers as they are around under $2k under budget. If they have a fleet of these old servers I imagine a Zen5 one can replace at least a few of them and consume far less power and space.

https://opencollective.com/f-droid#category-BUDGET

Not sure if this includes their Librapay donations either:

https://liberapay.com/F-Droid-Data/donate

View on HN

> This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days?

This is not always a given. In our virtualization platform, we have upgraded a vendor supplied VM recently, and while it booted, some of the services on it failed to start despite exposing a x86_64v2 + AES CPU to the said VM. Minimum requirements cited "Pentium and Celeron", so it was more than enough.

It turned out that one of the services used a single instruction added in a v3 or v4 CPU, and failed to start. We changed the exposed CPU and things have returned to normal.

So, their servers might be capable and misconfigured, or the binary might require more that what it states, or something else.

A developer on the ticket writes: "Our machines run older server grade CPUs, that indeed do not support the newer SSE4_1 and SSSE3"

Ooh. They are at least ~15 years old, then. Maybe they have scored on some old, 4 socket Dell R815s. 48 cores ain't that bad for a build server.

It's kinda good they use such old systems, as the vast majority of pollution occurs during manufacturing of devices since we usually use them only a handful of years. Iirc the break-even point was somewhere around 25 years, as in, upgrading for energy efficiency then becomes worth it (source: https://wimvanderbauwhede.codeberg.page/articles/frugal-comp...). 15 goes a long way towards that!

On the other hand, I didn't dig very deep into the ticket history now but it sounds like this could have been expected: it broke once already 4 years ago (2021), so maybe planning an upgrade for when this happens again would be good foresight. Then again, volunteers... It's not like I picked up the work as an f-droid user either

While I appreciate the sentiment, I think you may be misreading the "Emissions from production of computational resources" section of that link.

It says for servers that 13-21 years is the break even for emissions from production vs consumption.

The 25 year number is for consumer devices like phones and laptops.

I would also argue that average load on the servers comes into play.

Moot point imo, no one says they have to buy new hardware. Used, affordable, but still much more modern hardware could still save them plenty on power usage and replace several systems with one.

> about to ask people to donate, but they have $80k in their coffers

I'd still ask folks to donate. £80k isn't much at all given the time and effort I've seen their volunteers spend on keeping the lights on.

From what I recall, they do want to modernize their build infrastructure, but it is as big as an investment they can make. If they had enough in their "coffers", I'm sure they'd feel more confident about it.

It isn't like they don't have any other things to fix or address.

I would too but do you have a link to them talking about it?

They publish weekly reports now on the work they're doing: https://f-droid.org/en/news

For example, here's a recent NLNet sponsorship that helped reproducible builds ship (an effort that began in 2023): https://f-droid.org/en/2025/05/21/making-reproducible-builds...

Their 2025 year in review: https://f-droid.org/en/2025/01/21/a-look-back-at-2024-f-droi...

$2-3k ? That’s barely the price of a lower end Threadripper bare cpu not a full Epyc server ???

At our supplier $2k would pay for a 1U server with a 16 core 3GHz Epyc 7313P with 32GB RAM, a tiny SSD and non-redundant power.

$3k pays for a 1U server with a 32 core 2.6GHz Epyc 7513 with 128GB RAM and 960GB of non-redundant SSD storage (probably fine for build servers).

All using server CPUs, since that was easier to find. If you want more cores or more than 3GHz things get considerably more expensive.

Yes but thoose are Zen 3 Milan cpu released in 2021 I believe.

Not that they are bad and would not be way better than what they have, just that I though the parent was quite the optimist with his Zen4/Zen5 pricing.

OP did say "consumer Epyc", so presumably referring to the parts using the AM5 socket. From a quick check on Newegg, it looks like barebones servers for that platform start at under $1000, to which you need to add CPU, RAM, and storage. So a $3000 budget to assemble a low-end Zen4/5 EPYC server is realistic: $570 for the 16-core EPYC 4565P, a few hundred for DDR5 ECC unbuffered modules, a few hundred for an enterprise SSD, and you have a credible current-gen server from readily available parts at retail prices, without any of the enterprise pricing and procurement hassle.

That was my intention; mATX AM5 parts.

I imagine they would need quite a few servers to replace their current setup.

Then there's also the overhead of setting up and maintaining the hardware in their location. It's not just a "solve this problem for ~$2,000 and be done with it".

I don't know the actual specs or requirements. Maybe 1 build server is sufficient, but from what I know there's nearly 4,000 apps on FDroid. 1 server might be swamped handling that much overhead in a timely manner.

One server with today's tech can easily replace several servers that are 12+ years old. 4000 apps doesn't sound like a lot of work for one machine, unless you assume almost all of them are releasing new builds more than once a week. A 16-core CPU can rebuild a full Gentoo desktop OS multiple times a week.

Is that $2k/$3k for the year?

That's $2k/3k to get a box with fully assembled hardware delivered to your doorstep or to a DC of your choice.

Space in your basement or the colo rack of a datacenter along with power, data and cooling is an expense on top. But whatever old servers they have are going to take up more space and use more power and cooling. Upgrading servers that are 5+ years old frequently pays for itself because of the reduced operating costs (unless you opt for more processing power at equal operating cost instead)

Low end EPYC (16-24 cores) especially for older generations are not that expensive 800-1.2K ime. Less when in a second hand server.

Perhaps the servers run Coreboot / Libreboot?

I'm not even sure mainline Linux supports machines this old at this point. The cmpxchg16b instruction isn't that old, and I believe it's required now.

32 bit Linux is still supported by the kernel and Debian, Arch, and Fedora still supports baseline x86_64.

RHEL 8 is still supported and Ubuntu is still baseline x86_64 I believe for commercial distros. Not sure about SuSE.

> 32 bit Linux is still supported by the kernel and Debian

Deprecated for Debian

https://www.debian.org/releases/stable/release-notes/issues....

>> 32 bit Linux is still supported by the kernel and Debian

> Deprecated for Debian

> https://www.debian.org/releases/stable/release-notes/issues....

32 bit Linux is still supported by the kernel... and... 'Debian, Arch, and Fedora still supports baseline x86_64'.

Please do not take things out of context.

CMPXCHG8B is required as of a month or two ago, not 16B (i.e., the version from the 90's is now required)

See https://lkml.org/lkml/2025/4/25/409

>they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers

I would also like to know this.

Probably a case of "don't fix it if it ain't broke" keeping old machines in service too long, so now they broke.

That's like ignoring your 'Check Engine' light because the engine still runs.

I would much rather they spent that on having the devs network and travel, the servers work.

Why are the builds failing then?

planned obsolescence by Google

Beginning to use a CPU opcode that is 19 years old doesn't feel like planned obsolescence. if anything, it feels like unplanned obsolescence... "Oh hell what do you mean your CPU doesn't have that opcode no we've just been running the compiler with the default flags and that opcode got added to the default two months ago after a 10-year fight about the possible consequences of changing defaults!"

Although I'm a little surprised to learn that the binary itself doesn't have enough information in its header to be able to declare that it needs SSSE3 to be executed; that feels like something that should be statically-analyzed-and-cached to avoid a lot of debugging headaches.

> "Oh hell what do you mean your CPU doesn't have that opcode [...]"

hobbyst dev? sure

Google? nope

Did they make any explicit guarantees that their newly-cut binaries would continue to support 20-year-old architectures?

Googlers aren't gods. It's a 100,000-person company; they're as vulnerable to "We didn't really think of that one way or the other" as anyone else.

ETA: It's actually not even Google code that changed (directly); Gradle apparently began requiring SSSE3 (https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153) and Google's toolchain just consumed the new constraint from its upstream.

Here, I'm not surprised at all; Google is not the kind of firm that keeps a test-lab of older hardware for every application they ship, so (particularly for their dev tooling) "It worked on my machine" is probably ship-worthy. I bet they don't even have an explicit architecture target for the Android build toolchain beyond the company's default (which is generally "The two most recent versions" of whatever we're talking about).

They clearly don't

Yeah and everybody was complaining how slow the builds are for years. I really want to know too

Reply on news.ycombinator.com