Nix/NixOS S3 Update and Recap of Community Call

What percentage of the total repo is source code? I would think the compiled artifacts could be tossed at a moment’s notice, but would not want to lose the source.

S3 is definitely not in a great spot on the cost effectiveness axis, but it wins on a lot of other axes

> The cheapest tier, Glacier Deep Archive, is $0.00099/GB.

Bad example. "Latency", if it can even be called that, is 12-48 hours.

It's not cloud storage, it's archiving.

Yeah, $0.00099/GB (roughly $1/TB per month) sounds cheap, but retrieving that data will cost you almost two orders of magnitude more than the storage, so not really such a good deal. Also, obviously the latency of the "glacial" tier is so awful that it is useless for anything other than archival of long-term data without any need for speedy restoration.

Glacier Deep Archive can take hours to access abucket, likely because the right tape should be shipped from a warehouse.

Nix packages are blindly and automatically signed by a single private key last I reviewed it. Bribe or compromise one person and the entire model fails. There is good reason to do this too given that many major blockchain projects use it for builds. Hundreds of millions to be made by one that impersonates the right maintainer before the right software release.

Also no signed commits or signed authorship means someone with Github access can just fake history and inject whatever they want after code reviews are completed, which will then be blindly and automatically signed.

Some of the people with write access to the nixpgs repo even have SMS recovery enabled on their github recovery email accounts. One sim swap to compromise all nix users. I will not call them out, but go try to do a email password reset on recent committers for yourself. A malicious github employee could also of course do whatever they want to an unsigned repo. Or a well placed BGP attack. Lots of options. It is hard to prevent such things, but author commit signing would mitigate the risk and can be enforced.

I made my case for this to the nix team but in the end it was concluded people would stop maintaining packages if they had to do the bare minimum like commit signing or hardware 2FA. https://github.com/NixOS/rfcs/pull/34

All this is fine, but it means effectively a decision was made for NixOS to be a hobby distro not suitable for any targeted applications or individuals. It really sucks, because I love everything else about nix design.

Instead I am forced to bootstrap high security applications using arch and debian toolchains which are worse than nix in every way but supply chain integrity given that all authors directly sign package sources with their personal well verified keys. They have a ton of other security and even their own supply chain problems but they at least can survive phishing, a malicious mirror, or a sim swap. It is a low bar nix sadly does not meet.

Trustix and CAN are -fantastic- for the problems they are intended to solve, but they only cover the consumption side of the supply chain. You can use these to ensure the binaries are built from the published code, or that you are using the right published code. You have no idea if the published git commits were not made by a sim swap compromised github account... because maintainers are not required to do the bare minimum such as signing commits.

Compare to arch, fedora, debian, and basically every other linux distro that has existed more than a decade. Every maintainer signs their own contributions with well known keys so they cannot be impersonated and so later stages of the supply chain cannot tamper with them.

Newer distros like Nix and Alpine decided do get rid of all that security overhead in order to attract a huge pile of randos as maintainers. I mean, it worked, but at a very high price.

How can Hydra know the nixpkgs repo was not tampered with? Maintainers impersonated?

How can anyone know the Hydra signing key was not tampered with?

These are problems other linux distros have solved for decades by just requiring maintainers press a blinking yubikey or similar to sign their contributions.

Signed by a single centralized key on an internet connected stack that how many people have control of?

Errr... What do you mean by "easily affordable"? A quick look on Amazon says that I'd be paying something like €18.80/TB, so buying 400TB would be €7,520 - far out of the range of the majority of hobbyists. Plus a 100GBE NIC would be another €800 or so, plus you'll need all the other ancillary things like a case and a processor and RAM and cabling and whatever the monthly cost to host it somewhere, unless your ISP gives you a 100Gbps connection (which I'm sure will be extortionately expensive if they do). End result is the hardware costs alone approaching €10,000 - significantly more if you want any sort of redundancy or backups (or are you planning to run the whole thing in RAID0 YOLO mode?), plus probably low-hundreds per month, not including the time investment needed to maintain it.

Depending on how bad & frequent you want outages to be and whether you want the whole thing to go down for scheduled maintenance, it can get quite a bit more complicated. A pair of servers with some level of RAID is probably the minimum to achieve any amount of availability better than a server in someone's closet. Then you need a cold backup, on top of that, no matter how many live servers you have.

I've also had much better luck with connectivity, transfer speeds, and latency, for clients in some regions, with AWS than with some other hosts (e.g. Digital Ocean). It seems peering agreement quality really matters. Not sure how the big physical server hosts fare on that front vs. AWS (which is basically the gold standard, as far as I can tell) and it's hard to find such info without trying and seeing what happens. This matters if you're trying to serve a large audience over much of the Earth. The easy single-server solution might well see some clients lost 95% of their transfer speed from your service, or not be able to connect at all, compared with AWS.

The old binaries are not necessarily simple to rebuild. Projects move / go offline. Tags get moved. Binaries for the impure systems may not be public anymore. You can't easily rebuild everything from that cache.

S3 stores 3 separate copies across 3 AZs so for that redundancy (at 400TiB), wouldn't you need more like 6x 14x16TB servers if you're making a direct comparison?

there is human cost associated with operating this that s3 and other fully managed object stores don’t have.

there is just no reality where you build your own service for this with better reliability, less management, and less maintenance than s3 (and when i say s3 i’m including all managed object stores).

pointing at hetzner and saying, “look how cheap this is”, is missing the point.

Sure, but someone still has to store it. If the foundation wants you keep the old builds, they still need to pay for that 0.5PB somewhere.

My org does the live-store-on-zfs model with a thin proxy cache on the front end to take the load off the CPU constantly regenerating the top-5% most popular NARs.

It seems to work well for us, but I am curious how that would look at 425TB scale— would filesystem deduplication pay for the losses in compression, and if so, by how much? How does that change if you start sharding the store across multiple ZFS hosts?