Hacker News Reader

Comment by georgyo | Hacker News Reader

I am no fan of the cloud, but with that many disks, failure rate goes up. So you also need someone to go to the DC. All the disks will bought around the same time, which means failures start to cluster up as well.

The machine also needs to be replaced every $interval. After 4 or 5 years, keep it up is more hassle that replacing it.

A single machine patches and will need to reboot. What should happen to people trying to access the store?

Keep replicas help, ensuring a petabyte or even half a petabyte of small files is in sync across multiple sites with various link speeds is not as trivial as rsync.

View on HN

Most datacenters offer remote hands, so there is no need to travel to the DC for those kinds of things (just ship them the drives and have them replace it). One can also easily add extra spare hard drives to the pool that ZFS or mdraid can automatically put into production in case of drive failure, so there is no need for active intervention after each drive failure.

And if any downtime for upgrades is unacceptable one can go for distributed storage solutions like ceph, seaweedfs or minio. Or simply 1:1 mirror everything to a second machine (which you want for disaster recovery anyway) and switch over during the maintenance period. And since for this specific usecase the server(s) would only serve as CDN origin, a brief downtime might even have very little actual impact.

If the NixOS team cannot handle this then I question their capabilities. This is cake work.

Handle yes, but is this “cheaper”

As is, Nix would have to spend $108k/year to continue to host from S3. For comparison, OpenStreetMap's entire budget for infra/ops is 169,197/year (FY2023).

Of the cloud storages, AWS S3 is the most expensive for both storage and bandwidth.

A half petabyte at other places such as wasabi would cost 35k a year, including bandwidth.

Open street maps by comparison in 196GB compressed including full history. 20 years of full snapshots would amount to only .2PB, or 20k a month to host on cheaper storage providers.

I don't think using OSM's budget was the best example.

Reply on news.ycombinator.com