Hacker News Reader

Comment by hyperman1 | Hacker News Reader

Yesterday, while discussing backups, someone said end users should basically trust the cloud. This is why it is a bad idea.

The risk is data loss, irrespective if it is because of hardware failure or because of cloud failure connected to overzealous legal enforcement, algorithm decision making failure,or service deprecation.

Part of the risk equation is how big the chance is, and I would really love to see the numbers of hardware faulure vs cloud failure.

Another part is impact. Hardware loss might be recoverable via specific services. Cloud failure, basically you're on your own, and they might revoke access to your email a.k.a. digital identity certificate, or even sick the copyright cartel on you, worst case ending in a police visit taking out the other backups.

View on HN

Hardware loss is easily avoidable with more hardware... create backups, store them on multiple external drives (for most users, a single modern external drive is enough for all the backups, so the additional ones are for copies), and you can then keep one at home, one at work, one at your parents place, and replace them every now and then to 'refresh' the backups... Most home users don't need hourly or daily backups, they just nee their photo and home video collections saved, their documents scanned etc.

With cloud providers, storing stuff in different datacenters doesn't help you, if you get banned from google, because someone didn't like your youtube comment. Multiple cloud providers might be using the same amazon/azure/... datacenter in the backend. Then there are different L0-L8 problems, from political, where your country gets embargoed due to some ongoing political struggle, or you can move to another location, where the internet is slow. And now, it seems that you can lose data due to stupidity as in the original post.

> easily avoidable

> and you can then keep one at home, one at work, one at your parents place, and replace them every now and then to 'refresh' the backups

Yes, many of us do this. But I wouldn't consider it "easy", esp for the average user.

What % of home computer users own 2+ external drives? Or even 1? I bet it's under 20%.

I tell all coachees if they are on AWS, also put backups on GCP/Rsync.net/Herzner/... and vice versa. If your data and backups are with one provider it's a non zero risk of losing everything because of account removal.

This is sound advice but it's such a pain in neck to implement. Cloud providers have every incentive to make it difficult for you to duplicate your actions to a competitor, and to make it difficult for any third party who wants to provide such duplication as a convenience.

As long as it is plain files, rclone takes a few minutes to set up, and removes those difficulties.

Especially if it's encrypted, e.g. restic/bupstash/duplicity+pgp/tarsnap/perkeep/borg/kopia/... Good luck finding the hash of a nipple, copyright-claimed material, or other TOS violations in that data.

I haven't had a hardware failure in so long, it's scary. I keep wondering when my next failure is going to be but my countless laptops, hard drives, NAS, and desktop systems keep chugging along without issue.

I know of a Netgear consumer NAS I installed in 2011 is still running a business, untouched. It uses two enterprise drives in a mirror config.

And this is why I put together a NAS for my home, which then backs up to a cloud provider, encrypted - and if that provider decides to play silly shenanigans I can just as easily colo another box and back up to that as well as any other cloud provider.

ZFS on Linux is stupidly easy to setup, something like URBackup for all your clients to backup onto the NAS and you've got a nice 1-2-3 (with 3 being offsite) backup scheme.

> ZFS on Linux is stupidly easy to setup

For a user familiar with Linux. Even many macOS and Windows based developers would need some time and research to set something like that up, let alone a non technical person.

If this is a problem…

Just buy a ready to use box from synology, qnap, western digital, etc… or just use a usb drive.

With pre-made boxes, you're viable to run into issues when they graciously decide to force a security update on you. QNAP makes extremely decent boxes but last year's behavior that broke a lot of stuff [1] has all but killed my trust in them.

[1]: https://www.zdnet.com/article/decryptor-released-for-deadbol...

TBH, synology and qnap have awful track records when it comes to data integrity and cause a ton of issues even when using the software the "intended" way. SMB is a chatty protocol in general which makes it hard to troubleshoot, and because of default lack of write-through, you have a non-trivial chance of a situation where the SMB stack returns "this write was a-okay!" but in fact the data never landed on disk and only hit the cache.

My team works with a lot of clients for backups and we've had so many silent corruption cases with lousy synology and qnap boxes that we just don't support them when using SMB or NFS anymore; the built-in stack is just too unreliable. iSCSI is a bit better, but iSCSI isn't how a lot of people want their NAS to work, so it's always a tough discussion. General purpose servers with proper endpoints have always been better, but it's a hard sell to clients.

Directly-attached USB is marginally better in that you don't deal with network protocols, but a lot of companies cheap out on USB controllers and it's a challenge sometimes to convince clients that the on-board controller is the issue when single (small) file writes work without issue.

But long story short, I'd propose a small general purpose server (even Raspberry Pi!) over a QNAP/Synology any day of the week. The latter does everything simply but with excess mediocrity. From my point of view, they hit MVP across a dozen + items, but competency in none.

People relying on reliable backups building their own storage servers from scratch. Clearly this is universal sage advice that will certainly not end badly for the vast majority of people -_-.

I'm on my second Synology, the first was a cheapish 4-bay; now I'm using a much pricier 8-bay. This is for personal use, but the time and effort saved by using this over hand building has saved me factors more money. I do recommend NFS over SMB for the cheaper models as smbd is a lot more CPU heavy and transfers actually became CPU bound pretty quickly. I've never had data corruption issues in my 10ish years of use, but one subjective data point is pretty crap.

I don't imagine that there are many personal users who need/want ISCSI or other more raw block I/O protocols. As for businesses, off the shelf solutions in your requirements range are often available, and unless you're looking for very specific optimized workloads, there's probably a bit excessively expensive vendor solution that can meet those requirements.

Qnap has had security problems.

But I have never heard of people saying they have lost data due to flaws in synology.

You can't just buy backup. You have to think about scenarios and you have to test the restore.

As an example, I have an online account and a synology nas. I test my backup every time I move to a new laptop, by doing the data migration trough the restore.

Synology, it turns out, had changed its backup solution, and finding the installer for the old tools needed to restore was non trivial. They also wanted me to click on every individual file, with 100+ K files to go. The alternative was to download a zip file, except this actually timed out before it could complete. All the data was there, except not in a decent restorable format.

Apart from that, this is not an offline backup. If your house has a fire or whatever, your backup is also gone. A NAS on its own is not enough.

I was hoping to use rsync.net as online provider, but they had no way to receive money from a SEPA IBAN. They also store data outside the EU, exposing me to a foreign legislature with markedly lower consumer protection. When looking to the EU, the storage landscape becomes fragmented quickly. There are options, but its not easy.

Western Digital, Netgear, or Buffalo will do it for you. The cost is a trade of your time...plus the added value of a device that does one thing and does it well.

As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud, given the following constraints:

- data retrieval is not needed except for recovery case

- ideally, the backup process on the home server side should not be more difficult than a cron job running "rsync -avz --delete /mnt/raid user@server:/mnt/storage"

- integrity of everything should be assured - there's a couple of filesystem-level backups made with "rsync -av" on the data store which means UID/GID, chmod, symlinks, special files (e.g. device files) and whatever Samba uses to store Time Machine xattr metadata must be kept, and there should (but not must) be a way to verify if the file content in the backup is still intact.

- there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest

- it should be somewhat affordable (e.g. Amazon Glacier is ~33 $ a month, Backblaze ~40$ a month)

- ideally, there should be some form of asymmetric encryption be used so that decrypting the data requires the possession of one of three off-site YubiKeys with each having a distinct on-key-generated RSA4096 key and the corresponding password.

The easiest way to accomplish the first three targets would be to simply spin up a tiny AWS EC2 instance with an attached LUKS-encrypted EBS volume or rent a dedicated/colo server somewhere with the same setup and run "rsync -avAHX --delete" on the home server, but that's not affordable and there is a risk of the provider/a hacker accessing/manipulating the cloud server while it is running.

Duplicity plus any "dumb storage" seems to fulfill almost all constraints, but it seems to require either symmetric encryption or access to the private key on the home server - otherwise, how would it be able to decrypt and read the existing backup to find out what has already been backed up?

> As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud

I personally just throw it all into a .7z file and then upload it to a cloud service (Mega, specifically). I chose 7z as it has the capability to encrypt file names, which would otherwise leak a lot of metadata to the cloud storage provider.

The main downside of this is that Mega deletes your stuff after 3 months of idle time, so I need to periodically check in on it to make it think I'm still using it.

I am really not sure how 8TB of data would fare with this setup though. For one, you'd definitely want to set up some sort of streaming thing instead of buffering the .7z file on the server and then uploading it.

Re-uploading 8 TB each month is impossible, here in Germany all you can get on a residential VDSL line is ~10-20 MBit/s, which means you're stuck at 24/7 uploading for way over a month.

Germany: I get 100mbit upload consistently and with 1gbit fiber would get 200mbit+. If in dire need with a second provider that has the fiber in my home already I could get 400mbit+ upload.

> Re-uploading 8 TB each month is impossible

Woah, each month? I was assuming you would be syncing deltas, i.e. only upload what's actually changed. An ideal system would be to keep a list of files and their respective hashes on each 7z archive task, then do a final checksum of the 7z file so you can ensure its integrity when you download it from the cloud.

A quick look at the manpage[0] shows there's an update option, although wouldn't you have to keep the whole archive locally?

[0]: https://linux.die.net/man/1/7z

Look into restic or Borg backup. Restic with rclone can directly backup to a lot of cloud providers, with Borg you may have to remount the remote share.

Used rsync.net with borg successfully for years, +1

Aws Glacier deep archive would be $8 a month

At least until you need to restore, then I understand that you will pay much more.

Or just throw Garage on a few nodes and duplicate the data all over the place.

An option which seems to match most if not all of your requirements is borgbackup. As for the private key, borgbackup has several modes, the default being one where the private key is stored on the server but encrypted with a passphrase; if you don't like it, you could use the mode where the private key encrypted with the passphrase is kept locally and then backup that (very small) key file separately.

Rclone does the encryption and incremental backups

> ideally, there should be some form of asymmetric encryption be used so that decrypting the data requires the possession of one of three off-site YubiKeys with each having a distinct on-key-generated RSA4096 key and the corresponding password

This last one seems to drastically change the design into a place that hasn't really been developed. It would be quite neat, in addition to adding other properties like a compromise of the backed up server couldn't be used to destroy a backup. But assuming your goal is to pragmatically get it done, I would forgo this and accept that your backed up server is going to have full control over the backups.

FWIW I view online/cloud backups as having the strength of being synced often, and therefore up to date in case physical damage happens to the server. For longer term storage I use offsite drives in a safe deposit box, that take care of things like infrastructure being compromised and deliberately deleted along with online backups. For long term data integrity I use ZFS, borg verify, and ad-hoc sha512sums (for things that don't change much, like music collection).

In general keep in mind that your 8TB to backup likely has a power law distribution where your really important filesets are much smaller and can therefore be inexpensively backed up with more copies.

The best pricing I've seen for block devices is Kimsufi (10 USD / 2TB-mo) and buyvm.net (5 USD / 1TB-mo) although I haven't used the latter.

> In general keep in mind that your 8TB to backup likely has a power law distribution where your really important filesets are much smaller and can therefore be inexpensively backed up with more copies.

In my personal experience a simple ignore list can reduce size by an order of magnitude. Things that doesn't make sense to backup like thumbnails, cache files, development dependencies. And smaller is more robust, not only can it be inexpensively backed up with more copies but it can also be synced more frequently with a lower risk of partition and also is quicker to download and restore.

> there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest

Gocryptfs.

Friendly, pedantic, hopefully helpful reminder:

> - ideally, the backup process on the home server side should not be more difficult than a cron job running "rsync -avz --delete /mnt/raid user@server:/mnt/storage"

A mirror is not the same thing as a backup.

I think https://www.tarsnap.com/ might do the trick for you.

Check out Crashplan small business - $10 a month per endpoint.

Not officially supported on a headless server, but there are docker containers that work.

Actually, this shows why using Google Drive as a backup storage may be a bad idea.

Reply on news.ycombinator.com