How I Backup | Hacker News Reader

How I Backup

sive.rs

115 pointstwapi82 comments

Two additional suggestions:

1. Test your backups. I see no testing procedure described, and although maybe the author's scripts have spot checking beyond what is described in the article, code will only ever catch the cases it was programmed to expect.

Manual testing does not have to be frequent or extensive, but making sure the backups are still recoverable and complete (take five minutes to spot check some old and new files) every six months helps ensure no silent issues are occurring

2. Less importantly: use append-only backups to an independent system. While ransomware for consumers is comparatively rare these days, it would give me peace of mind. The author's deposit box mostly fulfils that purpose, but whether it gets filled with bogus data or wiped by the malware is a matter of luck in timing (with the odds heavily in their favor due to the irregular connection of the drive and the existence of another computer that the malware probably isn't programmed to expect).

Of course, "append only" is a bit of a misnomer because eventually you'd run out of space, but the system should independently delete old data rather than being able to command it remotely (from the potentially infected system) to wipe anything. Restic has docs with considerations for data deletion in an append-only setup (disclose: I helped write them), which are also relevant if you don't use restic because they describe the method more than the specific implementation: https://restic.readthedocs.io/en/latest/060_forget.html#secu...

Doesn’t sound like backing up, but keeping your stuff massively in sync.

What happens if you discover something is missing or corrupted after 6 months?

If it's not something super critical in my experience the simpler solution is better for longterm or fool-proof backups.

E.g. if I use some special backup software that creates incremental archives or whatever, and then I die... My family would have absolutely no idea how to access that data. While with a normal NTFS partition and data on an external HDD they could just plug it in some PC and look at the family photos.

I use a rsync script that does incrementals but keeps full copies of each backup. Although mine is significantly different, it is based on http://www.mikerubel.org/computers/rsync_snapshots/ by Mike Rubel.

It really depends on what we're comparing this to. If one compares it to a commercial solution that allows backing up to user-owned storages, then the commercial solution is simpler while having more functionality (including versioning).

The ZFS clone should deal with that via regular snapshots.

Only if you have infinite storage space.

Sadly, like any other option, ZFS can't magically create more storage space.

People usually do something like keep hourly snapshots for a week, weekly snapshots for a few months and monthly snapshots for a year. My policy with data is similar to real life things: if I didn't notice it gone for a whole year then I probably don't need it.

Are you perhaps hinting at using offline storage for long-term archival? What options are there? Tape? As far as I know, keeping hard disks online is still the best way to achieve this, sadly.

> here is how I do my backups. It takes me about ten seconds per day and five minutes per month to maintain

> rsync everything to each of them, bringing one down to a safe deposit box downtown, and taking out the one that was there from my last visit

Sounds like you are underestimating how much time you spend on backups. Sounds good though

I assumed the author has to make the trip to the bank anyway. I integrate my off-site, offline backup storage with visits to my parents, for example.

My absolute final resort "solution" for data loss of keepsakes:

Accept the impermanence of data, memories and memorabilia.

Ideally I'll strive to keep that data backed up as long as I can, and will mourn its loss, but mentally I prepare myself that one day I might lose access in some way. Most importantly, that data is also 99% irrelevant to the next generation, maybe save for a handful of photos or videos ("what did grandpa look like?"). So in the grand scheme of things it's not like I have to be an epic archivist of my own life.

I suppose it helps that this was done for me in the past - toxic family members threw out all my childhood family photos a decade ago.

"Toxic" seems quite the understatement! How sad.

I went to school with a boy who had no photos from his youth. His parents took many photos but didn't get them developed, and when they took all the films to the pharmacy (here in the UK) to get them developed, the pharmacy sadly burned down and they didn't ever get any photos. Quite sad I thought.

I use a similar, but I think simpler, setup.

"Files I use and change every day" are on Dropbox, which gets synced to a NAS.

Photo and video files are directly stored on the NAS and don't live anywhere else.

The NAS is rsynced automatically every day to an external disk (connected via USB) of which there are three copies. One is in a safe at home, the other at my parents' house. The one in the safe is rotated every week, the one at my parents' every month.

If my house gets robbed or even burns down it's likely the disk in the safe will survive and I lose only one week of data. If the house gets completely destroyed or the safe is cracked, then I can turn to the disk at my parents and lose one month.

It's true that rsync is not a backup. But by default rsync only adds data. Delete is an option one has to turn on. My daily rsync does not have the delete option. I have a different script with delete, that I run manually.

The theory was that I would review the delete process with a dry run before committing to it. In practice I don't do this and simply run the delete script occasionally, which ruins the purpose. But I guess I could be more careful if I really wanted to.

I've been happy with: https://kopia.io/

Fairly easy to configure, does snapshots to S3 and has a icon in my tray I can watch :)

kopia is one of those things are forever in my todo list while i use rsync daily.

It seems that Kopia doesn't support continuous backup, which may be a requirement for some users (detecting changes on a large number of files is resource-intensive).

Interesting. Just for sharing, for backup on Linux I've been using rsnapshot, restic (which accepts Rclone as interface) and ReaR (Relax and Recover) for full linux recovery; on Windows, commercial Acronis and recently UrBackup. Old good "pathsync" from Cockos also helps on LAN.

Everything boils down to your amount of files or GBs when some solution becomes "too slow" doing incrementals and/or file processing. My two cents.

On Linux: https://www.borgbackup.org/

> Whenever I turn on one computer, I sync it from the other.
> Daily rsync to an encrypted Linux USB stick attached to the desktop.
> Daily rsync to an encrypted MacOS USB stick attached to the laptop.
> Daily rsync to an encrypted ZFS SSD inside the desktop.
> Daily rsync to OpenBSD remote attached storage at vultr.com

> [...] I do it a few times a day, especially if I’ve just made or saved something of value. And always right before I shut down the computer, which I do almost any time I step away from it for more than a few minutes.

Have you ever noticed blog posts rarely go viral if they describe a normal setup, and are much more likely to do so if they describe a really complicated setup?

I'm all for people coming up with needlessly complex homelab setups if they have fun doing so - but sometimes I wonder whether newbies see this sort of thing, and think Linux needs this level of complexity.

What complexity? rsync to 4 locations, likely from some script?

It says in the post he runs them by hand.

And how should he run them? He has to manipulate HW things to do the backup. It doesn't mean he has to do 10 steps for each of these backups. It can still be in a script.

Describing systems in the article's terms ("5UP3R 51MPL3, BR0S") is disingenous.

Backing up to multiple manually managed locations requires also maintaining those locations, which means spending time on system administration, which is not negligible (it's sysadmin work).

At best, the sysadmin time is amortized, but in real world, it never really is.

It's difficult to compare this strategy to actual backup programs, because it's not really a backup (it misses some functionalities of backup systems).

If the independence requirement is removed (which can make the comparison improper), then for 10/20$ a month there are backup programs that require no maintenance at all, adding other functionalities at the same time (which can include user-owned external storage locations).

Manage what? The OP talks about SSD USB sticks and external HDDs. You plug it in, format it, maybe enable some encryption once at the beginning, and that's it.

If your data fits a single drive, there's no management to speak of.

Then he mentions 2 managed online services, that he doesn't manage himself.

He mentions three cloud services, two of which (vultr.com, ZFS.rent) just provision a bare VM which he has to manage.

And it's not the easy type of sysadmin work where the systems are identical - he's got a mix of Mac, Linux and BSD so 3 different init systems, 3 different disk encryption systems, and so on.

And despite the fact he's paying for three different hosted off-site backups, and he's got five backups within his own home, he still feels he needs to haul an extra backup to the bank every month?

This is the sysadmin equivalent of building a bunker stocked with canned food "just in case" there's a nuclear war.

Yes, "bare" single-user VM with no publicly available services is something you barely have to manage if at all. It's not like it's bare metal where you have to deal with it failing.

4 rented/hosted offsite backups seems overkill. But I can understand that the number of on-site backups doesn't really matter for the safety, because there are several event types that can destroy them all at once. 1 owned backup device in a different location seems perfectly reasonable. Hosted stuff can easily go away if you stop being able to pay for it or lose access, for any reason.

Anyway, of course some "I trust backblaze with everything, incl. quality of their backup software" setup, will be much less complicated and will require much less knowledge.

My backup system is "I store all my files on my NAS, which takes ZFS snapshots every few hours, and it automatically rsyncs everything to rsync.net".

Yes, it took a while to set up, but I don't have to ever think about backing anything up. It's all just done automatically, and it takes me zero seconds per day and zero seconds per month.

> I wonder whether newbies see this sort of thing, and think Linux needs this level of complexity.

To be fair, people who aren't already familiar with Linux, won't even come across these articles to begin with.

But I can see when this type of articles reach someone through a network on Twitter or reddit, when they hear alien technical tools and script this and that they think: "Cool, that's what IT specialists do, not for me, I'm doing fine."

> Drive downtown to a safety deposit box.

Resilient? Yes. Simple? No.

>> Drive downtown to a safety deposit box.

> Resilient? Yes. Simple? No.

If you have to go into the office once or twice a week (i.e., not full remote), then bring the drive to your office (assuming you have an assigned desk).

Heck, have a bunch and take one in and bring one back as a rotation scheme.

That makes it simpler indeed.

Another good option is to have a little rapsberrypi (like) powered NAS at a friends and them at yours and you do rsync to each others house.

Question for rsync users on linux. Does your rsync preserve creation times --crtimes ?

I'm aware ext4/3 filesystems don't store creation times. But I want to use rsync on Linux to sync files between an NTFS(supports creation times) and a Btrfs partition(also supports creation times) without losing the creation times.

Currently rsync just sets the creation time to the modification time on the destination directory. Which is not what I want.

When I use `--crtimes` I get a "This rsync does not support --crtimes (-N)" error.

Weirdly, just using `cp -a` works. But it doesn' t have any of the checksumming and differential copy bells and whistles of rsync.

How do you rsync on linux while preserving creation times(on supported file systems)?

I just migrated from a QNAP NAS to a custom build with ZFS. I have files with creation times dating back to the 90s on my original FAT16 filesystem and I didn't want to lose them, so I recently did a (way-too-)deep dive on this, I'll summarize my notes for you.

First, ext4 actually does support creation times, called "crtime". But there's some internet confusion about it since this support predated linux kernel support, so you had to use ext4-specific tooling (on an unmounted filesystem) to access it, e.g.:

  debugfs -R 'stat <inode>' /dev/nvme0n1p1
  debugfs -R -w 'set_inode_field <inode> crtime <new-time>' /dev/nvme0n1p1

The btrfs situation is similar, but btrfs called it "otime" (for some reason?). Linux 4.11 introduced kernel-level support unified across all filesystems, calling it "btime" (birth time).

But the normal file syscalls only support reading btimes, not setting them to arbitrary values. And rsync on linux, as you saw, can't do anything the kernel doesn't have a syscall for. For a while the only option was to:

1. set the system clock

2. create a file (at which point the kernel sets btime to the system time, plus a few nanoseconds)

3. restore the system clock

Obviously a huge hack, and needs root, but tools like s-tar automated it (search for "time storm" in this manpage):

https://web.archive.org/web/20220331080358/http://schilytool...

I almost gave up and spun up a Windows VM (since Windows has supported reading and writing creation times since the beginning). But then it clicked -- the kernel interface to the filesystem module takes the btime as an explicit parameter. So if you could find a kernel module that talks to the filesystem module directly (instead of going through the usual high-level file syscalls), you can pass along any btime you want. And there just so

I use Syncthing on my PCs and NAS, and Backblaze to trivially do a 3-2-1 backup strategy.

My work directories are synced on all devices. With my NAS up 24/7, I can always pick up the last changes when I switch computers. The irreplaceable documents (all my notes, taxes, company accounting,...) are also backed up by the NAS on Backblaze, keeping all versions. I occasionally create a snapshot on Backblaze for those files – should totally automate that but I'm lazy.

I do agree though: it's more "syncing stuff between multiple places" than a real backup.

I keep my important files on my DIY NAS -- a cheap NUC with two external HDDs (Btrfs in RAID1). Every night at 1am a cron job takes a restic snapshot to a Backblaze bucket. I feel like bang for buck my solution isn't half bad. The NUC is also my home server (web, Pi-hole, Home Assistant, ...), so it was already there, the HDDs were cheap and I currently pay less than $3 a month for over 300 GB of data in the bucket.

I do something similar[1] with rsync --link-dest and hard links, so that I can go back in time if I delete something. OP seems to just have a copy of his actual files, but if he sees a file missing after 2 weeks after deleting it, he has no way of recovering it.

[1]https://31337.it/incremental-backups-with-rsync/

Or if you don't want to write a DIY solution you could use this (shameless plug): https://rsync-backup-ras.sourceforge.io/

Everything that matters is checked into fossil. Box hosting that uses zfs. That rules out most failure modes other than gross incompetence, partially mitigated by a periodic backup to external drives.

In the before times I put everything in Dropbox. Dropbox synced with an empty directory by mistake and deleted everything. Not going to use Dropbox again.

I'm using :

- Dropbox for "core files" (small subset of files I don't want to lose, mostly my notes, papers, etc) - Arq with S3 for backups (do a backup each hour) - An external SDD to do semi-frequent backups (mostly used in case I change laptop to ensure I have a backup of my files, but otherwise I mostly rely on Arq)

Blank lines prevent the bullet points running together

I use Vorta. I have two profiles, one for my home dir, and recently I added one for all dotfiles I would want to keep when setting up a new machine. It's only about 5 files but it avoids using yet another tool for dotfiles.

Are backup stored to Glacier is my house-burned-down archive for about $5/mo. It may cost a lot to get out, but at that point our family pictures and movies are gone anyway, so it would be worth the cost.

Thanks for sharing! I'm going to go and test my backups now...

that's a somewhat expensive setup to still do all the work of dealing with dead drives and manually doing replication and dealing with conflicts (I fear the pain of dealing with his 4way daily backup of the same data over desktop/laptop/2 pen drives)

zfs rent = 4+TB drive + $10/mo (base 1TB) + $5*3 (extra 3TB) = drive + $25/mo

hetzner = 5TB $10.9/mo

vultr = $5/mo (probably more)

4x ~4TB drives = $800

assuming you replace drives every 5yrs: $3,254 or $54.23/mo.

assuming the drives last 10yr which are common now: $5,708 or $47.56/mo.

A 4tb drive costs 80-120€

7€/month for 5y

that price is for barracudas. then you will be lucky to replace them every 2 years.

wd gold and others that will last 5~10years are actually more expensive for 4TB ($500) because they are old stock for people that for some reason need that model.

The sweet spot for top quality HDD is always 200~300. In whatever capacity du jour. (14~20TB today)

nitipicking drive cost aside, the monthly services are the bulky of the cost.

Anyone know/recommend something like ZFS.rent in Europe?

I use rsync.net, but I don't know if their `zfs receive` is encrypted:

https://www.rsync.net/products/zfsintro.html

rsync for backup is a recipe for disaster if you don't realize you just rsynced and overwriten good data with corrupted or ransomwared data.

If you want to use rsync, do rsync and snapshots...and those snapshots should automatically made on the remote destination (can be just a daily cron), only accessible read only from the source machine without having any administrative rights to tamper them.

Author: describes a simple setup which probably puts him in the top 0.1% of personal users in terms of resilience to data loss.

HN: nitpick about potential problems which are present even in some industry grade data loss solutions.

And honestly, I never lost data in my whole life as a private user. Exceptions: Own stupidity in which case even back-ups don't help.

And that included the time when HDDs had a tendency to fail. And what little data loss occured was actually a grand total of 3 hours of work to recreate it.

I simply don't get the obbsession people have with data back ups for their private needs.

I had the great privilege of my hard drive dying 4th year university during exams / finals season.

And so I had the great luck of paying "student rates" for a data recovery service.

That was a well worth early lesson in life :)

Most people just cannot imagine losing all their data. They think it cannot happen, or it won't matter, or it will all be fine. What we have is a problem of imagination.

I run through a simple exercise with my non techie friends whave have endless faith in technology : "if I went to your phone and laptop with a hammer right now and smash them to pieces, how much would you cry?" (Some tear up right then and there ; others still think it'd be fine until I ask some follow up questions to instantiate the risk:).

Personally, I have scanned photos of childhood and tax and immigration and education and legal records and books I purchased and work documents and client photos from side photography gig and a million other things it'd be a pain to lose. So I have a local NAS and pay a cloud backup service and have reached my comfort level of data loss risk :). "Never lost data my whole life " is pure luck, and I wish it continues, but it's a bad plan - it's a bit of "I played Russian roulette and I'm fine!" :->

I had my mobile phone die out of the blue. I sync everything, so I wasn't worried, but later on I realized I had lost some old ringtones and notification sounds I quite liked. Nothing terrible, but it definitely showed me the importance of testing your backups (whenever you buy a new phone).

"Yes, but" - unfortunate number of my non-techie friends do not have any kind of sync / backup enabled on their phone let alone computer :(.

(linked somewhat directly to number of my non-techie friends and family who do not have good lock on their phone)

> Exceptions: Own stupidity in which case even back-ups don't help.

Quite a few times I've accidentally deleted the wrong thing. Backup saved me.

Yeah, last time it happened it was a weird combination of things... Historically, de facto back ups are clustered across computers, sticks and drives. Nothing intentional, only general messiness. I helps I hardly ever delete data, my problem is finding it again.

If you have to do a clean install of the OS so, well, you should check first which data is actually saved as a file and which is only saved in a database of software being deinstalled...

> And honestly, I never lost data in my whole life as a private user.

I have never been in an automobile accident in my whole life.

I still use my seatbelt.

My hard disk failed which had the last 20 years of my life on it including wedding photos etc. which I hadn't developed yet.

Now I back data up religiously. You don't think you need it, until you actually do.

> I simply don't get the obbsession people have with data back ups for their private needs.

Once you get bitten you strangely get obsessive about it!

Around 2005 I had the working copy on the laptop's only HDD + two backup copies on separate HDDs. The events that unfolded were as follows (timeline and details editorialised as my memory is fuzzy, but you get the idea):

- On a weekend, my laptop crashed, hard. It was on Linux and something broke. Upon reboot it fsck'd for way longer than it should have, then it yelled loudly that it failed. I had some bootable live USB thing, but that Linux outright refused to mount it. A quick analysis/recovery attempt showed that the ext2 data structures were all messed up. The few files it managed to recover were either unimportant or a garbled mess. The disk was fine hardware-wise, but software failed and resulted in unrecoverable corruption.

- No worries! I had a - weekly - backup from a few days before (Thu) so worst case I lost Friday's work on my Master's thesis work. Annoying but I can manage. I plug in the drive and even before attempting to mount the device greets me with a defying clicketyckicetyclick. The disk is physically dead.

- I'm now anxiously looking at my last - monthly - backup drive, which, since the next one was due some time the following week, was almost a month old. I'll have to scramble to redo shit for my Master, but, at least, a decade of keepsakes and digitised paperwork would be fine, right? RIGHT?? I plug it in. I hear a faint puff of magic smoke. The disk does not power up. Something in the disk controller is dead somewhere, and it's the kind that is straight USB from the board, no SATA. The disk is not produced anymore, but I somehow manage to find a presumably identical one second hand, at least as far as the part number is concerned. I swap the boards, plug USB, the disk spins up but it's unable to read meaningful data. I notice there's a small "REV4" marking on the new board, the old one is "REV1". When looking closely, some components are slightly different. Of course the identical part number is a lie. The data is there but I have no means to read it. Not within my already stretched budget anyway.

I now make sure to backup on systems with vastly different failure modes.

"data loss solutions" is a term/product category I can totally get behind :D And indeed, there's planty of those in the IT industry!

Please make your substantive points without snark or swipes.

This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.

Here I was starting to think 2 USB (platter) drives and the occasional rsync -uavPi is too much.

All power to the author, that's a hell of a set up.

My setup :

- servers and laptops (laptops : except /home): restic to local minio instance + rclone to B2 storage

- /home of laptops : kopia to b2 storage

I'm using rdiff-backup to have versioned copies, however at some point of time trimming of the snapshots is required

I do a zfs snapshot and than do a sync.

I never delete those snapshots because the big items like images don't change.

I've read USB sticks and backup in the same phrase, something must be wrong.

Given that it's just one out of several backup media, that's kinda fine, I guess? Just don't put all your eggs in one basket. Which is why I'd add at least one HDD to the mix, as all the local options appear to be flash-based. Flash memory can develop defects especially if it's not turned on for a while. For long-term backup storage, a HDD might be a better solution.

Redundant USB stick might be more reliable than a single USB stick. But for me a USB stick is just too unreliable.

I prefer putting my eggs in soft cosy baskets. Not in concrete baskets full of holes.

Depends on the stick, I have a couple of ~15 year old sticks that still work 100% reliably whenever I need a 4GB installation media, compared to HDDs that are less than 2 years old and riddled with remapped sectors :/

I think I've only ever thrown away 1 USB stick due to failure, microSD is my nemesis when it comes to putting 'reliable' and 'flash' in the same sentence, everything else seems to last just fine as long as it's not used constantly.

Just checked, my eldest is a 2007 lexar "twist n turn" 4GB, last formatted in 2019 with intel lan drivers on it, still 100% reliable, files on it verify ok.

USB sticks are much more reliable than hard drives.

That really depends on the stick. In Germany, there is a brand called Intenso that is unfortunately rather popular here, you'll find their low-priced USB sticks everywhere and I've had nothing but problems with the few I had to use. They often were much slower than advertised, sometimes they would just break down completely in the middle of copying files. It always felt like they try to source the cheapest flash chips that barely pass quality assurance. Not to mention all the "fake" USB sticks that report a larger size than what their actual capcity is, which you may only find out after it's too late.

In contrast, I have had a single hard disk with a fatal failure in the last twenty years or so.

They are not.

Most of this misconception is when people evaluated flash memory quality, when flash memory was expensive.

Today flash memory is the most cheap stuff you can get your hands on. And quality nose dived, and nobody re-did the comparisons.

Meanwhile HDD are not used anywhere but datacenters, and quality went up. For the same reasons flash storage was good quality before: low inventory only bought by quality conscious clients.

Why?

Quality of most USB sticks (meaning not all, but almost all) on sale is crap. As a result you might get a stick working for many years, for all files you put on it, while many others with important silent data corruption in a little time. Of course you can use them with a "modern" filesystem with block-level checksums but... It's like playing Russian roulette.

SSD tend to be better but are still good for fresh storage only, HDD are complicated, most have evolved toward crap, but good ones exists and are reliable and cheap and big enough for most usages.

Long story short: choose three HDDs from good brands/series and use them normally the likelihood of multiple concurrent failures are scarce and the final price is not that high. For really important stuff a simple desktop tape reader and few tapes TOGETHER with HDDs complete the games. SSDs inside desktops/laptops/home-servers do their jobs well.

I was of the opinion that the reliability of the storage medium does not matter much for backup purposes, so long as it's reliable enough to survive the days between a failure of another copy and the rebuilding of that.

But you've convinced me that it's not a blanket statement one can make, due to silent data corruption. Even with systems that have checksumming, you're right that I'm not doing a full data read on my restic backup regularly. I guess it can still be an option to use regular USB sticks, given the right software and discipline to run its checks, but at that point, it's probably easier to use a more suitable storage medium, especially since HDDs are the cheapest option per GB.

(Not that I personally use USB sticks for backup currently, but I do repurpose old hardware for things like my email server.)

Thanks!

I've used Backblaze for years and it's worth the money. I've only done a full restore from it once after a laptop was bricked and it was seamless. There have been a few times that I needed a recently deleted file and it worked for that too.

Ah yes, HN's automatic title editorializing strikes again, removing 33% of the title's wordcount and changing it from "click here to see my backup method" to "hey, I do backups".

> I Backup

Well, that's a fun casualty of the title filter!

I backup, therefore I am.

There was also (Why you should) “Still love Telnet” from some time ago[1].

[1] https://news.ycombinator.com/item?id=36179850

Is "Backup" an Ancient Roman name like Claudius perhaps?

Reply on news.ycombinator.com