Don't have the grizzled old storage admin / DBA test the backup. They know a million and one weird necessary workarounds and just execute them. However, if you need a restore and they are currently exploring caves or something, things turn dire. Have a chipper junior restore something based off of the documentation (and prepare to spend a few days updating documentation...)
And make sure to test backup you don't regularly touch. And very much test those backups you really don't want to test.
Have a different person do it each time, having them add and refine the documentation and any tooling once they’ve done it. Keep any tools and scripts used fastidiously current-few things are worse than “to fix this issue, run the repair.sh script” only to find it stopped working 6 months ago because it relied on some extremely specific lib somewhere.
I think Postgres did it right by abbreviating their "Continuous Archiving and Point-in-Time Recovery" as PITR because it's very close to PITA. But PITR and CHECKPOINT actually make Postgres probably one of the better database systems to backup (and restore!), so yet another reason why I think it's a fantastic database.
If you don’t delete files as you go, now you have an unknown partial backup state that basically doubles your needed space.
If you delete as you go, sometimes something happens and the process stops or corrupts so your data is now split and you may have lost something.
Even trying to log all the failures during the process is amazingly difficult and solutions to work around that specific problem, themselves, somehow introduce more and new types silent failure in some type of irony.
I do think that file systems that support snapshots - like ZFS, but I think LVM can be used for stuff like ext4, and Apple APFS does too - is the way to go. Not sure how well NTFS's Shadow Copies/Volume Shadow Service work, I heard horror stories, but not sure if those are one-off freak accidents. Probably worth considering ReFS anyway these days on a Windows Server. But with a Snapshot, you're at least insulating yourself mostly from changes to the data you're backing up. At the expensive of managing snapshots, that is, getting rid of old ones after a while because they keep taking up space.
I like to keep a separate database of what files I expect to have along with their hashes. The off-the-shelf tooling for this is weak, to say the least. Even S3’s integrity checking support is desultory at best, and a bunch of S3 clones don’t implement it at all (cough minio cough).
Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices?
The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.
We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.
My parents tend to take a lot of photos whenever the family is together, and it used to bother me. Only in recent years I started to understand them.
Today you might take 10 photos of your family and keep the best one where everyone is smiling.
But 10-20 years from now you will probably appreciate having kept the other 9 where the baby is crying, the kid is making a face, and grandma has started to wander off.
Also, you can probably tier your data. Maybe you don’t need same level of backup for all your 40TB.
Glacier is always super cheap as long as you don’t need to recover, and then it’s ferocious.
For sensitive data I would run something else that can be a Restic target so backup data is encrypted, I currently use a cloud drive that supports WebDAV for that.
My NAS has something like 5 TB used. It's all synced to an old server that can hold about 8 TB and that's off most of the time (no fun living next to a jet engine). This cold server lives at my parents' house.
My "really important stuff" on the NAS, which is a few hundred GB of pictures and such, is regularly backed up to a bucket with object locking.
My "super important stuff", which is my company's accounting and other such documents, and lives on my laptop, is backed up to the live NAS and handled there as the really important stuff. I also back up my laptop to two normally offline external drives, one of which lives in my apartment and the other at my parents' house.
Everything non-cloud is ZFS, so after each backup to an external drive or "cold NAS", I run a scrub to make sure it is still operational. The live NAS runs a scrub every Monday morning.
Granted, this is not a "modern NAS" environment, since it made no sense to me to forego the free servers that my employer was going to send to the trash and buy some expensive off-the-shelf solution without the guarantees of ZFS (despite the issue TFA talks about). I know about power usage, but my live NAS eats less than 50W at idle (which is 99% of the time), so breaking even with the electricity prices in France would take forever.
You only have to backup the data that is important to you and you don't want to lose in case your house gets robbed, floods, burns down, etc.
If you don't mind losing 40T of data, you don't have to back it up at all.
Otherwise get another NAS, installed it at family/friend's house, and set up a VPN between the two: then use rsync/zfs-send/whatever.
Which is how it should be. Yes bugs can happen but there's only so many 9s most of us can chase on our budgets. And "always test backups" in particular adds cost. Testing means restoring onto hardware that you can then use live, separate from your actual primary hardware or at a minimum on primary hardware with >2x the set size and enough performance to squeeze it in during downtime or around work. So yet another big increase in cost. "Testing backups" isn't trivial.
Downside to tape backup is you need throughput, or the ability to do disk-disk backups
20 TB of SSD costs about $1000.
Or you could get a 20 TB hard drive for $300.
I have had multiple hard drives fail and been left stranded. Tape fails but not nearly as often as disks
LTO6 and LTO7 are not expensive for 20TB
Backup what you value the most, ignore what you don't and apply tiers depending on what needs to be kept but you can deal with transferring it back home slowly and what you need immediately in case of a failure.
My rules of thumb are:
- always invest 3x the price of your hot live NAS storage in backups. If you can't afford buying 40TB of storage, you can't afford having 10TB of live storage. Period. Goal is to have at least one copy locally and one externally and have more space to on the backup storages to account for retentions, changes and help with migrations.
- if you can't afford 3 redundant storages(RAID), favor having 3 times non redundant storage (no RAID) over having less copies of redundant one.
Additional tip to reduce cost and avoid expensive cloud offering is to find a reliable and trustable relative or friend that can host your external copy of your backup. Nebula or Tailscale now makes it very easy without having to configure routers and stuff. In exchange you can offer that person to host his/her backup storage.
Also digitalizing material stuff is nice, but printing digital photos is also a great way to preserve copies. I'd rather save the photos I cherish the most than having 3 backup copies of 10TB of blurry or non outstanding photos. After years of having them all digitally, I am inveting back in printing photos and making albums. You can also print photobook multiple times and have some stored at a relative's place.
I know you can use ZFS snapshots but I'm not experienced enough to trust that I could make a 20-40tb snapshot without screwing something up. Plus it's all video files so I can roughly keep track of what's what and I can ignore the stupid LTO compression.
It takes days, its noisy, and very tedious. But thats #hoarderLyfe lol
If you can figure out how to split the data into categories, you could save money as well. E.g., which of this data is truly irreplaceable - stuff like personal photos, source code, whatever it is that can never be re-created. If you're running a business, then stuff that needs to be available immediately in order to keep the lights on. Those things needs to be on storage that also gets backed up daily, preferably in full, and preferably to multiple clouds.
Stuff that can be re-created from sources (e.g., rendered outputs) are less critical because in the worst case, you can just spend some days/weeks to re-create it.
Also consider regular offline backups - put it on a tape drive or on some hard disks/SSDs or even optical media (yes, it would take something like 400 BDXL disks to back up 40 TB, but I assume the data doesn't rapidly change) and put it in some offsite storage facility in case your place burns down.
Glacier vaults in S3 are quite affordable these days.
Also afaik tapes need a stable storage environment: how do you manage that?
Bit rot is less of a thing with LTO, but still a thing.. I.e. you will at some point need to update your LTO system and it’s storage media.
The robot I owned was the library storage for movie frames at a major motion picture studio. We would upgrade every other release, so while I was there we were upgrading from LTO-5 to LTO—7.
The robot was big, and would write data to two redundant tapes. One copy would be sent to Iron Mountain, the other stayed in the robot.
Creating a backup like you are isn’t really protecting much if you don’t have a good facility to store the tapes in.
Part of the point of paying for a service like AWS Deep Glacier is that it’s an offsite backup.
An LTO backup has no advantage over a hard disk if your home catches on fire.
I also ran IT departments for the last 30 years, so you probably shouldn’t use me as the scapegoat :)
I did upgrade to an LTO-5 drive last year so so, after finding a new-in-box from a liquidation sale for something like $450. The nice thing about LTO is that it's 2 Generations R/W and 3 Generations Read - so the LTO-5 drive will Read/Write LTO-5 (1.5 TB) and LTO-4 tapes, and read LTO-3 tapes. I think with one of the new standards (LTO-8?) it's a bit more muddy, so check compatibility.
I think that LTO-4 and LTO-5 is the sweet spot for hobbyists: You still need to spend some money on a drive or two and buy brand new tapes, but it's reasonably affordable. That said, for a business, I'd just bite the bullet and buy a new drive. Dell sells an external SAS LTO-7 drive brand new for $3700 list price, but I think there might be cheaper options. Together with some tapes and a SAS Controller, I'd say that for $5000 you can get a decent, brand new setup.
I put the tapes in Turtle LTO Cases (https://turtlecase.com/products/lto-20-black), and they sit in a closet. It's not climate controlled or anything, but the place is roughly at a similar temperature year round. The tapes aren't THAT sensitive, but I'd definitely not store them in the garage where I might get a 50+ degree temperature difference throughout the year. That said, there are companies that offer off-site storage options with climate controlled environments. I haven't looked into their pricing since I didn't need it, but the nice thing about tapes is that you can just backup to two tapes and send the second tape off-site. LTO has built-in encryption support, so that's an option.
Twice a year or so, I run a restore of the tape and compare it to the SHA256 that I took while backing up the file (I did build myself some rudimentary cataloging system to SHA256 hash every file, then back it up to tape with tar, and make a record of what file with what SHA256 got backed up when on what tape). I've yet to encounter any bit rot/defective tape issues, but YMMV.
I do use Backblaze's B2 service as well for cold-ish storage. Though I only back up truly irreplaceable or inconvenient to recreate data into B2. That way, I have multiple copies of truly important stuff, I have stuff readily available where I am, and I have terabytes of stuff that isn't worth the expense for the cloud since I can re-create it, but nice to have a copy of.
Tape Drives may be overkill for many and external hard drives (plural!) might be a better option for many. What I like about tape drives is that the media isn't "hot". If I have ransomware running wild, connecting an external hard drive puts everything on it at risk (hence the need for multiple drives), whereas with a tape, it would have to specifically try to rewind the tape and start overwriting, and I would notice it. But YMMV, I never had a ransomware problem myself, but I do have stuff I really don't want to lose, so multiple backups of it in multiple ways (Daily .tar archive on a hard drive, backed up to tape, and backed up to the cloud) should hopefully give defense in depth and the ability to at least recover some older state.
Backup that isn't checked isn't done.