Ask HN: How do you manage files and backups as an individual?

Similar Asks (perhaps some inspiration?):

Ask HN: How do you manage your “family data warehouse”? (127 points, 10 months ago, 135 comments) https://news.ycombinator.com/item?id=37520374

Ask HN: What tools/strategy do you use for backups? (48 points, 2 years ago, 22 comments) https://news.ycombinator.com/item?id=34197837

Ask HN: How do you manage your important personal documents and other data? (91 points, 2 years ago, 60 comments) https://news.ycombinator.com/item?id=33615785

One more here: Ask HN: How do you do personal backups in 2023? https://news.ycombinator.com/item?id=38576809

Those threads are super interesting! I remember seeing some of them a while back and being surprised at how many people were using some pretty advanced setups!

I started using Nextcloud for file/contact/calendar syncing a few years ago, and have gradually moved most of my digital life into it. Documents and photos, but also scripts to automatically set some things up for me when I do a fresh install of Linux (I've been playing around with a few different distros lately). The only thing that doesn't live in Nextcloud are some old DVD rips, and that's mostly due to "haven't gotten around to it yet". Besides those, if it's not in Nextcloud, it's not something I care too much about losing if a disk were to fail or I decide to wipe the disk to try a different distro.

My Nextcloud server is then backed up to another drive on the same machine plus two off-site locations - an old server I still run at my parents' house, and and external HDD my friend let me plug in to his home server. The on- and off-site backups are done using borg [0], which does deduplication and encryption (with the keys backed up in 1Password).

I've been meaning to set up an automated restore on one of the offsite servers - a script to automatically unpack the latest backup, set up a fresh DB, and make things read-only - firstly to verify that the backups are functional and complete, but also as a backup in case my home server or home internet goes down. I know in theory I've got everything I need in the backups to do a full restore, but I can't recall the last time I actually tried it out...

[0] https://www.borgbackup.org/

I really like the idea of Nextcloud, but I don't quite trust anything self hosted to have Google's level of availability.

I don't think I'd want to build a workflow that has a hard dependency on a DIY server, or anything that would take more than a half hour to set up from scratch if it breaks during some big project.

That's a super interesting idea to have off-site backups at a friend or relative's house. Seems like it could be a pretty good use case for SyncThing's untrusted nodes[1], because then you wouldn't have to deal with any network config stuff.

Borg is what I'm currently using to back up to a USB drive and it definitely seems to be top notch(Besides me breaking it one time by accidentally installing a global pip package, because Linux distros are insane and use a global python environment...).

[1]: https://docs.syncthing.net/users/untrusted.html

I use the following config:

  Fujitsu D3417-A
  Xeon 1225v5
  64GB ECC RAM
  WD SN850x NVMe 2TB
  PicoPSU 160W with Bojiadafast Adapter

All in all it costs about 200 bucks (Fujitsu Celsius W550), draws 10W Idle and has enough performance for a home server.

OS is Proxmox with native encrypted ZFS root using zfs-auto-snapshots (ransomware protection) and docker LXC with Portainer, Nginx Proxy Manager, Letsencrypt SSL and OpenWRT+duckdns.org to make it available locally.

Docker containers I use:

  immich - photo backup
  flatnotes - my personal notes
  kavita - ebooks
  navidrome - music
  audiobookshelf - audio books
  jellyfin - video
  pihole - adblock
  stirling pdf - pdf editing

Backup happens to an external usb drive with one small zfs bash script and a tasmota shelly plug (Switch on the shelly plug, wait until it is available, run incremental zfs sync:

  zfs send --raw -RI "$BACKUP_FROM_SNAPSHOT" "$BACKUP_UNTIL_SNAPSHOT" | pv | zfs recv -Fdu "$DST_POOL"

To keep it save I also backup to my parents house regularly and the most important files via

  restic

to an old Dell T20 as well as burn it on a blu-ray. I also considered Hetzner Storage Box for this.

Google is an absolute no go to me, especially unencrypted.

I don't know I just manually manage everything on my computer. I have a older computer that I use as a NAS that has all solid state storage in it which is a mixture of SSDs and NVMe storage and it runs RockStor. As I know I'm going to need something I just throw it in my NAS. Over time when I need space I will throw in my Intel ARC A750 to transcode movies or TV shows I have downloaded to AV1 using Staxrip. No set bit rate for that because that is stupid and inefficient. Read about the codecs that exist for audio and video and once you learn about them you know how to correctly use them and you could use a quality setting instead and get more efficiency.

When thinking about a backup system you can get really caught up in the weeds with having automation and scripts and these fancy ideas of cold storage going to an external drive and then synchronousing with a cloud service and then setting up another thing that will automatically synchronize what's on your disk with a close friend that you have who also has a big NAS for a second off-site storage... None of this matters if you have never actually tested a scenario where you need to get your data back on whatever devices that end up failing. Before things get too bad, simulate a drive failure on your main computer. What would you do if you had to restore things?

Once you've worked backwards from the best way for you to actually get back whole and working then I think you will be able to answer your own question on ways to manage backup.

Run an OS like TrueNAS (core == BSD, scale == Linux) and back your data on ZFS so you can run snapshots, and write snapshots to offline storage.

Implement a 3-2-1 model: 3 copies of your data, 2 locations of media, one copy held offline.

I use google drive, but many people I know like Apples pricepoint and others like the bundling with Microsoft.

Tarsnap is worth investigating. The Author writes here, and is a long standing FreeBSD developer outside of this archive service.

Nothing lasts forever. Be prepared to continue to invest on the capex side because your NAS will eventually die.

Tarsnap looks pretty good for some really niche use cases, but it seems pretty crazy expensive if you don't need the ultra privacy guarantees.

Is ZFS actually a possibility on 512MB of RAM? I always thought you needed actual server-class hardware way beyond a cheap consumer NAS for that, at least if you want the deduplication.

It does look really nice, it would be super cool to be able to just deduplicate everything globally, even between backups of different machines.

tarsnap is extremely expensive because it's pretty gold-class. If you can't afford it, then the majors (apple, google, microsoft) are probably your best bet because they do volume.

Yes, you can run ZFS on machines with small memory, if the primary role is archive media. There is a huge misunderstanding about why ZFS gobbles memory which is to do with cacheing for fast access. When you run it as a clone-backup strategy, you're not doing the kind of workflows which demand a giant ARC or intent-log.

But indeed you cannot run de-duplication on that, and there are people who say it's unwise anyway. I don't run it, I do run lz4 compression and noatime. With snapshots after the first copy, it's incrementals so the only blocks being written are against files with changes. There is some wastage, I agree but I think lz4 makes up for it.

A counter case might be my partners windows backup (pre-11 which hides this stuff) which has a huge chain of Outlook files, mostly the same data, all slightly different. But with 12TB class disk, this becomes less of an issue.

I moved to a zimablade with SATA card, from a pi4 with USB-SATA bridge. The ZFS migrated fine. I have a USB3 removable for my cold storage and write the ZFS incrementals to it periodically (monthly)

I probably should have bought a 4 or 5 disk TrueNAS capable chassis, and I would aim for 1-2GB over 512 if you can. If its a single disk NAS, you're exposed to more weaknesses if you don't have something else. But I have to keep reminding myself a ZFS Raid is not a backup.

I got the 512MB 223j mostly just because the refurb version seemed to be the cheapest well-reviewed, low power, and current model 2-bay NAS, and I assume the tock firmware is probably optimized to work well enough.

I can think of a lot of cool extra possibilities with more memory though.

I'm not seeing a ZFS option for Synology's stock firmware, and I was hoping to stick with their OS, but if it's vastly and overwhelmingly better I could see switching to something else.

Are snapshots actually necessary? Do they provide something incrementals to the external drive don't? I guess they're lighter weight and you can do them more often?

If the FS implements snapshots "down low" you get COW efficient, and pretty much lock-free atomic "it's done" -if you run userspace "get me the difference" then it has all kinds of runtime costs and risks.

That said, rsync is pretty good.

Your J model can't get more memory. And you look to be right, its the OS it comes with and thats it. The NAS you have is the best NAS sometimes. It's all very well positing better ones, but they aren't the one you have: So make the most of it.

Yup, the best NAS is the one you can actually justify buying!

I use a NAS, with some custom scripts.

Mac laptops backup to the NAS using TimeMachine. Windows laptops backup to the NAS using Backvp2 (using its own scheduler). iOS phones use PhotoSync to sync photos to the NAS. Various real and virtual Linux boxes sync to the NAS using rsync in cron. Music is kept on the NAS, and served via Roon+ARC (atm, anyway).

The NAS has four drive slots. I use two in an LVM RAID mirror for the primary storage volume, formatted with btrfs. A third slot holds the "offsite" backup drive, which takes a btrfs snapshot every morning at 4am. Every month I send the offsite drive to my friend (who sends my other offsite drive back).

I also take irregular Carbon Copy Cloner snapshots of the Macs to an external drive, in case the SSD dies. This makes a simple restore of the OS and apps quick, and then I can pull the latest daily snapshot of the home directories over that.

There's also an archive folder on the NAS for stuff that's no longer live on any laptops, but I want to keep. Old projects, old scanned documents, etc.

I'd love to have a proper hierarchical storage system set up, but I haven't looked into how possible that is in the last few years (and last time I looked, the solutions weren't great).

That's pretty close to what I was planning, minus the Rysnc and with Borg instead.

I'm surprised at how many people are swapping off-site drives with friends, that is actually really cool.

I still haven't decided on if I want to do upgrade to RAID or not yet.

Just using one drive as a backup for the other seems like it has more flexibility, you can upgrade just one drive at a time, make use of the extra space with mismatched storage, etc, but upgrading both drives at once might be a better plan...

I have a pair of (different) 8TB drives in the mirror: one Seagate, one Western Digital, both NAS-oriented models.

The offsite drives are both Seagate 16TB. As the content on the live mirror changes they slowly fill up with snapshots.

I’m getting close to needing new drives, but I haven’t decided what to buy yet. The mirror has about 1TB left. I’ll probably change it to 16TB, and the backups to 24 or maybe 28? Not cheap though.

You've got everything that you need in Synology.

* Use Time Machine to automatically backup your computer to Synology NAS

* Use SMB Share to manually manage big files or archives when you need it(use rsync if you lazy)

* Use CloudSync(synology app) to automatically back up your Google Drive to your Synology

* Use Synology Photos to (semi)automatically backup your photo library form your phone to your NAS

* Use Hyper Backup to backup your NAS to the cloud using S3/Backblaze B2/rsync.

One thing that might be missing is auto transcoding. 223J might not have enough power to transcode videos, but if you get somehting similar to DS920+, it can do transcoding for you. I've got mine setup with Jellyfin container, and it plays very well because CPU can transcode that.

Old crappy PC has a shared folder on it. That folder is backed up using backblaze. Dump stuff I want backed up from other PCs there.

OneDrive family plan turned out cheaper than having to setup everything myself with Multiple phones, laptops and OSes with elderly parents not living close to you.

I keep an external ssd with some important backups and that's about it.

Take printouts of photos as your digital copies will likely go away with you or when you stop paying.

I think the main idea of self-hosting is privacy and avoiding vendor lock-in, not to reduce costs.

Aren't you concerned that your files will be used to feed AI models, or to answer questions about you, or to increase ad targeting?

Old HP Microserver running Ubuntu & more importantly ZFS. Drives are two sets of mirrors. All contents get hourly/daily snapshots which persist for 35 days and then offsite backup using Duplicacy to S3. Roughly 500GB for around AUD$10 per month.

Wasn't expecting to see so much ZFS for personal use! I should probably check it out seeing as how it seems to be universally loved!

You can set it up in a pretty simple manner and it is just reliable and solid - nothing much you need to do.

I use a backup tool, Xopero, for automating by backups

Nextcloud hosted on my local network and syncthing.