Comment by ebrahimh | Hacker News Reader

Comment by ebrahimh | Hacker News Reader

I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest.

All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.

I plan to implement it like this:

  vps1:
  - restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
  homelab:
  - backrest container, to back up volumes/, do prune/check, replicate repo to offsite
  - rest-server container, will store backups from vps1, homelab, offsite
  offsite:
  - restic container, backs up volumes/ to homelab every 24 hours
  - rest-server container, store copy of backups from homelab

Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.

The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost

I've been trying to find a solution for this too! I was considering using Rclone but too many things are using SQLite for me to trust rsync. I was also going to go with CoreOS but I'm leaning towards Fedora Cloud now in case I need to manage things a bit more (and "auto updating" is not something I want as that suggests auto rebooting).

Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?

I like CoreOS because of the fact that all config, etc files, sysctls are in one place.

Previously, I was using artix and had this “etc” directory[1] checked in to keep track of system configuration, but there was no good way to keep track of config drift (other than remembering to update this dir).

Haven’t gone through a CoreOS update yet (been using for ~2 months), but doubtful it would break anything. I’ve tested to make sure all my containers shutdown gracefully etc.

The SOPS (secrets.yaml) pattern is more common in NixOS configs, and I found it works nicely here too. In the artix setup, I had a bunch of .example files strewn around [2], which I had to remember to sync with the real versions.

Encrypting a key is just prefixing it with “enc_priv_”, SOPS will encrypt and decrypt it automatically. I keep “public” values plaintext to maybe help someone setting this up for themselves. Just have to double-check git diff before committing.

[1] https://github.com/ebrahim37/infra-template/tree/00eccff06ae... [2] https://github.com/ebrahim37/infra-template/blob/00eccff06ae...

Are you aware of sqlite3-rsync ?

It is a tool that was created by the author of sqlite. It does just what you would expected to do.

It is here:

https://sqlite.org/rsync.html

Wow, I had no idea. This is probably going to do the trick nicely. I really appreciate the suggestion and link!

If your system has a way to take consistent snapshots in the filesystem (btrfs, zfs) or volume manager (lvm2, perhaps with a bit of filesystem support to obtain fs consistency), that can be used to avoid database downtime, if desired.

https://www.postgresql.org/docs/current/backup-file.html

Reply on news.ycombinator.com