Hacker News Reader

Comment by progbits | Hacker News Reader

> One way is to ensure that machines that must be backed up via "push" [..] can only access their own space. More importantly, the backup server, for security reasons, should maintain its own filesystem snapshots for a certain period. In this way, even in the worst-case scenario (workload compromised -> connection to backup server -> deletion of backups to demand a ransom), the backup server has its own snapshots

My preferred solution is to let client only write new backups, never delete. The deletion is handled separately (manually or cron on the target).

You can do this with rsync/ssh via the allowed command feature in .ssh/authorized_keys.

View on HN

Another thing you can do is just run a container or a specific backup user. Something like with a systemd-nspawn can give you a pretty lightweight chroot "jail" and you can ensure that anyone inside that jail can't do any rm commands.

  pacman -S  arch-install-scripts            # Need this package (for debian you need debootstrap)
  pacstrap -c /mnt/backups/TestSpawn base    # Makes chroot
  systemd-nspawn -D /mnt/backups/TestSpawn   # Logs in 
  passwd                                     # Set the root password. Do whatever else you need then exit
  sudo ln -s /mnt/backups/TestSpawn /var/lib/machines/TestSpawn
  sudo machinectl start TestSpawn            # Congrats, you can now control with machinectl

Configs work like normal systemd stuff. So you can limit access controls, restrict file paths, make the service boot only at certain times or activate based on listening to a port, make only accessible via 192.168.1.0/24 (or 100.64.0.0/10), limit memory/CPU usage, or whatever you want. (I also like to use BTRFS subvolumes) You could also go systemd-vmspawn for a full VM if you really wanted to.

Extra nice, you can use importctl to then replicate.

I fall into the "pull" camp so this is less of a worry. The server to be backed-up should have no permissions to the backup server. If an attacker can root your live server (with more code/services to exploit), they do not automatically also gain access to the backup system.

I also implemented my backup scheme using "pull" as it is easier to do than an append-only system, and therefore probably more secure as there is less room for mistakes. The backup server can only be accessed through a console directly, which is a bit annoying sometimes, but at least it writes summaries back to the network.

It is not particularly hard either. Checkout restic server.

https://github.com/restic/rest-server/

I used to do this too, using rsnapshot to backup ssh:// locations.

At some point I switched to instead using Syncthing to sync all my files to the backup server, and then the backup server did local backups from the Syncthing folder to the backup disk using Borgmatic. Works better for laptops.

Now I daily drive a Mac, and switched to Arq backup with a BackBlaze remote, instead of hosting my own backup server. More of a turnkey solution but works fine, especially given all the settings to suspend backups depending on e.g. battery status, WiFi connectivity, etc. when roaming around.

This is also why I use rclone copy instead of rclone sync for my backups, using API keys without permission to delete objects.

I do both. It requires two backup locations, but I want that anyway. My backup sources push to an intermediate location and the primary backups pull from there. The intermediate location is smaller so can hold less, but does still keep snapshots.

This means that neither my backup sources nor the main backup sinks need to authenticate with each other, in fact I make sure that they can't, they can only authenticate with the intermediate and it can't authenticate with them⁰. If any one or two of the three parts is compromised there is a chance that the third will be safe. Backing up the credentials for all this is handled separately to make sure I'm not storing the keys to the entire kingdom on any internet connectable hosts. The few bits of data that I have that are truly massively important are backed up with extra measures (including an actual offline backup) on top.

With this separation, verifying backups requires extra steps too. The main backups occasionally verify checksums of the data that hold, and send a copy of the hashes for the latest backup back to the intermediate host(s) where that can read back to compare to hashes generated¹ at the sources² in order to detect certain families of corruption issues.

--------

[0] I think of the arrangement as a soft-offline backup, because like an offline backup nothing on the sources can (directly) corrupt the backup snapshots at the other end.

[1] These are generated at backup time, to reduce false alerts from files modified soon after they are read for sending to the backups.

[2] The hashes are sent to the intermediate, so the comparison could be done there and in fact I should probably do that as it'll make sending alerts when something seems wrong more reliable, but that isn't how I initially set things up and I've not done any major renovations in ages.

>My preferred solution is to let client only write new backups, never delete. The deletion is handled separately (manually or cron on the target).

I do this by making backups visible to users on a `/.snapshots` directory that is the same as the target of the backup script, but mounted NFS read-only.

> My preferred solution is to let client only write new backups, never delete.

I wish for syncoid to add this feature. I want it to only copy snapshots to the backup server. The server then deletes old snapshots. At the moment it requires delete permissions.

You can do this by using a dedicated syncoid user and ZFS delegated permissions: https://openzfs.github.io/openzfs-docs/man/master/8/zfs-allo...

You'll need to add the --no-elevate-permissions flag to your syncoid job.

Reply on news.ycombinator.com