Hacker News Reader

Comment by izacus | Hacker News Reader

Can anyone recommend a Linux (preferrably cross-platform) tool like this which is appropriate for use in personal laptops?

It needs to:

- Handle random sleeps / disconnects properly without corrupting backup,

- Resume backup in progress if the process was interrupted due to sync or network issues.

- Be resilient to sleeps/wakeups in a way that it won't just fail to backup the machine for months on end.

- (Bonus) Be able to skip backup when connected to mobile hotspots or other slow networks.

Any ideas? I'm very happy with Arq backup on Windows/macOS, but I just can't seem to find anything on Linux. Everything just assumes a hardwired workstation/server and completely falls apart on a laptop.

View on HN

Can you mention what you've tried so far? I'm happy with borgbackup, but I don't know if it matches your bullet points.

Still, I'd be surprised if mainstream backup solutions corrupt your backups in case of sleep / disconnect. That would be a pretty big bug. Borgbackup also handles resumes IIRC. Your third point is unclear, you'd have to try in practice. But if throughput * time awake < size of compressed deduplicated data, you obviously cannot expect your backup to complete. I guess you just rephrased your second point.

For the last point, personally I think I'd take care of mobile hotspots / slow networks with a script wrapper rather than expecting the backup software to handle this.

I couldn't find any borg configuration to schedule and resume after sleep. Unless that changed, how does it fit my requirements?

Regarding sleep that would be the role of e.g. systemd (or whatever you have on your system) and you'd have to write a service unit handling that (although I'm not sure what you'd want exactly, if you have a program running when you put your computer to sleep, it'll still be running once it resumes, so not much to do).

Regarding schedule that could be part of the service, but a simple cronjob can also do the trick depending on what you want exactly.

You can just interrupt the backup on sleep and start it again on wake. It'll resume, though if nothing's changed it'll probably take a bit of time to realize that it doesn't need to sync anything.

Yeah, I just tried that and it works with restic, at least in my quick tests of it: slept the computer with restic in mid-scan, and it was ok resuming on wake a few minutes later.

A simple crontab entry to run restic every hour should work. I tested it running every minute and that worked fine.

If there's a chance that the laptop would sleep and stop restic in a mid-backup that was started by cron, then this[1] suggestion would probably be sufficient to work around that:

[1]: https://news.ycombinator.com/item?id=25552894

What I'm not sure about is whether restic would resume backing up a large file from where it left off if interrupted, or if it must re-backup the entire file.

I'm late to the thread but borg or restic or any of the attic/bup/Xbackup lineage will meet your needs. I haven't tried the GUIs but you can schedule it to run with anacron once per day and it'll run when the laptop first turns on. You can sleep 10 minutes before running to give yourself a chance to get connected. Because of how the repositories work, if a backup fails due to list connection the next one will use that partial data so the second backup will have less to transfer. NetworkManager knows the metered state of the wifi and so that can be read in a shell script to decide whether or not to run the backup. You may want to whitelist known good wifi networks instead.

Perhaps someone could take the excellent Deja Dup GUI, which is a front-end for the technically inferior duplicity, and have it shell out to borg instead.

"Can anyone recommend a Linux (preferrably cross-platform) tool like this which is appropriate for use in personal laptops?"

I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... the remote has no access to the data.

The borg website is here:

https://borgbackup.readthedocs.io/en/stable/

and a good description of how it works and why you should use it is here:

https://www.stavros.io/posts/holy-grail-backups/

Borg has zero scheduling functionality which means that it doesn't fit any of the requirements (at least not without a frontent).

It's literally the tool I was thinking about when I said "all tools expect a wired workstation".

I don't want to manually write bash scripts to resume and check backup status.

This is not intended to solve your use-case - in fact, what I am about to write is primitive and childish ... however, I have done it myself and seen others do it:

You can replace a command like this in your crontab:

  rsync blah blah

with a command like this:

  rsync blah blah ; rsync blah blah ; rsync blah blah

See what I am doing there ? I sometimes deploy this hack with very spotty WAN connections but the laptop sleep/wake is basically the same thing ... you wake up the laptop and the running command bombs out and the next one starts.

The reason this "works" is because rsync picks up right where it left off on a broken transfer. I believe restic and borg have similar behavior ...

What about a split or shadow repository? That is, have a host-specific local cache of the metadata from the main backup server, so that a snapshot backup can be made locally (say to an attached thumb drive or SD card), then have a process that sync's that up periodically when a stable network connection is available?

If this is a strong enough use case then I have a couple ideas on how to implement it (I'll throw it in the wishlist). Otherwise, currently if you have an interrupted backup the next one will still re-use the files already on the server, even if it is part of a partial backup.

I’d also be interested in recommendations that meet the above requirements.

My current work around for my main linux laptop is setting rclone to run with crontab at a time that I know i’ll be online for an extended period (evenings during long homework sessions). I’ve been doing this for several years and have yet to run into major show stopping bugs, and I frequently do checksum verification on my backups.

Yeah, that's sadly not very useful for laptops I manage - people tend to turn them on and off at random times and I had multiple cases where the backups just ended up in hosed state (because the tool got interrupted at a bad time) or just didn't backup for week because the laptop never ran for enough time without interruption after creating a huge file.

Have you looked at git-annex on an anacron-job?

https://joeyh.name/code/git-annex/

It lets you use git to back up large files. However, it's built for a git-sized number of files and not built for, say, your entire home directory.

resuming backup after network has disappeared for a while or after sleep is problematic and mostly impossible to do in an atomic way.

If you have enough local storage, I'd put a local backup process (restic/borg/bup), and then rclone/rsync your backup repository to remote storage.

If you don't have enough local storage, you can restic or bup to a remote server reliably as long as you can complete one scan. On my laptop with a 256GB SSD, a common restic scan with cold caches takes less than 15 minutes; an hourly backup is a few hundred megs, a daily backup is often close to 2 gigs, depends on how many files you have, and how many (and how much) of them change.

Alternatively, you can do an rsync-based snapshot.

There are multiple Windows and macOS tools that manage to do that - without writing my own bash scripts which can fail in corner cases.

I want to avoid writing them - last time I ended up with devices not backed up for months because the backup tools ended up in a partially broken state that caused silent failures.

Hence why I want to have a tool designed for this use-case and not a bunch of bash shell scripts in cron.

I use Syncthing, which just silently does its job, to the point that I forget it's there. I use that to get files to my Linux PC, where they can be backed up along with everything else.

I don't know if Syncthing has your bonus point, as I don't use mobile hotspots much.

Syncthing (and SaaS equivalents like Dropbox) don't really qualify as backup tools because as soon as the file is updated on one device it's updated on all devices. Paid services usually give you a certain amount of rollback to cover the "oops didn't mean to save/delete that" case but even that won't help for something like ransomware trying to keep you out of files. Of course many backup solutions also have the ransomware problem, like plain rsync+cron.

That's right, but replicating changes from multiple devices onto one that then runs a proper backup tool with versioning and remote storage seems simpler than trying to do backups properly on each separate device.

Arq does this really good, I know exactly what you mean. It's more or less fire and forget. And unfortunately it looks like you're not getting an answer.

Dropbox?

Reply on news.ycombinator.com