Snebu – Simple Network Encrypting Backup Utility

Can you mention what you've tried so far? I'm happy with borgbackup, but I don't know if it matches your bullet points.

Still, I'd be surprised if mainstream backup solutions corrupt your backups in case of sleep / disconnect. That would be a pretty big bug. Borgbackup also handles resumes IIRC. Your third point is unclear, you'd have to try in practice. But if throughput * time awake < size of compressed deduplicated data, you obviously cannot expect your backup to complete. I guess you just rephrased your second point.

For the last point, personally I think I'd take care of mobile hotspots / slow networks with a script wrapper rather than expecting the backup software to handle this.

I'm late to the thread but borg or restic or any of the attic/bup/Xbackup lineage will meet your needs. I haven't tried the GUIs but you can schedule it to run with anacron once per day and it'll run when the laptop first turns on. You can sleep 10 minutes before running to give yourself a chance to get connected. Because of how the repositories work, if a backup fails due to list connection the next one will use that partial data so the second backup will have less to transfer. NetworkManager knows the metered state of the wifi and so that can be read in a shell script to decide whether or not to run the backup. You may want to whitelist known good wifi networks instead.

Perhaps someone could take the excellent Deja Dup GUI, which is a front-end for the technically inferior duplicity, and have it shell out to borg instead.

"Can anyone recommend a Linux (preferrably cross-platform) tool like this which is appropriate for use in personal laptops?"

I think you should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... the remote has no access to the data.

The borg website is here:

https://borgbackup.readthedocs.io/en/stable/

and a good description of how it works and why you should use it is here:

https://www.stavros.io/posts/holy-grail-backups/

What about a split or shadow repository? That is, have a host-specific local cache of the metadata from the main backup server, so that a snapshot backup can be made locally (say to an attached thumb drive or SD card), then have a process that sync's that up periodically when a stable network connection is available?

If this is a strong enough use case then I have a couple ideas on how to implement it (I'll throw it in the wishlist). Otherwise, currently if you have an interrupted backup the next one will still re-use the files already on the server, even if it is part of a partial backup.

I’d also be interested in recommendations that meet the above requirements.

My current work around for my main linux laptop is setting rclone to run with crontab at a time that I know i’ll be online for an extended period (evenings during long homework sessions). I’ve been doing this for several years and have yet to run into major show stopping bugs, and I frequently do checksum verification on my backups.

Have you looked at git-annex on an anacron-job?

https://joeyh.name/code/git-annex/

It lets you use git to back up large files. However, it's built for a git-sized number of files and not built for, say, your entire home directory.

resuming backup after network has disappeared for a while or after sleep is problematic and mostly impossible to do in an atomic way.

If you have enough local storage, I'd put a local backup process (restic/borg/bup), and then rclone/rsync your backup repository to remote storage.

If you don't have enough local storage, you can restic or bup to a remote server reliably as long as you can complete one scan. On my laptop with a 256GB SSD, a common restic scan with cold caches takes less than 15 minutes; an hourly backup is a few hundred megs, a daily backup is often close to 2 gigs, depends on how many files you have, and how many (and how much) of them change.

Alternatively, you can do an rsync-based snapshot.

I use Syncthing, which just silently does its job, to the point that I forget it's there. I use that to get files to my Linux PC, where they can be backed up along with everything else.

I don't know if Syncthing has your bonus point, as I don't use mobile hotspots much.

Arq does this really good, I know exactly what you mean. It's more or less fire and forget. And unfortunately it looks like you're not getting an answer.

Dropbox?

For a simple tool, use rsnapshot ( https://rsnapshot.org/ ).

Pull system, which rsyncs (over ssh) the data from machones to folders on a server, then, on the next day, creates a hard-link copy of the backup folders, to get a separate "yesterdays" version, and rsyncs over the "todays", to get the new "todays". ...plus some scripts to do a weekly,and monthly versions and to keep the number of backups at a desired number (config).

Only downside is, that it needs root access to rsync whole filesystems, and that you need to manually set the excludes for each machine.

The "simpleness" in Snebu comes from using standardized tools such as find and tar, file storage using LZOP, and the backup catalog in a standard SQLite3 DB, allowing manual recovery of files.

However Restic and Borg do have the advantage of writing directly to cloud backups or "dumb" backend data stores such as sftp targets, or API-accessed cloud targets (although Restic doesn't compress and Borg has issues with multiple hosts going to the same repository, they do still have their use cases). Whereas Snebu is more of the use case that would be solved by more complicated to manage tools such as Bacula, Amanda, or commercial backup solutions but with lower setup and administrative requirements.

To address the question in your edit:

When the key file is created, you are prompted for a password to encrypted the private key. That password is used (along with the public key) to generate the HMAC key which gets recorded in the keyfile and is used to sign each file in the backup. The encrypted private key is sent to the server, and gets sent back when you do the restore, but the HMAC secret key stays on the client.

Since you need to type in the password to restore the data, the HMAC key gets re-generated at restore time.

The reason the passphrase is hashed with the public key (again using an HMAC-SHA-256 hash) is so that if the same passphrase is used on different hosts with unique encryption keys, the HMAC key still ends up being unique.

So a compromised key file on the client will compromise the secret HMAC key, but it won't compromise the encrypted data. Which is better than having a single symmetric encryption key laying around for both encryption and authentication (encryption being the more important item). And the key is tied to the backup snapshot, not to the repository so each client can have their own keys.

Further information is at https://www.snebu.com/tarcrypt.html which gives the inner workings of tarcrypt (some of the description of the tar file extensions are dependent on a working knowledge of the basic tar format and the PAX extensions, which are documented in the GNU tar documentation).

Inital commit of snebu was from january 6th in 2013 while restic's alpha release was in 2015. So snebu is older. That probably also explains why sha1 was chosen in the first place.

Glad you asked. First, Snebu goes back about 8 years, although the encryption code was developed more recently. That also explains thy it the hash is sha1 (at the time the thinking was "it is good enough for Git"). Right now it is kept for unencrypted backups for backward compatibility with previous versions. But it is something I would like to address -- possibly by using sha-256 (or truncated sha256) or make it user selectable on new repositories. This will be on the next version, didn't want to make too many changes at once. However encrypted backups do use a sha256 HMAC.

Compared to Borg and Restic, by making the system server-based, it does limit things like being able to add direc-to-cloud backup, but a future capability may be added to efficiently replicate a Snebu server backup to cloud. However what it gains is better manageability, especially if you are backing up a number of hosts to one repository. In my instance I'm backing up about 50 hosts with 14 daily, 6 weekly, and 12 monthly snapshots.

Compared to Restic and Borg, it uses public key encryption (so you don't need to leave a sensitive symmetric key on your clients), but with HMAC-256 based hashes. So an attacker would need both your public key and the HMAC secret (only a one-way hash of the secret is stored on the server). The client reconstructs the HMAC key using the public key and the passphrase for the private key. Oh, and you can use multiple keys -- a client specific key, and a backup key (or more). Deduplication will work across all clients that share the same set of keys (the HMAC of all keys is used for the file storage hash). But for this attack scenario (forged data), you will need to keep the client key file safe and stored with appropriate permissions, and you can use a different key (and lose cross-client deduplication) across clients that are in different trust zones.

As for the SQLite DB size -- on one of my installs the DB is taking 50 GB, and is storing over 1500 backup snapshots, from 73 hosts (CensOS and RHEL some with Oracle DBs), with a total of 4.5 TB of disk storage used.

The SQLite DB is critical as it stores which file belongs to which directory path on which host -- so it should be backed up independently (at the end of the backup script I use sqlite3 commands to do a dump of it once a week and run a vacuum command). Todo: add these maintenance commands to the Snebu binary to limit the need to run the Sqlite3 CLI.

Other features unique to Snebu -- you can have the server "pull" backups without installing an agent on the clients (the "tarcrypt" will still need to be on the client if encryption is needed). You can have the clients "push" data to the server using a restricted user account that allows backups but not deletes or restores, and have a separate restore ID for example. You can give different administrators IDs on the the backup server access to different groups of hosts in a look-but-don't-touch, etc.

Another item I like -- let's say you tell it to expire all monthly backups older than 6 weeks. But then if the host hasn't successfully backed up in more than that time (say it has been taken offline), there is a default to keep the most recent 3 backups in a retention schedule, so you won't accidentally lose all you backups in this case (the "preserve" number can be adjusted via command line parameters).

It uses standard "tar" and "find" commands on the client. If you have something like a database, then you should exclude the dbf files and uses the database's tools to put the DB in hot backup mode (using a snebu-client plugin script -- there is a template script in the manpage for snebu-client-plugin(5).

You can also create a plugin script that performs a disk snapshot (using LVM snapshot or similar), then mount and backup from that.