Not saying that Borg is necessarily the best solution, just that we should evaluate the important metrics.
To an extent, anyway. I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs. Unfortunately I didn't uncover this problem until I actually needed to restore, which extended an outage by about half a day. Having a vague sense of the tool's performance is a good thing, I think.
Yes, as long-time Tarsnap users, we got a very unpleasant shock a few months ago when our main server literally went pop one morning and we needed to do a full restore of everything to get the replacement up and running. It took many times longer than we were expecting to download all the data, and consequently caused us extra days of unnecessary downtime.
To his credit, Colin was responsive when we asked for advice and did suggest something we could try to make things a bit faster (which it did). But really, when you’ve just had a catastrophic failure, messing around hand-holding your backup tool just to get your essential data restored as quickly as possible shouldn’t be needed. Backup tools need to Just Work, quickly and reliably, every time.
Given that Tarsnap also seems to work out very expensive these days, it’s now in the category of software where I’m happy to have used it and it did do its job even if not ideally, but unless there are dramatic improvements on both counts in the near future I think we’ll probably be looking for another option.
No. I need a backup that consistently finishes before I close the lid to put my laptop in my backpack at the end of the day. Too slow -> no backup.
My multi-daily backups (always incremental in Kopia) take 3 minutes, with a working set of 4 million files.
Incidentally, some backup utilities run into unexplainable performance issues trying to delta diff very large files such as VM images. Rsync in particular sometimes stalls completely when trying to sync a 30GB VM image to a remote server.
You can schedule the backup to start some time before you leave at the end of the day.
disclaimer: I am the author of vimalin
The reason why I think it's fine is that it doesn't really try to do "delta diffs" at all. Instead is uses a rolling hash to cut the input file into variable length chunks and those chunks are added to the chunk storage, and then actual files are constructed from those chunks.
Obviously such copies should be done when the files are stable, so either not running or using a snapshot. (If using LVM, this can be arranged easily with any filesystem, except ones that don't like getting mounted multiple times..)
So while I guess size-related bugs could be there, there's no any fundamental property of making the task bigger as the files themselves grow bigger.
That being the case, I much prefer being able to re-image my home dir partition into a new laptop/ssd and get back to paying work, over to having to fiddle with individual VMs that only exist as partial backups.
Edit: I've actually considered the opposite of what you suggest, just doing a raw partition image for backup. SSD is really fast, ethernet is fast, and the runtime would be consistent. Just haven't got it out of me to try it in practice.
What do you test differently? What do you avoid changing? How do you avoid breaking things? How do you get enough beta testers when the costs of failure are so extreme?
As for restic, yes it's available on Windows, but has a serious speed degradation when datasets increase above 2TB, mainly due to chunks being 2-5MB, where Borg uses 400MB chunks. And we're not talking a 10% slowdown. It's to the point where prune operations on repositories >2TB can/will take multiple days to complete.
For clients, I evaluated multiple solutions, one was Duplicati (https://www.duplicati.com/), and while it appears to work well most of the time, it sometimes stops backing up for no apparent reason. Backuppc also looked promising, but doesn't (didn't ?) support source encrypted backups.
In the end i ended up with Arq (https://www.arqbackup.com/) for Windows and Mac clients. It works reliably, and checks all the right boxes.
For servers i was initially really hooked on restic as it allows backups over HTTPS instead of SSH, but the performance issues were a showstopper.