Hacker News Reader

Comment by rsync | Hacker News Reader

We[1] built borg into our environment[2] as soon as it was stable, release software. In the years since, it has (ironically) supplanted use of rsync as the de facto standard that our users back up to us with.

As one of our users have said[2], borg is "the holy grail" of backups as it does everything rsync always did, and produces remotely encrypted backups that the provider has zero insight into.

It also does not have the inefficiencies that the older, duplicity software has.

If you are willing to go without (borg specific) technical support and do your retention with borg instead of our zfs snapshots, there is a special, discounted rate available.[4]

[1] rsync.net

[2] https://news.ycombinator.com/item?id=17408624

[3] https://www.stavros.io/posts/holy-grail-backups/

[4] https://www.rsync.net/products/borg.html

View on HN

Borg is definitely not the holy grail of backups, unless one has very loose requirements.

Desktop backup solutions have typically a set of functionalities to compare against (block-based, safe on untrusted repositories, with realtime backup, compression-efficient, and optionally, with a functional GUI), that show how open source solutions are strangely lacking, each in a way or another.

> produces remotely encrypted backups that the provider has zero insight into.

If you're implying an untrusted repository, this is not entirely correct. From https://borgbackup.readthedocs.io/en/stable/internals/securi...:

> in a multiple-client scenario a repository can trick a client into reusing counter values by ignoring counter reservations and replaying the manifest

The last discussion on HN was actually significantly more critical: https://news.ycombinator.com/item?id=18952839.

Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed.

> Borg also has no efficient compression, as it doesn't use multithreading. There are multiple GitHub issues on this subject; I've just checked, and they're closed.

Depending on your use model, this may or may not affect you. Whether you use multiple threads to compress a single stream or not probably matters a lot less if you have 15-30 backups going on at the same time.

Nice to see someone offering public borg hosting! I have two security questions:

(1) Are you doing any hardening of the borg remote side? The server is pretty complex and it has lots of code paths which can be exercised by a potentially untrusted client.

(2) Is there a way to protect compromised clients (like a cryptolocker)? borg offers "append only" mode, but it does not allow pruning (obviously). And trying to run "prune" using separate trusted connection will commit untrusted changes as well.

   inefficiencies that the older, duplicity software has.

As much as I love duplicity, it does sort of suck when you get into larger numbers of files and gigabytes of data. It's so darn slow sometimes it's almost unusable. Good to know borg is better, I've been meaning to check that out!

It's incredibly slow (backing up 1.5TB took more than a day), but the built-in support for PAR2 makes it worth it (for me).

Sadly, it seems like borg has no built-in support for any kind of redundancy [1]

[1] https://github.com/borgbackup/borg/issues/225

Duplicity and Borg use two very distinct technologies under the hood - duplicity does differential partial backup archives (which all require a full backup AND a full list of all previous differential archives) and Borg uses deduplication. So yes it is much more efficient in many ways, not simply because it is newer and somehow is more efficiently coded, it is so because it uses different technology and backup model.

I've been using borg to backup small things, like schoolwork, config files, mail, and code, to rsync.net for ~1 year now, and can vouch that it's pretty simple to setup. If you've got a simple use case like I do, the example config in the borg tutorial is pretty much all you need. I've got it set to automatically run as a cron job, and I haven't had to touch it except to occasjionally do sanity checks that everything is being back up correctly.

If your customers back up multiple servers, are they still under the same rsync.net account and could they see each other's data?

This is one of the issues I aimed to solve with BorgBase.com[1]: every single repo is its own backup user and can't see other repos. This separation allowed to add some Borg-specific features, like append-only mode, monitoring for stale backups or using a specific Borg version.

1: https://www.borgbase.com

rsync.net has "subaccounts" which allow you to control side-by-side backup directories using standard UNIX permissions and ownerships. That is, if the default configuration is not exactly what you'd like.

Subaccounts even get their own .ssh/authorized_keys file.

It's all very unix centered and command-line focused. There's no web interface for features like this.

I see that you offer "append only mode" as a core feature. How do you deal with pruning of old backups in this case?

Regular borg only offers two options: either pruning does not decrease space used at all; or pruning permanently commits changes, negating any security advantages. Both of those seem bad enough to prevent offering "append-only" commerically?

I use borg with both rsync.net and borgbase.com. Both are great.

Why aren't you price competitive with Google Drive, Amazon Drive, Wasabi, etc? Their rate is ~$0.005/GB/month once you start buying +1TB

$5/TB is getting pretty common now (Backblaze B2 comes to mind) and if you roll your own you can get ~1.6EUR/TB from providers like Hetzner (https://www.hetzner.com/dedicated-rootserver/matrix-sx).

How do you arrive to the 1.6€/TB/month?

In your link the cheapest option is 76€, with an 82€ setup fee.

EDIT - I see it now, had read 4 TB for the cheapest option, but it is actually 40 TB !!

Too much for my typical home user needs, though :-)

How reliable is Hetzner? Is there major downtime, enough bandwidth (Around 100Mbps minimum during congested times?)

In my experience pretty reliable. I've not seen major downtime and bandwidth will depend entirely on routing to you in particular (FWIW my ISP peers with Hetzner directly and congestion hasn't been an issue, I can typically saturate my 1Gbps connection at any time of day).

> borg is "the holy grail" of backups as it does everything rsync always did

I tried borg for some time for full backups from / and there were 3 things I didn't like about it:

1. Mounting and navigating a snapshot was extremely slow. Like wait a few seconds for any `ls` to finish.

2. It needed the encryption password for many things I think it shouldn't.

3. The resulting backups are very opaque.

I've since switched to making my backups by rsync'ing to btrfs subvolumes. I snapshot the previous backup and rsync the differences to the new one. I get to navigate each snapshot with the same snappiness as any other directory in the filesystem. btrfs can also instantly tell me which files differ between 2 snapshots, no matter how huge they are. I don't remember all the options that borg offers on synching, but I doubt it has the same breadth of options as rsync. So for me, rsync + COW filesystem > borg.

I am a little concerned with my use of btrfs, but I'm planning to do the same thing using ext4's reflinks for some disk and filesystem redundancy.

How do I get rotation schemas to work with borg? Say I want to keep the last 3 daily snapshots, one per week for the last 4 weeks, and one per month for the last 6 months...

I understand this is kinda hard with "zero knowledge" encryption, but it is possible and not a wild feature to request from the self-named holy grail of backups soft.

The prune option is what you’re after there

https://borgbackup.readthedocs.io/en/stable/usage/prune.html

Yes, but which server will run the purge command? It cannot be the same server as the one you are backuping, since one of the goal of the backup is to protect the server if it is compromised.

You can run command from inside your backup script, from the client. It needs the encryption key anyway to add backups, this way it is just one extra command that does all the pruning (quite quickly as well). And you don't have to worry about full/incremental backups, nor really about any "rotation" or managing them manually, it's all deduplicated and Borg just keeps X number of daily/weekly/whatever snapshots (like duplicacy, like restic, like bup, etc. etc.).

Yes, I got this. But if your server is compromised, then the attacker can erase your backups, which ruins the point of doing backups in the first place. This is why there is an append-only mode.

Don't rely on append-only mode in Borg, it does not work really as advertised. Every transaction in that mode is recorded in a log, which if you are very careful you can go back into (but even reading or monitoring it is a PITA, the "interface" requires you to carefully track id's of files and check with dates etc).

And being careful is difficult, because in Borg, once you do any other backup command from a full access account (for example to do pruning) - it will automatically, no warning, go through the log and apply it. You should really really read up on that functionality first before relying on it, the way Borg has implemented it is close to anti-feature.

Regarding compromisation of strictly the server itself, I believe there are commands to check the state of the repository? Isn't that enough?

I don’t know the best method, but my method is to have a locked down backup box in the same DC as the machine(s) it’s backing up.

It mounts the machine to be backed up and runs Borg on that.

The backup machine does need locking down in of itself but that’s a lot easier to do than locking down something public facing

What I do is to have two different SSH keys. The one that is used by the backup script is --append-only. The one that I keep locally on my notebook has full access, so I `borg prune` from there when I see the disk usage reaching critical levels.

If we are still talking about rsync.net, and an attacker gains access to your account and deletes your borg archive, you can still restore your backup from the rsync.net ZFS snapshots.

The snapshots are not deletable even with your full credentials.

It says 1.5c per GB month but then starts at 100GB minimum. Any options for users who would want to use less storage than that?

Generally it's not worth using a credit card for something less than $1.5 per month. Various cloud services will give you 100GB for free.

Reply on news.ycombinator.com