Hacker News Reader

Comment by bloopernova | Hacker News Reader

(Alternative product recommendation, please downvote/remove if you feel that isn't appropriate)

For Unix/Linux backups, may I suggest Borg Backup? It encrypts and does dedupe astonishingly well. It also works over SSH incredibly fast, and restores are via a mounted FUSE filesystem so they're easy to pick and choose what you need. It prunes really well too, and is a single executable so it's easy to distribute via Ansible/Puppet/etc. I've been using it for several years and it hasn't failed me yet.

If y'all want to laugh at my bash scripting skills, I have a backup script that sends backup status to a Zabbix monitor server at https://gist.github.com/anthonyclarka2/cef41d201dd5b890dae67... (I'd appreciate improvements to that script if anyone wants to critique)

View on HN

I evaluated Borg and Restic and found that both of them fall over once you get to (what I consider to be) production level volumes; in my case that's ~1 PB and in the range of a billion files.

Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

In what way do they fall over? Also, did you file bug requests?

They start taking unrealistically large amounts of time to deal with their catalogs. For instance, with a 500 TB dataset, Restic on my (128 GB ram) server took over 4 days to do its prune operation.

In that case, what do you use?

We're using Bacula, because in our case it's the only thing that works with our volume, and also works with tape (which is the only cost effective way we've found to archive multi petabyte datasets).

But I'm not very happy about it, because it's insanely overcomplicated for no really good reason, and because of it being file-based.

Backblaze has done some cost comparisons between LTO and cloud storage:

https://www.backblaze.com/blog/lto-versus-cloud-storage/

If you're interested, I'm doing experiments with another site with 500T to backup, where I added sampling and sharding to HashBackup (I'm the author).

Sampling allows you to do faster simulated backups to determine the best backup parameters to use. In his case, we determine that a very large block size - 64M - was the best way to backup his data.

Sharding automatically partitions the filesystem so multiple backup can run simultaneously to get backup speed in the 250-400 MB/s range.

It's more at the proof of concept stage, but having another large site to work with would be fantastic! A couple of the larger sites using HashBackup are EURAC (European Research Center) and HMDC (Harvard MIT Data Center)

Yeah, I read that Backblaze thing, and used their spreadsheet. Tape came out to be nearly 9x cheaper for us, given our large data set but extremely slow growth rate. Basically, we have a lot of data we need to protect from disaster, but it changes/gets added to very slowly, so capex (buy some tapes, put the data on tape, put the tapes in an offsite box and forget about them) is vastly cheaper than opex (pay by the month for hundreds of terabytes).

> Sadly, the only thing I've found so far that works at all at those scales is Bacula, and that is file-based -- i.e., if you have a gigabyte file that changes by one byte, it backs up the whole gigabyte again. Not ideal.

Would it possible to split those bigger files before doing backup (log files with log splitters, sql databases with incremental specialised backup tools like e.g. xtrabackup), or are these e.g. image files with different versions?

I really like the concept of byte level deduplication, but have often thought the price you pay for the space reduction might not be worth it considering today's network speed as well as storage sizes and prices.

Would be interesting to hear your experience concerning this!

I'm definitely looking into what I can do as far as getting logs rotated (and thus split) better, but a lot of what I back up are legacy medical systems where making any kind of change is extremely difficult both technically and politically.

Okay, I see. Thanks for the feedback!

A slightly hacky method might be an overlay filesystem (using FUSE or similar) that mirrors the underlying filesystem for small files but represents the larger ones as smaller units (so bigfile becomes bigfile.block0000, bigfile.block0001, ...). That way only the changed block would get transferred if you one-byte change is a modify or append rather than on insert or delete.

If the backup service is using file timestamps as the only key to refresh then it would have to store the last modification date and a good hash of each block, and when the date of the file being mirrored is updated scan each block to see if there is a change there and update the stored hash and timestamps accordingly. This would need to be orchestrated to reduce the risk of temporary corruption if there are several updates and the rechecking process coincides with a backup sweep (i.e. make sure you don't present updated dates for any block until you can present them for all needed).

For restoration, you either manually concatenate the parts or have an overlay filesystem that operates in reverse: showing the smaller block files as a single large unit.

You'd have to very thoroughly test the overlays and their interaction with the backup service before risking it on important data, so it might not be something you would genuinely consider...

Dude. That’s more than just “slightly” hacky. Points for the idea though.

Manic me has some wonderful ideas that are simultaneously excellent and terrible. I have to keep a careful eye on him in the day job! One of these days I intend to implement and collect a few of his musings under the banner "the ministry of silly code".

I can't say i've been testing with PB sizes, but for my "meager" 8TB backup, Borg works well. A daily backup & prune operation takes less than 20 minutes.

Restic falls over as soon as you cross 2TB sizes, and prune operations are painfully slow. Backing up 8TB took 4 weeks with Restic, and i gave up waiting for prune to finish. It was crossing the 24 hour mark, making it unsuitable for daily backups.

Yeah. I back up a few terabytes a day with no problem in Bacula.

For bash scripting: have you seen shellcheck? It's a very solid first-pass for any script, and the codes it emits have excellent documentation: https://www.shellcheck.net/

Yes I have, I made sure it passes shellcheck before posting it here :)

I do sort of wish that Visual Studio Code had the "auto-fixes" already built for shellcheck's error messages. I should probably create those and make them available somewhere.

I don't see anything wrong with that bash script. It's well formatted and very simple, there aren't any functions or anything, just a single switch and two conditionals. Not much to optimize.

Simple and explicit is better in shell scripts than being fancy.

Thank you, I really appreciate the feedback.

Borg is hands-down the best backup utility I've used. I also use Borgmatic for some added niceties around checks/notifications/etc.

Does anyone know how it compares to Restic? That's what I've been using and have been pretty pleased. It seems like the main difference is that restic supports integrations with other services, whereas Borg seems uses SSH for its connections

Borg is also big on deduplication

Deduplication is one the things that drove me to Borg: it has the speed and space saving of an incremental backup but every single backup can be used as it was a full backup.

It is also very easy to setup a pruning policy for old backups, so that you can say that you want to keep one backup per day for 90 days and after that only one backup per month for 2 years.

The dedupe in Borg is downright magical. I know I lose some space by keeping my hosts in separate repositories, but even then, I get insane space savings.

Restic looks pretty interesting, I wasn't aware of it, thank you. I don't know how Restic can achieve its goals, since AFAIK you need a binary running on the server to diff/check data/verify restores/etc without transferring all that over the network.

Borg runs a binary on the server to do this, that's why it can't easily back up to S3 et al.

IIRC Restic keeps indexes of data it uploads to configured backends. I think it caches that index locally, but it can also download it from the remote backend to compare.

I've used both Borg and Restic pretty heavily. I settled on Restic because it was much lighter on resources. A few of the servers I manage have small amounts of ram, but large amounts of data. Borg would often get oom-killed, but Restic wouldn't.

Restic also starts uploading changes as soon as it sees them, whereas borg has to scan the entire list of files before it starts uploading changes. Unfortunately for multi-tb datasets with millions of files, borg would take hours just to scan all of the files.

Pruning old archives also worked a lot better in Restic compared to Borg.

If you're happy with Restic, I'd keep sticking with it.

Interesting. I had deployed restic to my fleet of 100 servers around a year ago to backup to a common Backblaze B2 destination, and rolled it back after running into a production service getting OOM killed because of restic memory use.

That's interesting, did you have a lot of small files or large files?

I don't know if it's an option for you, but have you considered something like rsync from each server to a central backup server and then running restic/borg/whatever on that server? Sometimes that works out nicely too, and would be less impactful to other services on the machine.

What did you switch to instead of restic, if you don't mind me asking?

I've been quite happy with Restic, doing always-incremental encrypted backups to S3.

Restic also works on Windows, whereas Borg doesn't (yet), if that's a consideration.

I use it on Windows via the Linux Subsystem of Windows 10. It's not a pure native solution, but it works quite well for me.

Is "it" Borg or Restic here?

it = Borg

It works very well under Cygwin, so no problem, if you can install that.

One big disadvantage of restic at the moment is that it doesn't do compression (https://github.com/restic/restic/issues/21).

I was using restic and switched to borg because of that and could save hundreds of gigabytes.

IMO compression in backups is overrated. (I run Relica, a backup service built atop restic: https://relicabackup.com)

Most large files are already in compressed formats: mp4, zip, tgz, jpg, etc.

What you _really_ want is deduplication (which restic does), so that chunks are only stored once, even if multiple machines sharing some data back up to the same repo. Dedup saves me about 30% per backup repo in my own experience.

Not saying compression wouldn't be nice (to have done properly, it is somewhat at odds with encryption), but I'm just saying I don't think it's worth making it a #1 deciding factor.

I agree that compression is overrated for most backups. It'd help with a lot of text-oriented data, but honestly that takes up so little space compared to a vm image or something like that.

If I was backing up a single computer, compression would matter more to me. But if I'm backing up 10s or 100s of servers, there's going to be a lot of duplicated data that would make a much bigger difference.

You can always back up onto a compression enabled file system to further save space + dedupe with rustic.

Borg is a fantastic tool but the time it takes to start and parse the files req on a large system negates the benefits of compression for me unfortunately.

Encryption totally defeats any raw compression. It would need to be inside the blobs themselves.

I recently launched a backup service for Borg – https://www.borgbase.com. Also big on monitoring (email, pushover, webhook), as this is a common concern for backups.

It just lacks terminal-based registration. :-) You could use the GraphQL API to manage everything from the command line if you really wanted.

This looks pretty cool, well done! I use rsync.net and am very satisfied, but yours looks great as well. I'm not a big fan of the initial price hike (the first 100 GB is 2 cents per GB instead of 1), but it looks like the service is well worth it.

This looks good, and competitively priced too! What's the backend cloud storage?

Just guessing here but probably Backblaze b2? It has a similar pricing.

Currently it's on RAID boxes, but I'll move to CephFS soon. The data volume is getting too big to copy it around. Ceph allows you to nicely expand your storage over time and account for some failures on the way.

Borg needs a "smart" backend with a filesystem, so it can't use only object-storage.

I've been using borgbase for a couple of weeks and am very happy with it.

Is there a limit to how many times I can download my backups?

No limits. Downloads are unmetered and free, as long as they look reasonable. Once you download your full backup every hour, I'll ask you for a reason probably.

You should make a habit of setting

  set -euo Pipefail
  if "${DEBUG:-false}”; then set -x; fi

The -o Pipefail is irrelevant in this case, as you're not using any.. but better it's there in case you ever extend the script.

As others have pointed out: readability comes right after correctness in shell script, do keeping it simple as you have done is always a good idea.

I hope you see this reply, I am sorry for it being so late:

Do you have a link to a good explanation of why you'd set those? The "set -e" seems to indicate that the script would fail immediately if the borg backup job fails, which would prevent the Zabbix "send" command from alerting the monitoring system.

Thank you for your feedback, it's given me more to learn and think about!

There are quite a few examples around, just google the parameters 'set euo pipefail' (first hit [0])

you're right that just putting the set options on top of your file would be a bad idea. you need to write your script to actually account for these errors, making possible exitcodes entirely transparent

    1 #!/bin/bash
    2
    3 set -euo pipefail
    4 if ${DEBUG:-false}; then set -x; fi
    5
    6 function random_exitcode(){
    7         return $(( RANDOM % 5))
    8 }
    9
   10 function handle_error(){
   11   case "$?" in
   12     1)  echo "handling exitcode 1!";;
   13     2)  echo "handling exitcode 2!!";;
   14     3)  echo "handling exitcode 3!!!";;
   15     *)  echo "encountered unexpected exitcode: $?"; exit 2;;
   16   esac
   17 }
   18
   19 random_exitcode || handle_error

or, if you don't like functions:

  random_exitcode || LAST_EXITCODE=$?
  case "${LAST_EXITCODE:=0}" in ...

[0] https://coderwall.com/p/fkfaqq/safer-bash-scripts-with-set-e...

Many thanks, I very much appreciate the effort you put into your reply, and your followup.

I would follow up with a suggestion to also consider restic, which is very similar to Borg Backup -- except it doesn't have the restriction that the server needs to be "clever" (you can upload to S3 or any dumb blob store).

I just run borgbackup and backup to a local folder, and rsync to my home nas if I am connected to my home network.

If I need to force a sync over the internet, there is a small shell script that would rsync the backup folder to the home nas.

This allows me to run hourly backups even if there is no network connection (or limited bandwidth), and then just auto sync when I get home, or force a backup when I get a good connection.

I am using zfs send and it works like a charm. Once you have configured filesystems for only data, you can use dedup on receiving system.

The encryption in Borg is weak but otherwise it works well if it fits your use case.

How so? When I glanced over it it looked trustworthy enough.

Can you give more details on how the encryption is weak?

https://borgbackup.readthedocs.io/en/stable/internals/securi...

> When the above attack model is extended to include multiple clients independently updating the same repository, then Borg fails to provide confidentiality (i.e. guarantees 3) and 4) do not apply any more).

Thanks for pointing this out.

The first entry on the Borg FAQ [1] is “can I backup from multiple servers into a single repository” and the answer is basically “yes, but it might be slow”. Seems to me like it should say “not really, it's insecure and might be slow”, plus the tool should make it clear you're doing something unwise if you try it anyway.

I've always shied away from shared backup repositories anyway, because it usually means data from host A can be read by host B, and that's just not what I usually want. The deployment instructions appear sane to me. [2]

[1] https://borgbackup.readthedocs.io/en/stable/faq.html#can-i-b...

[2] https://borgbackup.readthedocs.io/en/stable/deployment/centr...

I'm not the person you responded to, but here are my observations:

1. Borg's authenticated encryption is a composition of AES-256 in CTR mode and HMAC-SHA256 (or alternatively, Blake2b256). This consists of two distinct constructions. Most cryptographic vulnerabilities are introduced in the combination of distinct primitives and constructions. While combining AES-CTR and HMAC is a way of doing authenticated encryption, it's not the safest way. As a specific example - in order to avoid introducing a nonce-misuse vulnerability, Borg needs to implement specific logic beyond the construction to ensure that the CTR counter (containing a randomized nonce) is not reused. This kind of overhead is dangerous because it adds a lot of room for a developer to make a fatal mistake.

It would be much better to use a dedicated authenticated encryption scheme, like AES-GCM. AES-GCM is a more complex construction, but it also has very convenient interfaces and language bindings which obviate the need for developers to interact with raw primitives and constructions entirely. You don't need to implement AES-GCM so much as call its seal and unseal operations from your language of choice. There's far less room to footgun yourself here. The even more modern method would be to use something in the ChaPoly family. For example, XSalsa-Poly1305 goes one step further than AES-GCM by using an extended nonce. ChaPoly constructions also have ubiquitous language bindings and easily used interfaces to safe implementations.

2. Borg uses OpenSSL directly. This is not great because libcrypto exposes raw primitives directly to the developer. To its credit, Borg's documentation does provide a few reasons why its use of OpenSSL is trustworthy. But those reasons are more to do with TLS implementations than cryptographic constructions. As with above, working directly with raw primitives provides a lot of room for implementation errors and avoidable security vulnerabilities. It's generally better to abstract away your cryptography implementation to a known-good, trustworthy source which you can just call via a straightforward (and opinionated) interface. For example Borg could use NaCL instead, which would provide XSalsa-Poly1305 encryption out of the box. I hypothesize they'd eliminate a few hundred lines of code with NaCL instead of OpenSSL.

That being said, I don't know if I'd call Borg's encryption "weak." It's not ideal, and I don't personally trust it. But it's not like they're using AES in ECB mode or Mac-Then-Encrypt CBC mode.

The AES/MAC composition itself in Borg is done correctly, the problem is the way Borg uses the composition: with a single AES/MAC key for the entire repository which a) never changes b) cannot be changed and c) IVs/nonces used with those keys are primarily tracked by server-side, i.e. untrusted, state. This means that using multiple clients with one repository is never secure in Borg.

> It would be much better to use a dedicated authenticated encryption scheme, like AES-GCM.

Actually, in the scheme Borg uses, which is vulnerable to repeating IVs/nonces, using AES-GCM or Chapoly (or indeed any Wegman-Carter authenticator) would not only delete confidentiality, but also authenticity, i.e. it wouldn't just disclose plaintext to the attacker, but allow an attacker to potentially change your backups [1].

> That being said, I don't know if I'd call Borg's encryption "weak." It's not ideal, and I don't personally trust it. But it's not like they're using AES in ECB mode or Mac-Then-Encrypt CBC mode.

I'm calling it out as weak because it is an insufficient and poor design. If you say encrypted today it should better be up to me throwing my encrypted data up on a random cloud server and be sure that it's actually encrypted. This is not the case with Borg; partial credit for getting some aspects of the construction right (e.g. EtM, separate keys, properly salted master key encryption key derivation) isn't worth much when it fails to provide confidentiality in not-at-all unreasonable, practical usage.

[1] Probably not because by necessity Borg uses a separate HMAC over the plaintext for deduplication, and the manifest has a third layer of HMACing due to protocol issues, so it should be impossible to just change stuff even if you break the ciphertext authentication.

Can you walk me through how AES-GCM or ChaPoly breaks authenticity and confidentiality, but AES-CTR doesn't? I'm not following you.

Moreover, I'm pretty sure Borg's developers are actively considering ChaPoly for the future.

GCM/Poly1305 require unique nonces, so if they are repeated, the authenticity is broken. HMAC doesn't need nonces, so authenticity is preserved as long as the key is secret.

I'm aware of that, and mentioned that point in my original comment. What I'm not following is why Borg can't safely use AES-GCM or ChaPoly instead of AES-CTR + HMAC.

Again: the team is actively considering using AES-GCM and ChaPoly in the future. I don't see anything intrinsic to either that preempts their use in Borg.

Yes, in a different construction there would be no problem. And there have been plans since at least 2016 to replace/augment the current construction with something that uses a master key to derive per-chunk encryption keys; it just never has been implemented.

IIRC AES-GCM was kinda low on the list with a preference for just using Chapoly, because Chapoly just works and is also secure on any processor, unlike AES-GCM, which is very nasty to implement without hardware support for the arithmetic over GF(2^128).

Reply on news.ycombinator.com