BorgBackup: Deduplicating archiver with compression and encryption
borgbackup.org
borgbackup.org
Update: yet another take on basically the same approach, also as a self-contained binary: https://github.com/kopia/kopia
It contains everything needed (python, python libs, other libs) except glibc and related libs which must match the OS / kernel and therefore are intentionally not included.
restic fails on repo mounted via CIFS/samba on Linux using go 1.14 build https://github.com/restic/restic/issues/2659
Of course there are some cifs issues in the borg tracker, too, but most of them seem to be based on unreliable network connections - while the restic problem seems to be related to golang details.
With backups I would rather not trust a tool that has basic problems like accessing files on a network drive, but this is just me.
Kopia is too new for that state.
A few highlights (for me at least):
- They have implemented support for object locking and versioning to prevent a compromised set of S3 keys being used to erase backups. There seems to be a pull request outstanding to dynamically shift forward the compliance hold each time an object file is "deemed still used", but very few s3 bucket based backup tools seem to really think about resisting deletion and documenting best practice. I like to use minimum privilege and deny DeleteObject permission, but using versioning and object locking policies seems to be an interesting potential solution.
- Inclusion of (optional) Reed Solomon error correction codes in backup objects.
- a CLI verification option that will test a given % of your backup objects (at random) every time it's run, to give you opportunistic random verification to try find issues.
- reasonably well documented CLI options for recovering data were anything to go wrong - as the developers mention, there are a wide range of ways to recover data. Interestingly, they appear to have actually implemented these, rather than leave them as theoretical. For example, CLI commands to interact with indexes, manifests, blobs, snapshots, etc. You don't need them normally, but having these available lets you look around a simple backup and understand the format and gain confidence in it, and that the tools work.
The only feature it seems to me to be lacking is asymmetric encryption of backups, so you can keep the key needed for recovery away from the host you are backing up.
> - a CLI verification option that will test a given % of your backup objects (at random) every time it's run, to give you opportunistic random verification to try find issues.
FWIW Restic also supports this - see restic check --read-data-subset (https://restic.readthedocs.io/en/stable/045_working_with_rep...)
Also Restic doesn't have any problems with buckets with deletions disallowed as long as you allow them for just the `locks` directory. A policy I used for one of my targets looked like:
{
"Statement": [
{
"Sid": "AllowAdditions",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::BUCKETNAME"
},
{
"Sid": "AllowDeleteLocks",
"Effect": "Allow",
"Action": "s3:DeleteObject",
"Resource": "arn:aws:s3:::BUCKETNAME/locks/*"
}
]
}
Then pruning could be done once every few months. The egress fees could be high though.
Given that all of the options being discussed here look technically better and in some cases more than an order of magnitude cheaper than other popular backup services and software discussed on HN in the past you could afford to run full redundant backups with multiple combinations of software and backing storage and still have more options for a much lower price than a few years ago.
https://help.backblaze.com/hc/en-us/articles/217667478-Under...
So I settled on borg. I use it for offsite backup. I did restores of individual files as well of whole snapshots after broken hard drives. There even was a time I used it with WSL to backup my parents data.
Except that I tested them and they didn't. I hadn't "lost locally cached metadata". And it didn't just scan all the source files again, it re-transferred them over the network. All 4TB of it! That's why it took so long. (At that time the sever was still local, because I wanted to make the initial run not over the Internet. So there wasn't even added latency.)
I'm sure it was a bug, maybe even in combination with OpenBSD, which moste likely is fixed now. As I said it was years ago. I compared and was more happy with borg. Which stood the test of time and saved my ass multiple times.
`restic backup -v -r <location>/<repo_name> <source_dir>`
first `cd` to the parent directory of the `source_dir` to avoid too many nested directories in the repository and conflicts in the mounting point path (so that the file change detection of restic works when mounting the drive to be backed up at another point)
I suppose the latency of the remote server was too high.
The performance issues you experienced may have been resolved.
Is that also true for Kopia?
We rely on this in some places in fact. We have a couple of redundant hosts all serving the same file set. This deduplication allows us to push backups from all systems without coordination and only the first backup of the same file set requires storage beyond metadata.
> Another difference between the two programs is related to deduplication. Borg is designed with the assumption that each machine being backed up will use its own repository. Letting multiple machines backup to the same repository can impact performance, and simultaneous backups from different machines to the same respository are not supported.
So I should have said "Borg has limitations" instead of saying the support is absent for deduplication across machines.
When you say "without coordination", does it mean simultaneous backups from several machines to the same repo are possible?
It is, with a bit of (other) coordination. Basically, only one process can actively write to a borg repository. Once something is writing to the borg repository, the repo is locked and nothing else can write to that repository.
However, you can easily stagger backups from several hosts into the same repository - node 1 writes at 09:00, node 2 writes at 10:00 and node 3 writes at 11:00. This works without problems, it just needs some monitoring for quickly growing backup sets in case your timing goes awry. You can also configure a wait timeout, how long a borg process will wait to lock the repository, but that will require some tinkering with SSH heartbeats to avoid connection timeouts, as borg won't talk over the wire while waiting.
As the documentation correctly states, this slows down the backup writing process a bit, because each node has to synchronize its local chunk cache at the start of a backup or a prune. This wouldn't be necessary if each node had their own dedicated backup repository. This can take 5ish minutes on our large repos (4TB+) and usually takes less than a minute for our smaller repos (<1TB). It's not really a big deal imo.
However - and we made that mistake earlier - something like `borg check` and other commands become really slow if you have some 15TB - 20TB+ borg repository and borg repos that large become really messy to manage. It works, but some things work at a glacial speed on a good day - while locking out all other backups. That's why we tend to group our borg repos by dataset and storage type ("All databases supporting app X backup into the app-database-X repository"). This way, all databases are in that repository even after failovers or switches to geo-standbys and it's easy to setup a restore procedure. Hoqever the overall size of the repository is somewhat limited and manageable.
This may make sense, depending on your threat model.
And yeah, we tend to partition our borg repos along functionality and needs of restore. For example, database backups and file store backups are split into different repos, but several file store hosts in the same cluster all write to the same borg repo. After all, if something allowed to compromise one file store host, it will most likely allow compromise of all identically setup file store hosts in that cluster.
And on the other hand, in this way, we don't have to think about the backups if one of the file store hosts goes offline - the other two will just continue writing backups. This would be something that's really easy to forget and could bite very badly.
Borg can can run in WSL but has seen limited testing under such, per their own docs.
That's bad if you want to use it
1) on NixOS (I don't want backup configs laying around in `~/.config`). As Indy famously said: "That belongs in a Nix expression!"
Edit: On a second note: Why have mandatory config files at all? I don't have an issue with having the option or it being the default, but for my use case being able to specify the whole repository config via arguments sounds considerably more sane.
2) with ZFS snapshots (yes, I'm backing up `/path/to/dataset/.zfs/snapshot/<timestamp>/foo/bar`, but that should not be its path in the metadata!)
OTOH, it seems to have the upside that you can apparently alter snapshots after the fact more easily (e.g. if you find out you shouldn't have backed up that gigantic VM image you just moved somewhere temporarily). I leave the decision on whether this is a footgun or not to you.
And to be clear: The ZFS snaphot thing is also a pain with Restic, too. You can hack around it somewhat better with something like systemd-nspawn, but it really shouldn't be that hard.
Backup tool authors never seem to support this use case.
Instead of working out how to teach my backup tools about snapshots, I just mount them in a subtree and use that as a chroot env.
Looks like compression only added in the latest release of restic.
No mounts on windows.
No GUI?
Lack of compression was likely the reason I went with Kopia.
I recently stumbled upon the release notes for the (WIP) v2: https://www.borgbackup.org/releases/borg-2.0.html. Seems to address quite a few of the pain points of v1.
Eager to take another look at Borg and Kopia, etc.
as you see above, there is already a huge scope of what should be done.
to not grow the scope even further, some stuff shall not be done (now):
* no new locking system
* no public key cryptography (neither by gpg nor by reinventing gpg for borg)
* not adding cloud storage backends (like s3)
Feature list is only half of what it is.
Can you explain in more detail?
Duplicacy has been one of the best pieces of software I've ever used.
Will my automated hourly backups (via the Borgmatic wrapper) just start throwing errors and stop working?
The deduplication allowed me to move from Windows to Linux, while keeping a consistent backup. It's great stuff!
For Kopia, I do try it once a year, but I still find the docs and CLI args confusing. Running the server part behind a reverse proxy needs 2x HTTPS and searching the forum to get it somewhat working. For a webdav target, the progress display doesn't really work and it's not possible to cancel a backup run. So for now I'm observing and will retry next year.
We're looking to replace our self-written borg backup scripts with https://torsion.org/borgmatic/ which is a wrapper around borg.
(The FAQ doesn't seem to answer that)
ETA: found the 680GB minimum on the normal pricing page. So it’s 680 if you want a normal account, 100 for the “I know what I’m doing” account.
To second this, there are some threat models that are often overlooked with backups. Separating the encryption and restore keys makes sense, and is a design pattern few of the modern new backup tools offer - bupstash and duplicacy are the two that spring to mind which support asymmetrically encrypted backups.
The recent lastpass issue shows the importance of keeping the backup decryption keys offline and inaccessible. A headless server with backup script can upload backups, but if someone can access the script and backups, they have the golden goose (users' vaults).
Most other backup tools don't give you the ability to separate out creating and retrieving backups.
Separate moan - very few of the backup tools that use s3 style buckets give a proper description of the IAM security model you should use - I like to understand what object prefixes are used, and what the absolute minimum I require is, as I like to start from minimum privilege - PutObject, GetObject on an index or config file, and nothing else if I can avoid it!
Hopefully we see more focus on backup security after lastpass.
It was pretty fast already and recently got multithread support. It has been the only thing usable for backing up a few TB in a raspberry for performance reasons.
Keep in mind it's relatively new and the author does not yet recommend to use in production as the only backup solution.
Maybe it could provide a virtual read only filesystem, mounting the backup at a specific date, using the original atime, mtime and ctime
And the mount can restrict which directories to expose, so they only see the files they need.
(I wrote a small wrapper which can be run with sudo)
And borgbase.com has been a good place to host my backups. Really nice onboarding flow.
I believe this is roughly what borgbase does to implement their own backup protection features.
Deduplicating Archiver with Compression and Encryption - https://news.ycombinator.com/item?id=27939412 - July 2021 (71 comments)
BorgBackup: Deduplicating Archiver - https://news.ycombinator.com/item?id=21642364 - Nov 2019 (103 comments)
borg replaced https://rdiff-backup.net/ for us and gave: * nice speedup of backups/backup tests, * decent saving in the disk space thanks to compression and deduplication, * decreased backup replication time [ borg repo tends to have much less, larger files compared to rdiff which has in its repo at least as many files as your source data; rsync likes it ].
to finish backups in a reasonable time we had to parallelize backup gathering [ each server / vm goes to separate borg repo; this limits a failure domain in case of corrupted repo, but denies us benefit of deduplication on larger scale - across servers ] and borg archiving. without that - we would be a limited by a single cpu core performance [ borg is not multithreaded yet ].
it's worth testing the backups - we're doing it each day by using borg's repo self test and by extracting few key files and checking their checksums and content... just in case.
echoing other comments - https://kopia.io/ looks interesting but we have not tried it yet.
Anyone knows when 2.0 will be out of beta, and stable?
I backup to a Hetzner storage box and a Raspberry Pi at home.
Would I be better off using ZFS native encryption and replicating the encrypted stream without giving a key to the rsync.net side? I worry that one false move would mean I would have to do a full backup again. I think I kind of feel more comfortable having borg as a second technology in the system, too, so that my restore can’t be broken by a ZoL bug. Doing test restores with Borg is also nice because I don’t need a ZFS system to do the restore — restoring to macOS works just fine for example.
Funnily enough, I’ve never had to prune / ruler function any Borg archives. I just don’t have enough churn in my data to need to do it. Everything change is there in full.
It works so well. Thank you everyone involved for donating your time and ideas to providing these tools and services.
I think you mean you are using borgs "repo key" method where the key to the repo is stored, remotely, inside the repo, correct ?
Just to clarify for others:
That repo-stored key has a passphrase so this method still keeps all of your files encrypted in a way that nobody at rsync.net can access.
FWIW, I use this method personally - I like the idea of storing the key with the repo and unlocking it with a passphrase.
As I understood it, if my dataset is encrypted then rsync.net only sees the name of the snapshot and the encrypted data.
I’m not sure this would be a great solution for my most common restore use case though: fetching a single file back with borg while I’m at a third location that is remote to both rsync.net and my ZFS host. The fact that I can do that with just Borg and rsync.net is incredibly useful (yes, I travel with my keys!)
Prior to encrypted datasets in ZFS we did, in fact, have a stopgap measure where we would create your zpool on top of GELI devices such that it was, in fact, encrypted - but we held the keys. This was for HIPAA compliance and such things.
Currently, we run the latest stable/release ZoL codebase which includes bona fide encrypted ZFS datasets where you hold the key and we see nothing.
That being said, we do have a 4TB minimum[1] for these zfs-send enabled accounts where you get your own zpool ... so a smaller account with the (excellent) borg utility is a better bet for most individual users.
0, https://www.veeam.com/agent-for-windows-community-edition.ht...
Set it up on a bunch of servers with a simple cron job, the initial backups went quickly, and the incrementals were really fast. Made great use of an old Dell server that wasn't doing anything else and had lots of slow disks in it.
This is more than just data backup as we would need need to recover disk partitions/LVM metadata, boot records, etc. as well as all the data itself.
While other suggested, image-based solutions better fit your bit-for-bit requirement, you might also be interested in ReaR: https://relax-and-recover.org/
ReaR generates a bootable image which performs all that basic partitioning and is able to trigger an actual system/application data file restore using a variety of tools (including borg).
Slightly smarter approach is dd, then zero unused sectors and compress.
Both will produce an image which could me restored with DD (or mounted offline). Second will be smaller.
They should be run with unmounted partitions.
The most boring answer is "connect the disks to something else and use `dd` to copy the full blocks from start to finish into a file".
'dd' would have been my thought as well, I've heard of Clonezilla also but never used it and not sure it's really doing anything appreciably different.
I like the idea of 'dd' because I have a very clear mental picture of what it does. Just wasn't sure there was something else I might want to look at.
As I see it: I write some kind of configuration.
someproject-db is a deployment which runs a postgres db. Tool should connect to this DB, issue some kind of pg_backup command, capture output, retrieve some metadata about previous backup from S3, compute difference with previous run, compress that difference and store it to S3.
anotherproject is a deployment which runs an sqlite db. Tool should do the same but with sqlite-specific commands.
yetanotherproject-data is a pvc which has attached pv. Tool should find pod which mounted this volume, exec into that pod and retrieve pv data, again find different and store it to S3.
Of course things should be configurable. Like store difference every 15 minutes, store complete backup every week and so on.
I'm fine with manual recover and with manual configuration (I just don't want to write and test all the scripts myself).
What I don't want is some kind of magic tool which will backup the entire cluster, etcd and my grandparents automatically in some magic way only for $50k/cpu core.
Borgmatic will beautifully deal with DB dumps and there is a popular container image to run it. As for the cache ("retrieve some metadata about previous backup from S3"), you don't need to keep it locally. It can be restored from the backup repository.
Hope some of this applies to your K8s setup.
My suggestion is using ZFS.
I use Borg as backup of backup (zfs snapshots), so I'll be having multiple implementations of backups (also both are on different remote location) just to be on the safe side.
I don't use any other fancier ones as I don't like risking data on less reliable tools.
How does it matter if anyone else is using zfs? You either use a service that supports zfs target or run your own Linux instance which is just installing a single package for Ubuntu.
Even for my own use cases, not every server and system I maintain could use ZFS right away.
Still good to know about the encrypted volume feature. Will be sure to test this next year.
But I had to switch to restic because the S3 support was too good. Off-site backups in one command, instead of having to maintain my borg repos on NAS and external disks.
Been much more happy with my tries with https://kopia.io which also includes an optional cross-platform GUI, in addition to the CLI.
On one hand, on may think that three programs with very similar approach and features is a waste of resource. On the other hand, this is what refinement of the idea looks like: each project improves over previous attempts.
I wouldn’t trust my backups to Kopia (unless for experimentation).
I've recently decided to review my backup scheme, and have been looking at btrfs.
Anyone out there doing a btrfs+borg combo? What is your scheme?
I've found this as an example, but it's quite a bit more intense than what I need... https://mutschler.dev/stuff/backup/
What tool could I use that has a windows client, android and ios?
For syncing S3 storage across providers, I went with rclone (https://rclone.org/). Note that using rclone to sync across providers (e.g. from Amazon to Wasabi) does require the files to be downloaded to the client machine and then uploaded again. Not ideal, but if you have extra bandwidth it is a convenient setup.
And that is one of the main reasons why chunk-deduplicating backup tools (like borg, restic, ...) are better than full/incremental style ones.
so you can delete ANY backup archive without influencing any other backup archive. a chunk will be only deleted if nothing is referencing it any more.
also, each backup is logically a FULL backup (it has ALL files, references ALL content data). it is just made in a clever way, avoiding to re-transfer data that already is present in the backup repository, thus it FEELS like incremental (considering speed, amount of CPU and I/O used).
OTOH, full/incremental style backup tools build a chain of incremental archives depending on the previous incremental and the full backup, which gets more fragile the longer the chain gets.
because of that and also because you might want to delete older backups at some time, you are forced to create new full backups regularly (causing lots of CPU and I/O load).
No Windows :(
Luckily there are capable alternatives to Borg, including Restic and Kopia.
Also, if you rename a file or folder, that hardlink-based dedup will not work, because it is based on files having the same path. Also, if you change just 1 bit in a huge file, it will add the whole new file because the dedup only works on whole files.
borg does variable size chunk based deduplication. if a big file changed just a bit, your backup repo will also just grow a bit when backing up that file.
also you get compression, encryption, authentication and the ability to check backups if they are ok still.
also, borg won't create a gazillion of files/hardlinks in the destination filesystem, but way fewer files (each about 500MiB big by default).
borg also archives ACLs, xattrs, flags.
(1) The file I download is a command-line executable. I have to chnmod +x before I can use it. Why not share that small fact?
(2) How do I create a simple repository *with no encryption*. Yes, I want to start there. Why do you think I want to get a PhD in your project before trying it? No, I don't want to start by creating encryption keys; that comes later. Start with the basic, basic, basic instructions.
(3) Why did I export that environment variable? I never used it, did I? Not so far as I can tell. Is that basic? No, it is not.
(4) What does recreate do? Is it necessary? "borg rcreate --help" does *not* provide any help on the encryption options. It just provides an error.
Seriously, what is wrong with you people? Can you not imagine what it is like to come into an open source project with no experience and no knowledge of the matter whatsoever?
Is it machismo? Is it elitism?
What, in the name of God, prevents you from saying, "This is my project. It does this. This is the "hello world" method of trying it. Here are some cool options once you have the basics.
It happens almost all the time. I seriously don't know what is wrong with you.
Edit: I had to search forever to find the keyword "none" that you can put after the "-e" parameter. Found it by trial and error. It's in the docs, but, Jesus, people, can you not imagine for a moment that someone might be testing this on not the latest hardware and want to see how it runs without the complexity encryption first? You absolutely blow my mind. I don't know if you're putting me on, here. Do you not know how to communicate?
[1] https://borgbackup.readthedocs.io/en/stable/installation.htm...
https://vorta.borgbase.com is awesome. Simple and awesome. I wish it gains more exposure.
https://borgbackup.readthedocs.io/en/1.2-maint/quickstart.ht...
I'm wondering what's wrong with people thinking authors of open source software owe them something – anything. I really don't understand how people with a shred of decency can feel like this an appropriate way to behave.
People writing software and putting it out there for free can focus on whatever they like. And, for many, writing software is much more fun than documenting it. This might be unfortunate but is totally fine, given the nature of such projects. If you want better docs, you're (almost) always welcome to write and contribute them.
No cloud? Indeed, borg does not integrate cloud functionality.
You'll either need a directory or a server with ssh and borg to store your backups into. Some people use rclone to clone their borg repos to the cloud.