Hacker News Reader

Comment by rsync | Hacker News Reader

You should look into the 'borg' backup tool - it has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key to ... your cloud provider has no access to the data.

The borg website is here:

https://borgbackup.readthedocs.io/en/stable/

and a good description of how it works and why you should use it is here:

https://www.stavros.io/posts/holy-grail-backups/

View on HN

How does this help with the context we're dealing with here: the iPhone (and other iDevices) being heavily encrypted/secure and iCloud not being secure?

Quickly looking at Borg's website (thanks for the heads up - great tool/option) I see it doesn't support iOS or backing up an iOS device.

I assume you're just suggesting it as a general purpose option for general backups on the desktop?

Maybe backup to desktop and then Borg.

I wonder how efficient Borg would be with this setup.

After looking at a few alternatives (Borg, Duplicacy etc.), I setup Arq on my Mac yesterday.

One thing that irks me about these solutions is that they seem to scan my folders each time they want to backup. Are there tools that are smarter about this? For e.g., while running, they could keep a log of what's changing and only scan those while backing up.

You would probably need the kernel / filesystem to keep track of it to keep it efficient. And even then that is iffy if you implement it as a kernel extension vs. something integrated into filesystem code.

I've found arq backup overall slow once you start hitting 0.5TB overall. It's a design issue.

> You would probably need the kernel / filesystem to keep track of it to keep it efficient

I believe most operating systems have a mechanism for this. E.g. on macOS it's called fsevents, Linux has inotify

Yes, these utilities would probably need something in them such that fswatch implements (https://emcrisostomo.github.io/fswatch/) - very useful.

> After looking at a few alternatives (Borg, Duplicacy etc.)

I may have looked at Duplicacy. I'm sure that I looked at one of Duplicacy [1], Duplicati [2], and Duplicity [3]. Whichever one that was, I kept getting it mixed up with the other two when looking for information online, and finally said "screw this" and bought Arq.

[1] https://duplicacy.com/

[2] https://www.duplicati.com/

[3] http://duplicity.nongnu.org/

I've been thinking about setting up a backup for my Mac for a while now. How long does it usually take to scan your folders during a backup?

I am backing up ~130 GB data (with a combination of large and small files) every 12 hours to S3. The first upload quite some time but all the later ones take ~10 minutes in total.

Arq for Mac is great for backing up, though the restore util could use a little work. It's not exactly user friendly.

I've used it on my macs for years without any issues at all. I switched after Time Machine broke down for the n'th time in a month saying it needed to recreate the backup, and not once in the 3-5 years i've been using it has it every given me any problems with broken repositories, and every integrity check/restore has succeeded.

Arq on Windows is a different beast though. I'm sure it's technically solid, but the UI leaves a lot to be desired. On windows boxes i default to Duplicaty.

> Duplicaty

Did you mean Duplicacy or Duplicati?

Duplicati (https://www.duplicati.com/).

On my servers i use Borg like any sane person would, but the lack of a good client UI makes scheduling backups on a personal computer a lot more work than i'm willing to put in.

I already use Duplicacy for backing up family personal macs to the cloud (in addition to local Time Machine backups).

For linux servers, borg is great, and me and my companies are happy users.

Are there any advantages of using borg over rclone?

Yes.

Rclone simply copies data. If you `sync` `~/Documents` to your remote it will keep an exact copy.

This is a simple backup since you only have one version. Anything deleted, the next time it syncs, gets deleted.

Borg is a backup tool. Versioning is at its core. It does that efficiently by deduplicating file (chunks really) even if they’re not in the same location.

So with Borg, if you create a backup 1 of `~/Documents` today and a backup 2 tomorrow of `~/Documents` you can see both backups and work with each snapshot. The size it takes should be close to the amount of data changed in the whole source.

If you move directories or files inside, rclone has to reupload them. Borg detects but doesn’t have to store it again.

With rclone some remotes have versioning (Google Drive, Dropbox). This could help in this case, but it depends on the remote. With Borg this is built in and you can change the underlying storage and migrate without loosing any data. Using versioning with crypt would probably be a pain too due to the file names. Not sure if rclone has commands/flags to help with this that I simply don’t know about.

"Rclone simply copies data. If you `sync` `~/Documents` to your remote it will keep an exact copy."

...

"This is a simple backup since you only have one version. Anything deleted, the next time it syncs, gets deleted."

This is correct. It is widely advised to not consider a "sync" like this a proper backup.

However, for what it's worth, rsync.net does support rclone[1] and because of the ZFS snapshots that are created and maintained[2] in your account, you can just do a dumb sync because the retention is handled by the snapshots.

I am not sure if rclone is really the right tool for plain old cloud backups - I think rclone distinguishes itself for the ability to transfer data between cloud providers.[3]

[1] https://rsync.net/products/rclone.html

[2] ZFS snapshots reside in the .zfs/snapshot directory inside your rsync.net account.

[3] ssh user@rsync.net rclone file/in/rsync/net/account s3:/some/bucket

Reply on news.ycombinator.com