Arq 5: Massively faster backup and restore for Mac and Windows

Wait - what?

If your hard disk gets lost or destroyed, how do you protect those backups?

We're currently just rsyncing our pictures and stuff at home to two 3TB USB-drives (one active and one at my parents' place; using LUKS and btrfs with compression, snapshots). But even after running deduplication, they're filling up (raw files …), so I'm always on the lookout for other options. Upgrading to 2x4TB is a bit expensive, but I haven't yet found anything that'll cost me less than that while still having client-side encryption and Linux support. E.g. tarsnap seems to be about $250/TB-month, http://jotta.no/ is unlimited but has no Linux/encryption support, and I never understood Glacier pricing :-)

(and it's really convenient to be able to just restore from a local USB drive instead of having to wait for the network, though of course it's less convenient not having backups when travelling …)

I use self-hosted CrashPlan for several years, it's great, but restores and thinning archives feel slow (like half a day for cca 1 TB of backups).

I had an older version that I upgraded to 4.x because they added support for uploading backups to Dropbox and I was already paying for a Dropbox pro account that had plenty of unused space. The ability to have versioned backups on various cloud services that don't support rsync is a big plus.

Although the application itself is not open source, the developer has open-sourced a restore tool, to help ease the "what if they go out of business" concerns:

https://github.com/sreitshamer/arq_restore

Also, somebody else has created a Go utility that can read the backup format:

https://github.com/asimihsan/arqinator

Finally, if you really wanted to do it all yourself, the data format is documented here:

http://www.haystacksoftware.com/arq/s3_data_format.txt

http://www.dirvish.org

Although if an attacker has control writing to your s3 bucket, they could rack up a big bill.

I know there was some hue and cry a little while ago about Mac ransomware that can encrypt network drives and external hard drives, but there's a reason why the _encrypt_timemachine routine was an unused stub. From what I understand, Time Machine has protections built into the kernel that prevents existing backups from being modified. New backups after the ransomware attack would obviously end up backing up encrypted data, but the existing backups should remain untouched.

Both Arq and Time Machine create differential backups. Thus, any particular backup can be restored back in time. However, Arq targets non-file-based media (although you could trick it by a little SSH magic). Time Machine requires file-based access.

If ransomeware finds your file-based backup, it will encrypt it and render your backup useless.

The term backup gets bandied about, so it can mean one or more of the following: high-availability, synchronization, and/or disaster recovery. You'll want to look into these and the concept of the 3-2-1 method.

From my uneducated perspective, having simple software that just worked was a bonus - who knows, maybe with the addition of threading, and consumption of filesystem events, ARQ is going to become crummy, and a door will be opened for someone else to write simple backup software without those features that gets the job done and doesn't bog up your computer. I guess we'll have to try Arq 5 for a few weeks to find out. Fingers Crossed.

But getting the single threaded path solid before adding parallelism is a good choice.

They're pretty reputable for free speech and fighting for their customers too, so I trust them to a decent degree.

For Kloudsec, all you have to do is to update your A records to point to the CDN IP.

http://www.econtechnologies.com/chronosync/overview.html

SuperDuper is the easiest to use, and I would recommend it to anyone as their first choice.

CCC and ChronoSync both have more advanced features: including bootable network copies, and backing the recovery partitions.

ChronoSync also offers two-way folder sync's, enterprise features, and more.

[1] https://www.veeam.com/endpoint-backup-free.html

On Windows, my favorite by far is ShadowProtect. It's actually a sector-by-sector backup that pretends to be a file-by-file backup when you restore a file.

What's great about this is when you modify a huge file, it only backs up the actual sectors you changed. (Most Windows backup programs detect changes on a file-by-file basis.) I have it set to run a full incremental backup every 15 minutes; the backup typically takes 15-30 seconds and is unnoticeable when it happens.

Even though it's a sector backup, you can still restore specific files or do a complete system recovery, either to the same/compatible device or with a Hardware Independent Restore to different hardware. I've done each of these many times.

On OSX, I'm using Time Machine, but I wish there were something as good as ShadowProtect. It is a bummer when I touch a few sectors in a huge file like a VM disk image, and the best Time Machine can do is back up the entire file again.

https://bombich.com/

Protecting the entire system should be something that is done near line so if you have a catastrophic loss your time to recovery is less than that if everything was stored somewhere in the cloud. Or Just not do it at all and rely on a tool manage the configuration of your system.

Data files and configuration are really the best thing to protect with a tool like this. If you have a total loss your playbook should include something like replacing the failed hardware, Installing and patching the OS, replaying configuration of the system using ansible or chef, restoring data files.

To me this is the fundamental gap that cloud backup solutions need to fill to really capture the consumer market well. The SMB market already has this as you goto IT and get your system reloaded and then restore your user data, its pretty much the standard for larger corps.

For example, on simple, regular text (JSON) files I'm seeing about 45% worse compression than plain gzip (default compression level). I hope at least it's using the highest compression level, but I've found that "lz4 -9" is about as slow as "gzip -6", still with worse compression.

I'd be happier if the choice of compression was dependent on the size of files. The larger a file, the more you gain from compression.

Does Arq skip compression for already-compressed files (.gz, .xz, .bz2 etc.)?

Most of my backup set is pictures and compressed files, so maybe the lz4 decision was because they figured the majority of their customers fell in that category where no compression algorithm is going to help?

[Edit: Apparently the release notes for Arq 5 says it can now back up to a local folder.]

I have never tried Crashplan though. The main reason I chose Arq instead is the impression that Arq has a better thought-out encryption scheme.

[0] https://www.arqbackup.com/

edit: You get a free upgrade if there is a major version released within 90 days of your order (if you don't order lifetime upgrade of course)