Hacker News Reader

Comment by kogir | Hacker News Reader

Tarsnap works really, really well. Just make a consistent snapshot of your data (I'm using UFS snapshots), point Tarsnap at it, and you're good to go.

The documentation is thorough, and Colin (the owner/operator/author) responds quickly to emails.

Finally, compression and deduplication is amazing:

  [nick@home ~]$ sudo tarsnap --print-stats
                                         Total size  Compressed size
  All archives                               348 GB            76 GB
    (unique data)                             34 GB           6.3 GB

Yep, I've backed up 350GB of data, but since most of it is duplicated, I pay for storing 6.3GB. Win.

One word of caution though - this isn't a mainstream consumer backup service. If you lose your keys you lose your data. No chance of recovery. So make sure you back those up properly too, ideally in a different geography.

View on HN

Just make a consistent snapshot of your data (I'm using UFS snapshots), point Tarsnap at it, and you're good to go.

You're using the --snaptime option, right? It's necessary when you're backing up a filesystem snapshot in order to work around a race condition with them -- if a file is modified, the filesystem snapshot is created, and then the file is modified again, all within a single time quantum, it can trick Tarsnap into thinking that the file hasn't been modified later (which triggers an optimization of "this must be the same blocks as it was last time" in place of the usual "read the file and split it into blocks" behaviour).

Finally, compression and deduplication is amazing:

Well, if we're going to be posting statistics here...

                                         Total size  Compressed size
  All archives                               269 TB           121 TB
    (unique data)                            177 GB            72 GB

That's 269 TB of data backed up from my laptop, deduplicated and compressed down to 72 GB. This is what I get for taking a backup of my entire home directory every hour...

> You're using the --snaptime option, right?

Yep, emailed you about this in fact, and appreciated your detailed response.

Ah, that was you -- I remembered sending an email about snaptime recently but couldn't remember who it was to (and HN user names don't always correlate anyway...)

Excuse my ignorance, but what does duplicated in this context mean? Multiple copies of the same file?

Multiple copies of a file, or parts of a file. These can be spatial (your archive contains an SVN checkout, so there's an extra copy of every file in the .svn/pristine directory) or temporal (you take regular backups, and many of the files haven't changed very much from one backup to the next).

Ah okay, thanks. Another question: Why wouldn't compression take care of that? Isn't the point of compression to compact as many repeating sequences as possible?

Deduplication is a form of compression, yes. Most forms of compression are "local" however -- looking to match data against bits from within the past few MB -- so they won't detect duplicated data spread across entire archives.

In order for compression to work, the data must be 'solid'. Which means to add or remove something from the archive, you must reprocess the entire archive. This isn't a very good model for backups, especially when you have lots of them. (As an aside, .zip files aren't 'solid', so two copies of a file won't compress well. This is also the reason why most archives on linux are done through tar, to create a single stream of data).

Tarsnap uses variable blocks (in such a way that inserting into the middle of a file creates minimal differences). If a new block is detected during a backup, it only needs to send that block, as the rest are already stored on the server. It also means that new archives can refer to the old blocks stored, allowing each archive to be independent, and unused blocks removed when the last archive using it is deleted. Before sending, blocks are compressed then encrypted, so compression can't really help since you might not have all the old data locally.

You can also deal with backups using a master and incremental diffs. This doesn't work well with the tarsnap model, as archives are no longer independent.

Reply on news.ycombinator.com