Hacker News Reader

Comment by drivebyacct2 | Hacker News Reader

>And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit.

From a technical point, they largely are contrary aims.

Tarsnap gives deduplication, compression, and encryption. Some technical background on how it does this here: http://www.tarsnap.com/efficiency.html

There is a detailed breakdown provided as well. Sample output:

                                         Total size  Compressed size
  All archives                               697 MB           323 MB
    (unique data)                            215 MB           100 MB
  This archive                               148 MB            67 MB
  New data                                    17 MB           5.2 MB

Edit: Don't want to misrepresent Tarsnap's granularity. The print-stats option gives size in bytes. Above output was generated with the --humanize-numbers option.

I assume this is only deduplication within a single account? Because I don't see anything there on how one would provide dedupe across crypto domains (which, to the best of my knowledge, should be theoretically impossible if the crypto is done right).

Correct.

Just been looking at Tarsnap. It looks good but I'm curious what happens if the author/creator/owner/maintainer (singular) gets run over by a bus?

I do my best to avoid busses. (I've always been more of a fan of switched point-to-point connections than busses, to tell the truth.)

In all seriousness, if I get hit by a bus Tarsnap probably won't live on without me -- but it runs itself smoothly enough that there should be plenty of time for people to download their data.

Thanks for your reply. I'd been looking at the FAQ and it seemed like there wasn't much redundancy built in - not wishing ill of you at all.

You're quite right, and it's a reasonable thing to ask. I don't mind.

More sample output, from a heftier machine, in case anyone is curious:

                                         Total size  Compressed size
  All archives                        3321059166180    1113881916696
    (unique data)                      116996664943      27182261608
  This archive                          13831036740       4855451330
  New data                                336950399         62343192

(Yes, that's 3.3 TB deduplicated and compressed down to 27 GB.)

Can I automatically sync files using it? I work in multiple places and if I have to manually run a sync operation before I leave, I'm certain to forget or be unable to at some occasion, and then I'm screwed.

Tarsnap doesn't do sync. Doing sync while maintaining security is a much harder problem: Tarsnap's design requires each system to know what's on the server before uploading or deleting blocks, and that breaks if you have multiple systems accessing the same storage space.

The server need not be party to the key that is used to exchange the filenames themselves between the clients.

The content can be compressed in blocks before upload; dynamic, anchored block sizes would increase de-dup.

The compressed blocks can be individually encrypted with a key derived from the filename and block.

There's still plenty of opportunity for whole-file de-dupe there.

You can deduplicate within accounts; cross-account deduplication is indeed a problem.

Reply on news.ycombinator.com