Not sure why Tarsnap has bubbled up on HN again, but I'm always happy to answer questions about Tarsnap if there's anything anyone here wants to know.
1) "Every Tarsnap archive acts like it is completely independent of all other archives"
Lets say I'm backing up `/somedir` that looks like this:
# archive1 - initial backup
── somedir
├── file1.txt
├── file2.txt
└── file3.txt
── somedir
└── file3.txt (modified)
If so, how is that possible?
2) Does tarsnap stream the backup data to the tarsnap service or does it create the encrypted archive and only after this upload it?
For instance, if I've got 100 GB to backup (first time), do I need 100 GB of free space for tarsnap to work?
If archive2 only contains somedir/file3.txt, then if you extract archive2 you'll get somedir/file3.txt but not file1.txt or file2.txt.
But if you create archive2 containing all three files, tarsnap will recognize the duplicate data in file1.txt and file2.txt and not re-upload it; but when you extract archive2 you'll get all three blocks. Tarsnap reference-counts blocks of data so that when you delete archive1 it only deletes the data which is not used by any of the remaining archives.
I often tell new Tarsnap users that they should start by forgetting everything they know about incremental backups. Tell Tarsnap what data you want to have in an archive, and let it do the work of figuring out what's new.
Does tarsnap stream the backup data to the tarsnap service or does it create the encrypted archive and only after this upload it?
Tarsnap uploads data as it collects it. Tarsnap needs a small amount of disk space to keep track of which blocks have been uploaded previously, but it's less than 1% of the size of the data archived.
I currently use Arq[1] for backups (which has its own encryption[2] with a user choice of cloud backend), but Tarsnap has such a stellar reputation I'd definitely try it if I could :)
[1] https://www.arqbackup.com [2] AES-256 with PKCS5_PBKDF2_HMAC_SHA1 for key derivation, implemented by OpenSSL, with an open file format https://www.arqbackup.com/arq_data_format.txt
cant say anything about arq, as ive never head of them.
Or not. It depends on your use-case, but I'm just suggesting you consider that factor too.
[I was just trying out tarsnap, go to the restore test section, and was a bit stumped. It seems like "-xC /other/dir" might do it, but the man page is pretty opaque. I'd like to test a restore without the risk of clobbering my data with some misconfigured backup!]
There's also the trivial (but often unhelpful) advice of "be closer to the server" -- the problem is one of round trip latencies since the internal design of tar (and thus tarsnap) works on the principle of "read a block; do something with it; repeat". This is what I feel to be the biggest technical pain point in tarsnap, and I know in theory how to fix it, but every time I've started focusing my energy on it other things have come up requiring more urgent attention. :-/
Re simultaneous connections: The current release is limited to 500, but the next release (a matter of weeks) will be able to handle an arbitrarily large number of connections. That said, it uses select/poll based non-blocking I/O, so performance may suffer if most of your connections are idle. (This hasn't been an issue for any spiped uses I'm aware of.)
Any interest or thoughts about adding Google Cloud Storage as an option vs AWS S3?
S3 (us-east-1) runs $0.023 per GB. Google Cloud Storage in a single region runs slightly less at $0.02 per GB.
Howerver, bandwidth may be the biggest cost factor for you not storage.
Bandwidth isn't a huge cost; but the server which keeps track of where all the bits are in S3 and shuffles them around is surprisingly expensive.
What about if you were to age things into Nearline and Coldline using lifecycle policies? Also, someone claims you do multiple regions, which our "default" multiregional does at less than half the cost of S3.
Admittedly we only recently (finally!) added per-object storage classes. But unless I misunderstand how your blocks work in tarsnap, wouldn't that be amenable to having some large portion as Nearline and possibly even Coldline? If not, I'd love to understand! (contact info in my profile).