A Tarsnap-Alike for Private Storage | Hacker News Reader

A Tarsnap-Alike for Private Storage

synctus.com

71 pointsrlpb41 comments

As always we mention TarSnap, I can't but be amazed Mr Percival, having all the infrastructure in place just sort of a GUI, passes on the gold mine of the millions of Windows users that couldn't care less about the mere existence of a command line, but still would pay for a backup tool as good as TarSnap.

I wonder how much more business tarsnap can get, just by packaging a couple of debs, not even talking about a having a repository.

Maybe he decided that the time required to support Windows users isn't worth the money? He is quite a busy guy from what I've seen.

If he had a Windows version it should be able to make enough money to pay for a support person AND still be incredibly profitable. He just needs to have someone else do the selling in order to let him focus on the hard technical problems.

That being said, the market for a Windows version should NOT be individuals, it should be marketed to the enterprise. Tarsnap solves a problem that many companies have and pay buckets of money to cure.

Colin Percival is another classic example of someone being brilliant in their field, but a mediocre businessperson. This happens a lot; probably the most common examples are great chefs that open restaurants and bakeries only to find that actually running such things requires a set of skills that they don't really have, or are interested in acquiring.

(edit: I say that even though I don't even possess the aptitude necessary to fully respect his skills; he's a genius, and that's not a word I ever use lightly. I just hate to see a great work of engineering, like Tarsnap, languish in relative anonymity because it's poorly handled as a business.)

I get to work everyday helping some of the many brilliant individuals you discuss, who are mediocre business people. It's surprising how many don't recognise that business skills are just as important as their technical skills (though, for the most part, easier to learn).

It is super exciting sometimes running my business, helping give transfer some of those business skills that change their lives and those of their customers.

And I can't / don't speak about Colin in this comment.

Interesting. I've been using rsnapshot for a while for a similar purpose, although its de-duplication is limited to keeping unmodified files as hard links.

It'd be nice to see a brief comparison of ddar, rsnapshot, and other things that might fit that role.

rdiff-backup also.

I attempted a comparison with rdiff-backup here: http://www.synctus.com/ddar/rdiff-backup.html

Some others are briefly mentioned under "Alternatives", including a comparison with Tarsnap.

I only have direct experience of Tarsnap and duplicity. Additions/corrections welcome.

I use duplicity for incremental encrypted off-site backups. Would be interesting to see how it compares to the others.

As I store the backups on a shared machine, the most important feature for me is that the backupped data never passes through the target machine in unencrypted form, at any stage.

ddar seems more similar to bup. A comparison between those two would be awesome. I see that ddar supports removing old archives which bup doesn't yet. bup on the other hand already has a fuse filesystem which is on the todo list for ddar.

We've been using BackupPC (http://backuppc.sourceforge.net/) to accomplish something similar for a long time, and have been amazed at how it's able to store huge amounts of backups on relatively small amounts of disk space. It's highly configurable and has a decent web interface.

Sometimes, though, it can be challenging to get it working, and it has a few other issues (e.g., moving backups for systems from one pool to another) that can make it difficult in the long-run for some use-cases.

Because I'm guessing there might be others thinking the same question, I'll go ahead and ask -

How is this different from something like rsync?

rsync is for bringing one tree in sync with another. It can do this relatively efficiently if you already have a previous version of the tree at the destination, which is a fairly common case. rsync compares (more or less) each file with its parallel file on the other side and transmits just the differences. rsync won't save you any storage, just bandwidth.

ddar is for storing things de-duplicated, possibly over the network. It will save both storage and bandwidth. However, you won't get a tree on the other side, just an opaque container than you can extract the data out of again.

So: rsync to bring two trees in to sync, Tarsnap or ddar to store things (eg. for backups).

You could use rsync to store backups, but only really with the help of other tools. Tools like duplicity, rdiff-backup or rsnapshot.

Tarsnap was the first common system (to my knowledge) to do full de-duplication for storage. ddar works in the same way.

By full de-duplication I mean for the system to have an overview of all data in order to optimise transfers and storage.

For example, if I were to download a large image (say a Debian ISO) and then back up my machine using Tarsnap or ddar, then delete the ISO and perform another backup, then download the ISO again and perform a third backup, the third backup will optimise out transmitting the image because it is already stored. I don't think that rdiff-backup, rsnapshot or duplicity can do this, since they only consider one backup at a time. I believe this applies to renaming files around as well.

Thanks for the clarification. rsync is so fickle with it's deduplication that it doesn't handle file attribute changes either due to hard link limitations.

rsync does a lot to maintain file attributes across heterogeneous systems, does ddar address this issue as well?

ddar doesn't care. The intention is that you use ddar together with a tool that will do what you need, such as tar as in the examples. tar will deal with file attributes; ddar will deal with the de-duplication.

I think it's easiest to explain with an example.

So, every day at 10pm, I want to make a backup of my server to another server. I ddar or rsync the files to another server. Monday at 10pm, I do it; Tuesday at 10pm, I do it; Wednesday at 10pm, yep, did it. Now it's Thursday and for whatever reason, I have to restore the system to the state that it was on Monday. With rsync, changes have been overwritten.

ddar and Tarsnap are useful because they allow you to have a full backup of each of those days, rather than just the latest. Likewise, they package it better than the old "do a full backup and then do incrementals on top of it" way so that you don't have to restore the original and then replay all the incrementals. Now, you could just store full backups each night, but that would take an incredible amount of space. ddar and Tarsnap allow you to have the convenience of full-backup while having the data de-duplicated so that it doesn't take up too much disk space.

Rsync is more about saving bandwidth while transferring large amounts of files.

Are you aware of the --link-dest option for rsync? It seems to do the same thing. I've been using this (via dirvish) for years for doing full nightly, deduped backups.

Hardlinks are only useful if you have files which are unmodified. Tarsnap (and I assume ddar) will take advantage when part of a file is unmodified.

rsync with --link-dest will store each version of a file separately, including parts that are the same. It also won't hard link files in different areas of the tree that are the same, storing them multiple times also. Nor will it relink files that disappear and reappear, or change and then change back.

I wouldn't really call it deduped storage at all. The optimisation it does do might well be sufficient for your needs though.

Has there ever been a filesystem that does de-duplication on the fly? Or would something like that be too slow for real world use?

ZFS does dedup.

And so do LessFS, Open dedupe and some others.

Thanks for the info guys.

This looks very cool. I'm trying to use the examples, I've created a ddar directory with its db file and sha objects, but when I try a 'ddar tf /path/to/ddar/directory' I only seem to get either a timestamp or the -N name I've passed. Is this right? How to I see my filenames I've archived?

As far as I can tell you've done it right so far.

ddar is generic and just stores whatever you put in. You also need to use the tools you used to format the data in to read them back out.

If you've stored files using tar, then you want `ddar xf /path/to/ddar/directory|tar t` to see the files. Or if you've also gzipped them, then `ddar xf /path/to/ddar/directory|tar zt`.

Does this help?

Ahh that is exactly it. I am just not using the correct syntax. Thanks! This looks to be very cool, looking forward to trying it :]

Thanks for putting this up it looks really interesting! Is there a git repo somewhere that we can look at or are we stuck with the tarball?

Just the tarball for now, though it will probably make its way to GitHub soon.

"It just so happens that I’ve been working on de-duplication for Synctus"

OP should clarify that Synctus has no issue with this project, prove he has rights to release, etc, and stick that in the license.

This feels like a copyright morass (although probably isn't).

I'm sorry this isn't clear. I wrote Synctus, am owner/director of the company that owns it, and have released ddar under the GPL having authorised myself through my company to do this.

Right right, I actually expected that to be the case, but you won't necessarily always be the owner of Synctus, etc.

You should probably do the short amount of paperwork required: http://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html

"If you think that the employer or school might have a claim, you can resolve the problem clearly by getting a copyright disclaimer signed by a suitably authorized officer of the company or school. (Your immediate boss or a professor is usually NOT authorized to sign such a disclaimer.)"

Stick everyone's mind at ease.

Neat project (which is why I care enough to get the ducks in a row).

A disclaimer doesn't quite make sense for me. Let me explain: I fully own True Blue Logic Ltd and am sole director. Both Synctus and ddar are projects owned by this company. Copyright on both Synctus and ddar are held by this company. Synctus is not open source. This ddar release is open source (GNU GPL 3) as of today.

So my company retains copyright but licenses ddar to the public under the terms of the GNU GPL 3. Thus there is no disclaimer to sign, since my company still claims copyright.

Of course, now that ddar is released under the GPL, it will be forever (in the same manner as any other open source project).

None of this affects Synctus, which is not open source. If anyone contributes code to ddar, those contributions would not be permitted to be used in Synctus without a further licence from the contributor. I've considered this carefully and decided that this won't be a problem for me.

>So my company retains copyright but licenses ddar to the public under the terms of the GNU GPL 3. Thus there is no disclaimer to sign, since my company still claims copyright.

A disclaimer isn't needed now that it's clear in the COPYING file that TBL Ltd (the copyright holder) is the one publishing it. Thanks for clearing the air.

since you may have investigated this: how does this compare with eg: repackaging the tarsnap deduplication code sans the encryption work?

Tarsnap is not open source, so I intentionally avoided looking at the code to avoid any doubt over provenance. I imagine it would be harder, since Tarsnap is backed by S3 and AIUI Colin's own transactional storage layer to make it work with S3.

Oh wow, I started using tarsnap yesterday night!

And they've already launched a major addition. Imagine how incredible the company will be by a week from next Thursday!

Will this compile on Cygwin like Tarsnap does?

It should only need a couple of minor tweaks. I am looking into it (as well as OS X support) and will post on my blog when it's ready.

Thanks for the inforamtion. I have a combination of Linux or Windows machines at home and it would be handy to be able to use the same tool on both.

Reply on news.ycombinator.com