Aren't these storage prices absurd? Please let me know if I'm misunderstanding.
Tarsnap makes a lot of sense when you benefit from the encryption and (especially) de-duplication features that it offers. For me, all of my most important personal and business data, from multiple decades, compresses-and-deduplicates down to around 6GiB. Considering the high value of the data I store in it, tarsnap's pricing actually feels absurdly low.
Can you provide more detail why you think so? I don't believe there is any use case in which tarsnap makes sense, other than maybe some Plan-C backup solution which you fall back on in the highly unlikely event that neither Plan-A nor Plan-B worked.
Concretely, what benefits does tarsnap offer over restic or borg in combination with rsync.net, to make up for the substantial downsides (such as insanely slow restore, complete lack of wetware redundancy or being written in C[1])?
This ability is critical to prevent a compromised system from having its data wiped and having all backups wiped as well.
I haven't been able to figure out how to do this in any other system. But if someone has a tutorial, I am all ears.
The thing is that tarsnap deduplicates over arbitrarily long time periods, letting me make arbitrarily long staggered sequences of retained archives.
Perhaps I should really reconsider if I really need such long lived archives, but it is hard to bring myself to drop them.
I just now need a deduplicating asymmetrically encrypted backup program.
I've tried duplicity in the past, and maybe I should try it again. But my recollection is that duplicity will just fail to do backups at the slightest hint of any problem. Like maybe if the last backup was interrupted then no more backups for you until you attend to it.
Edit: More memories returning of having to dig out my decryption key to resync the metadata when duplicity gets unhappy, and then since my target server was append-only, duplicity was upset when it wasn't allowed overwrite any of it's incomplete metadata files. I guess the ZFS snapshot technique would alleviate the latter issue.
To be fair, if tarsnap gets confused it needs the keys to do its fsck command, but I recall this sort of thing happening regularly with duplicity and almost never with tarsnap.
An rsync.net account can have any arbitrary schedule of snapshots - including days, weeks, months, quarters and years.
I've been musing on this subject all afternoon. I'm a user of Tarsnap, and I do find it expensive, in the sense that I would prefer to backup larger amounts of data for less amount of money. At the moment I backup photos separately from Tarsnap and in an adhoc way.
But I still cannot figure out a way to get all the benefits I get from Tarsnap from any other software solution.
* Must be usable under Nixos.
* Backups must be asymmetrically encrypted so that backups can be automated, yet a compromise of the system cannot immediately gain read authorization to arcived data.
* Backups must be append-only without further credentials, or otherwise prevent a compromised system from being able to delete existing archives.
* Deduplication between archives while still allowing archives to independently be deleted.
Using the ZFS snapshot functionality with rsync.net, for example, with Duplicity comes close. However, as I recall, duplicity wants to regular (typically monthly) full backups and then incremental backups from there. You cannot remove these full backups without deleting the entire month's worth of backups, and because the full backups are independently encrypted, there is (of course) no deduplication between full snapshots, even though the data is still likely largely the same. And because the snapshots are encrypted, it is impossible for the rsync.net storage to see or even know that large parts of the encrypted data is identical.
AFAICT there is really nothing else that does what Tarsnap does.
* Create a S3 bucket and enable versioning * Create a new user and give it only s3:PutObject on your new bucket * Create an auth keypair for that user and put it on your server
Now any server compromise that gets those keys can only add new data to your backup bucket, and can't read, overwrite, or delete any previous backup.
There's no dedup, so that could be a deal-breaker.
There's also no real encryption (though that shouldn't be too hard to add I guess). I don't really see the gain though. Anyone who compromises the server keys is blocked from reading by AWS permissions. Granted, that's not quite as reliable as good crypto for blocking reading, but on the deleting side, there's never going to be anything but the auth system of whatever solution you're using to block that.
I get that there's some applications out there where preventing data exfiltration is important enough to need strong crypto (though is that really important when we're talking about full compromise of your server, which gets the attacker direct access to the data anyways?), but I decided that the risk of failing to implement properly or full data loss due to losing the keys or them being corrupted wasn't worth the risk of blocking somebody who somehow compromised the AWS account security from being able to read backup data.
My main machine is currently storing 1.6 TB (compressed) of total archives with tarsnap, but only 33 GB (compressed) of unique data within those archives. So if S3 is 50x cheaper, then not having deduplication would be a wash.
However other comments here suggest that S3 is only 10x cheaper.
Edit: just saw your sibling / reply-to-self comment. This setup would fulfill the requirements you posted, or at least I would assume that restic runs under (or compiles for) your nix OS. It doesn't use asymmetric encryption for this but the goal of append-only is there
> because the snapshots are encrypted, it is impossible for the rsync.net storage to see or even know that large parts of the encrypted data is identical
If they don't see a large amount of data incoming, they'll know large parts of the data are identical (or removed, I suppose). Hiding traffic volumes is fundamentally only possible by introducing dummy data
Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost
rsync.net : $0.015 / GB storage, no bandwidth cost
s3 : $0.023 / GB storage, some complicated bandwidth pricing
If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?
- your comment was a very valid question but rather quip-like, offhanded, seemed off etc etc. I mean something like that…
- Tarsnap is an hn darling
If I have to pick one I think it’s the latter :)
Oh dear. It’s an HN thing. I have had brushes with it only once or twice across various accounts across years but it’s very much an HN thing.
Whenever you see an utterly useless quip (or sometimes even name calling or offensive words) being heavily upvoted you should know that some alpha HNer has arrived on the scene :)
But to be honest I have never seen author Tarsnap engage in such privileged gentlemanly d-baggery. He is quite cool, as they say it.
Anyway I just ignore it and move on. But again OP could have worded the question better. I mean no matter how good or bad you want to feel about it — it’s just a vc run anon forum and just another forum.
Which is probably okay if you want to pivot from a geek-ish service to one that geeks don't use, of course. Does the owner want that?
If I want to store my 100GB of data now, and I want to have it stored for a year, I want to pay for that year's worth of storage of 100GB of data now and not worry about any money or account problems for that bit of data.
He’s brought far more value to the community than that, of course.
The 120Gb is the contents of my OneDrive and local repository trees. This is everything I've ever done that I want to keep and is approximately 115Gb of photos and not a lot else!
Backblaze: $0.005 / GB storage, $0.01 / GB download.
Tarsnap is a wonderful piece of software. You're paying for that.
That said, is the value of "Tarsnap" worth the price difference from "Borg+rsync.net"? (Or Restic, I've been meaning to look into Restic). I'm not so sure. These days I'm a customer of rsync.net, not of Tarsnap.
But I still firmly disagree with the "Colin's just exploiting the uninformed" angle.
Ugh.
Try picking a choosing specific file types or file extensions from filesystems holding thousands of files.
I ended up having to cobble together some god-awful pre-process powershell with multiple pipes just because restic fails to be able to grep using Windows reliably.
:(
That is news to me. I backup almost a million files spread across 4 Windows devices, with heavy use of --files-from and --iexclude and it seems to work. What am I missing?
I agree that restic filtering options are pretty limited. Too limited, really. But what's there seems to work?
With regards to duplicity, Tarsnap does full deduplication across all backups for any given "machine", while still letting you independently remove any snapshots you like. i.e. no special "full snapshot" that must always be kept around, and no need for multiple full snapshots that have no deduplication between them.
There are services like rsync.net that support borg at a lower price. Borgbase is one of them. I haven’t used either of these.
And rsync.net is even one of them!
"Special "borg accounts" are available at a very deep discount for technically proficient users." -- https://www.rsync.net/products/borg.html
...hrm, it seems they didn't update that page with last year's price drop. https://web.archive.org/web/20220319135035/https://www.rsync... It used to be a deep discount, now it's the same for <100TB. I wonder if they did drop the Borg prices too and just forgot to update that page?
Run rsync to the target and forget is quite easy, though I admit rsync.net's deal is getting worse these days posing minimum usages here and there.
I don't think so. Anyone who can use this software I'm sure knows what other options exist.
Geez, that's really not improving the comparison with Tarsnap.
That's pretty much any SaaS... look at the various log or metrics gathering solution, where you pay serious multipliers of what would cost to run same software on your own instance.
This "uninformed mom-and-pop" is potentially compiling the client application from source, but can't do basic math to compare tarsnap's pricing to the top 20 or so competitors that rank above tarsnap in SEO?
We also have .edu / student / nonprofit discounts. Email us.
Finally, Debian and FreeBSD project members get free accounts. See the committers handbook, etc., for details.
[1] Whenever we lower our prices, we increase quota on existing customers to "normalize" them to the new price/GB. If you do nothing, your rsync.net account just grows over time due to this.
I've been using Tarsnap for 10+ years. There's some Linux stuff getting backed up, configs and such. It costs next to nothing for this kind of usage.
While on the price, patio11 (Patrick) has written an article about tarsnap’s issues more than nine years ago (April 2014). One of the suggestions was to raise prices, IIRC. It’s a long post, but you can read it [1] and the HN post [2] from that time.
I have written about this some time ago if you’re interested: https://www.franzoni.eu/ransomware-resistant-backups/
So it's not just ease of use. It's actual _functionality_ to me - getting from raw object storage to a fully working, attack-resistant backup strategy, is not trivial; hence, comparing tarsnap (or rsync.net, or borgbase, or whatever) to B2 or S3 makes little to no sense.
You _could_ compare it to crashplan or backblaze personal backup if you like, but IIRC those don't work for *nix systems, only for Win and Mac.
Those restrictions are enforced by the service.
Thought it used readonly features of S3/Glacier or something..
There's also a service like rsync.net where you can just rsync to the destination and they do the versioning and so on for less than 10th of the cost of tarsnap.
Plus "written by cperciva and heavily battle tested by Serious Sysadmins" is a feature I couldn't recreate myself - notice that while there was an outage, part of the reason for it taking a while was a conscious choice to take a much longer path to resolution than bringing up the previous server in the name of paranoia. Paranoia about data corruption is a nice thing to have in a backup system and something I'm happily willing to trade-off uptime for.
However: For backups of bulk data then, yes, it's going to be relatively expensive. I wouldn't put e.g. my media backups on tarsnap, but "use tarsnap for your git repositories and other high value data, and something else for the rest" is both perfectly doable and an approach I suspect cperciva himself would endorse.
The tarsnap architecture still does more things.
You're welcome to feel that you don't need those things, but that wasn't my point.
As Actual Serious Sysadmin that Actually Manages Big Systems for Living that screams more lack of preparation than anything else.
Yes you should be careful but you should also have procedures in place and know the system well enough to trust it. And the fact is that the "boring" architecture of RDS DB instead of that S3 database abomination thing would just start right up if master DB server failed.
It honestly looks like a trap many intelligent people fall into where they turn their cool-but-ulimately-flawed mental excercise into bedrock of the product. I don't want to use baby's-first-database on my production servers (I'm looking at you Lennart Poettering and journald) and I don't want my data/metadata stored on some experimental one.
Without agreeing or disagreeing with those, "I'm not going to trust the filesystem on the existing machine" was the choice I was talking about.
[1] In case of an emergency, you will always be able to get back your data from tarsnap at a blazing rate of 50kB/s https://github.com/Tarsnap/tarsnap/issues/333.
How many of the world's best and brightest are doing all sorts of busywork? At least Colin has some time to do whatever he wants to do while running tarsnap.
It won't, though, because of the points mentioned by the post you're replying to. It's been 15 years; tarsnap is as popular as it's going to get.
I don't find that that logically follows from making bank. Not everyone who makes bank is a positive influence.
Tarsnap does provide value, even if I think it's less than its cost: I'm just commenting on the general case that making money would mean you're providing good value