What I would do if I ran Tarsnap (2014)
kalzumeus.com
kalzumeus.com
Tarsnap is technically impressive and was reliable software, but the billing system requires an unpredictable manual process requiring two credentials held separately in most orgs. Colin has told me in private email that customer deletion is a manual step not taken lightly, but I didn’t feel that one unscheduled manual process was fixed by epicycling on another one.
I migrated away several prod installs to pay more for predictable and automated billing. Even with usage billing that’s not easy to predict, the date of next intervention is printed on the back of the credit card. (Though really, it does cost less - the engineer time it takes to manually add funds costs significantly more than a picodollar.)
And it wastes my time when I get emails about billing renewals. And the billing person at my company doesn't see the email unless I forward it.
Tarsnap doesn't even have any tracker on its homepage. Tarsnap has had the same basic pricing structure for the past ten years. It does one thing and does it well. I hate the pursuit of growth and everything that comes as a result: bloat, shiny landing pages, a/b testing, conversation rate optimisation.
Reminds me of the adage of a Mexican fisherman.
> “Afterwards? Well my friend, that’s when it gets really interesting,” answered the tourist, laughing. “When your business gets really big, you can start buying and selling stocks and make millions!”
> “Millions? Really? And after that?” asked the fishermen.
> “After that you’ll be able to retire, live in a tiny village near the coast, sleep late, play with your children, catch a few fish, take a siesta with your wife and spend your evenings drinking and enjoying your friends.”
> “With all due respect sir, but that’s exactly what we are doing now. So what’s the point wasting twenty-five years?” asked the Mexicans.
I disagree. I think the world would be a better place if many millionaires had decided to retire instead of trying to extract as much wealth as possible from others.
From the posts I've read recently it seems like it does one thing and it does it by renting a single EC2 server that will bring the service down if it needs to reboot, and it does it by reselling S3 at 10x the cost.
It's funny because maybe it's a good service but going by HN, it's not reliable or cost effective.
That's a cheap shot, it's been as reliable as the underlying fabric, the only thing that really stood out for me is how utterly weird HN is when it comes to determining what constitutes reliability: no data was lost other than a tiny bit that was in inbound transit which can still be recovered (and which you could not realistically protect against). Note that this is a backup service and not something that is normally found in your primary business processes. As such if it stores the crown jewels safely, allows for them to be restored if and when needed and doesn't leak them in the meantime that's mission accomplished.
> or cost effective.
That depends on your use case, and not everything is about cost. The way it is set up I think the trust factor that even Colin can't read your data and that there will always be a way to get your data back out if you should need it is what matters. Backups that don't work are a net negative, a backup that does work can be, given the right circumstances, absolutely priceless.
If you want more than 3 nines availability neither Backblaze nor Tarsnap nor any other outside service would be able to serve your needs.
Also, parent is talking about ingestion. If your backups aren't configured well and the backup process fails, then your backup may not end up durable.
I also don't think your definition of reliable is generally recognized, which I'd generally call durability. I wouldn't say the scenario above is a durability failure, but an example of the consequences of poor availability.
That's a fallacy right there. Your assumption should be that any service you rely on will be down once in a blue moon, and possibly for a day or even longer.
> Also, parent is talking about ingestion. If your backups aren't configured well and the backup process fails, then your backup may not end up durable.
Yes, indeed, you need to do your work and you don't get to point at others for not doing it right.
> I also don't think your definition of reliable is generally recognized, which I'd generally call durability.
Reliability, durability and availability are all industry terms and have very clear definitions. These are not the same definitions that you would use in ordinary conversation with laypeople but when we're talking shop those are definitely allowed.
> I wouldn't say the scenario above is a durability failure, but an example of the consequences of poor availability.
No, it is a consequence of poor engineering on the part of the user of the service, and is a completely different issue. You engineer your service to ensure that your assumptions hold true and if you fail at doing that your service will fail. When is then only a matter of time and combination of circumstances, but fail it will.
The entire service going down for 24 hours due to a reboot is not a consequence of poor engineering on the part of the user. A production service which people rely on for critical data failing on the _textbook_ example of running a live service is poor engineering on the services part.
Tarsnap makes very particular guarantees, if you look into that then you'll realize that for some applications it is very useful and for other applications it is not, or that you may have to use not one but multiple backup services to be able to serve all your needs. This can be costly.
Availability and correctness are fundamentally opposed. The word "reliable" is contextual.
A backup service that is always available but serves up garbage is not as reliable as one that serves me the correct data, but only on Mondays.
And that's comparing to S3 Standard. Infrequent Access is 2x cheaper than that, and Glacier Instant Retrieval 6x (if your files aren't tiny).
I could imagine purchasing that software for a one–time price, running it myself, and paying AWS for storage. But then I’d have to monitor it, troubleshoot outages, maintain things, etc, etc. Or I could pay someone else to do all of that. I’m not currently a customer, but I know which I prefer.
> But then I’d have to monitor it, troubleshoot outages, maintain things, etc, etc.
None of these solutions free you from having to monitor your backups, including Tarsnap. Tarnsap requires setup on your server. You have to make sure it's running and backing up the correct files. And you really should verify you can restore a backup.
I'm really not sure what Tarsnap adds over these aside from saving you from having to sign up for B2 to S3 and punching in an API key.
Tarsnap does full deduplication across all backups for any given "machine", while still letting you independently remove any snapshots you like. i.e. no special "full snapshot" that must always be kept around, and no need for multiple full snapshots that have no deduplication between them.
Github resells a free product with a fancy UI. Stripe resells visa and mastercard by adding a 5x surcharge to card transactions. Steam resells stripe by adding a 30x markup on that (it doesn't, it uses worldpay but the point stands). Calendly resells an open calendar for $12/month.
This is a reductive argument that doesn't really show why people pay for services. Tarsnap doesn't resell S3 at a 10x markup, it sells a backup service for $0.25/GB/month.
That said,
> it does it by renting a single EC2 server that will bring the service down if it needs to reboot
Yeah, and honestly it's pretty unbelieveable that there's not _two_ servers.
This is a very unfair take, based on basically nothing but the single recent outage report it seems. Tarsnap is generally liked by HN and if you use it, you will know why. Its a great service technically, and _extremely_ affordable. I was a happy user for years but have moved to local time machine backups with B2 offsite replication just because its seamlessly integrated into my NAS (and is also very affordable).
So that kind of thinking is why every second thing I’d like to hobby-use is priced as a free trial with one missing crucial feature, then $300/mo. It might be rational even, but I’d expect the actual utility does have a negative term for I’m going to hate your service with a fiery passion (and probably also you) if you do this. (Cf recent discussion on customer “support” chatbots.)
> let’s boil it down to a simple intuition: people getting more value out of Tarsnap should pay more for it
That’s basically the definition of a discriminating monopolist and what gets you airline-style inscrutable pricing and the SSO tax, isn’t it? Again, screw that noise. I can’t really motivate this well, but to a first approximation I (a) dislike seeing pricing disconnected from costs; (b) cannot resist the urge to minmax thus cannot help disliking people who make it more difficult than it absolutely needs to be. Note that this does not contradict TFA’s conclusions, unlike the previous point, and another argument in it is actually very close to (b); it’s this specific argument for the conclusion that I’m disagreeing with.
> You know how every ToS ever has the “You are not allowed to use $SERVICE for illegal purposes” despite there being no convenient way to enforce that in computer code?
Yes I do, and I feel basically the same way about that as I do about stupid laws everybody tacitly agrees not to enforce: it erodes the whole edifice of a law/bureaucracy-based Enlightenment society. If you’ve put it in writing and not planning to sue over violations, you’re lying to me.
You seem to be under the impression that if people didn't charge so much money, you'd have stuff cheaper. That's not true - what would actually happen is you'd just have less stuff, because people wouldn't build them in the first place.
If someone can afford to create software and run it while charging far less than it's worth for your benefit, then wonderful, but it boggles my mind that you somehow think people owe you this service. Do you also expect people to go into their office and tell their boss "actually, I don't need such a high salary, go ahead and lower it"?
> That’s basically the definition of a discriminating monopolist and what gets you airline-style inscrutable pricing and the SSO tax, isn’t it?
You think it's discrimination to ask people who use more of a service to pay more? You think if an enterprise is using something for business purposes it's not ok to ask them to pay more for something than if a user is using it for hobby purposes?
> If you’ve put it in writing and not planning to sue over violations, you’re lying to me.
That seems both unworkable and kind of ridiculous. You're basically advocating for a "zero context" policy around contracts, in which people don't have any choice whether to sue someone. Even if it's a minor violation that isn't worth it to sue over, or a violation that they decide is ok for them in that context. Why would that be better than the alternative?
The point is "enterprise" plans are generally much more expensive relative to the use of the service or the extra feature (the most common extra features like SSO and auditing are generally cheap to provide, both in terms of resource usage and cost to implement and support). So while they may use the service more they wind up paying proportionally much more for it (the assumption being that theu are getting much more value from the core features). This is price discrimination, whether monopolistic or not (which is absolutely rife in B2B products). I'm not going to comment on the morality of it, but it can be very frustrating if you don't fit into the buckets the pricing structure assumes (the other thing that is common in B2B that pisses me off is "call us for a quote". Generally a lot of tools seem to have an overinflated sense of how much value they are providing me, but I am a little unusual in that I use a lot of different tools but not heavily, being a generalist in a small startup)
> You think it's discrimination to ask people who use more of a service to pay more?
Terminology confusion. In microeconomics, a “discriminating monopolist” is one who engages in “price discrimination”, that is to say providing the same service to everybody but charging each customer as much as they are willing to pay for it. This has nothing to do with the social justice usage of “discrimination” (except in the broadest sense of discriminating one kind of thing from another and acting on the result).
> You think if an enterprise is using something for business purposes it's not ok to ask them to pay more for something than if a user is using it for hobby purposes?
Insofar as they receive the same service (no or the same SLA, etc), or even insofar as the markup for the latter case is disproportionate to the actual costs, I think it is bullshit to do it. How much bullshit is acceptable, both in life and in selling technical services, is to be decided (certainly a nonzero amount), but at the very least I think it should cost the service provider some measure of trust and thus create a preference towards providers that don’t do it.
In the case described, all of this is not that huge; my strong reaction was mainly to TFA going from “fair” pricing to price discrimination.
>> If you’ve put it in writing and not planning to sue over violations, you’re lying to me.
> You're basically advocating for a "zero context" policy around contracts, in which people don't have any choice whether to sue someone.
I put it in absolute terms myself, so I guess I deserve this a bit, but still, no, that’s not what I was trying to advocate. Note that the original article was talking about putting in a clause the service provider would not make the slightest effort to enforce or would even be completely unable to.
Consider the law example I also gave: it’s one thing to allow for some discretion from a prosecutor; it’s another for people to know that nobody was ever convicted of a crime that’s technically on the books. I understand this is a slippery slope argument and those are always suspect, but I think it’s fair to say that this kind of neglect for one piece of law does tend to spread to other parts of it, at least when people are doing it consciously and not because they’re unaware of legal arcana.
In contracts, the situation is somewhat better because the punishment is not that dire. It is also worse because usually the only explicit penalty is termination of service, so in essence you have a bunch of rules which are all nominally enforced the same way except the provider will enforce some of them and won’t enforce others, at its discretion. (Naturally, I also think that the very common arbitrary no-recourse termination clause is completely asininine. At the very least, I never feel safe to rely on a service that uses one.)
> You seem to be under the impression that if people didn't charge so much money, you'd have stuff cheaper. That's not true - what would actually happen is you'd just have less stuff, because people wouldn't build them in the first place.
You are to some extent right, of course. To some extent, though, some of my experiences with price discrimination (see above) tell me you’re also kind of wrong.
For example, I can’t recall an airline or railway that on transitioning from all refundable tickets to a split of non-refundable and refundable tickets ended up making non-refundable tickets measurably cheaper (in the long run) or pricing refundable ones in a way that’d correspond to any realistic fraction of refunds. Similarly, increasing limits on luggage never seem to make things any cheaper. That looks like price discrimination, not adjusting for costs or anything related to actual costs.
The SSO thing, as another example, looks the same to me, and even actual service operators in this forum have said that it actually is. Granted, I’ve heard horror stories about the integration and support costs, but if even a couple of operators settled on a single very strictly defined subset of SAML, OAuth or whatnot, saying it’s their way or the highway, the implementations and the integration consultants would likely come. It’s just that nobody has the incentive to, and systems remain insecure as a result.
> If someone can afford to create software and run it while charging far less than it's worth for your benefit, then wonderful, but it boggles my mind that you somehow think people owe you this service.
First off, TFA was advocating for eliminating (and settling for deemphasising) a class of service that was at the time manifestly cost-effective to operate (though could become less so after scaling up).
Second, to a degree, yeah, I’d actually be happier with a service that provides no free or severly subsidized options at all than with one that has a free option, then a huge cliff, then a heavy-profit-earner option.
If this means a world where there are no accessible services of that kind, sure, because except for very costly services such a world is unstable: at some point somebody will build an accessible alternative, as long as they are not pushed out by a subsidized free option. This applies to tech-oriented services first of all, although when the expertise gap between wanting and building is larger such situations do sometimes persist (CAD, CAT/TM, arguably photo editors).
I don’t know how long is the long run for you, but Delta is right now offering RT to Vegas for $444 non-refundable and $638 refundable. I didn’t go check all the other carriers, but I definitely recall tickets with more flexibility (sometimes including refundable, other times creditable) being sold at higher prices than less flexible tickets.
As a buyer that makes sense to me; it costs money to provide flexibility; it’s worth it to some buyers but not all, so buyers can opt into the level of flexibility that matches their preferences.
Some airlines in my memory went from only having refundable tickets to having both types, and for some reason the price of the nonrefundable ones after the change ends up being about the same as the price of the (refundable) ones before that.
Also, well, I don’t know what Delta does—for some airlines, the advertising term “flexible” means some sort of option to change your flight as well as one to refund the ticket, being a tier above “refundable”—but if the buyer requesting a refund is the only possible cost, we should have roughly $444 = $638 / [1 - P(refund)], whereby P(refund) = 30%, and that feels ridiculously high. Not implausibly high (I’m used to a difference of 2x or more), but high enough that I’m dubious.
Was this over a long enough period where inflation reduced the value of the nominal currency and where jet fuel prices and costs of financing their fleet could have risen significantly?
It means I can always get a seat, I just have to pay more. It means businesses subsidize mine and everyone else’s flights.
When I’m travelling the world and have to use a train system with fixed prices, I don’t like that I have to book many days in advance or else the tickets are sold out. Just raise the price! Let the rich pay double so it’s cheaper for everyone else, and anyone who _really needs_ to use the service can weigh the costs and decide to pay more.
Must be nice to always have the money to pay more to get service immediately, but I don't think that's something we should strive for.
I don't know the inside baseball, but if I was @patio11, I'd be more than annoyed by this. I might ratchet up to lightly insulted, given how master-of-the-obvious some of the advice is.
Patrick's advice is very good: for Patrick. But for Colin it was more of an exercise in how you could run Tarsnap, not how he should run Tarsnap. Meanwhile, Tarsnap is still in business many years later, has happy customers and as far as I know happy people running it.
The detail that really pets me backwards is the whole "if you're here, you already know what this is for and how to use it" vibe. It's not just Tarsnap; many aspects of the OSS world uphold this most unfortunate tradition. I believe with all of my heart that it is unnecessarily hostile and antisocial.
Many of the founders and teams I've mentored over the years have experienced me spending a large amount of time and effort explaining the importance of clearly stating why a thing exists and why you should use it; how it works and how it will not create additional unknown risks; what tools it is intended to replace... as the barest civil minimum.
This is just my opinion, but it's a hill I'd die on if necessary.
Founders that are in the funding cycle would obviously benefit from Patrick's advice they are going to increase their chances at commercial success that way. But if commercial success isn't your #1 goal there are other viable paths based on your priorities in life. And that's a hill that I'd die on if necessary because I don't believe that everything has to be run as a marketing driven profit maximization engine. The OSS world upholds that tradition pretty good and I'm fully supportive of it, there are many paths to a happy life and not all of them run through a wallet.
I want to clarify, though, that in talking about the unfortunate number of OSS projects which convey the "if you're here, you need no explanation", I am absolutely not hung up on profit motives.
Instead, I am talking about the times where you are searching for a technical solution with only a vague intuition about the shape of an approach and some faith that it might exist. And you land on a project homepage that contains so little context that the only reasonable reaction is to have serious doubts that you're in the right place.
You're right, I do care. I have spent a huge amount of my life publishing OSS under different licenses and I want people to experience maximum benefit from that effort. It boggles my mind that people work so hard to build useful tools and then appear to actively gaslight potential users into feeling inadequate instead of attempting even the most basic onboarding advice.
Folks are not obligated to make their code or projects accessible, but if you can't justify even a simple "this is why this exists" statement somewhere prominent that conveys intent without inference or telepathy, I believe that they give up their right to grumble that nothing they do matters and nobody cares.
So what you are seeing is entirely the expected outcome!
I used to believe this. And maybe at some point it was, but then the world, and the sector, was absolutely _flooded_ with script kiddies.
Now things that immediately confuse people that don't at least have a halfway decent idea of what they're about satisfy me.
Your point is well taken! I appreciate also that you're not making an absolute declaration that advice is untrustworthy. I spent a long time as a professional advice-giver too, but if you lashed yourself to the mast and ignored my advice that there was a CBC padding oracle in your session cookie implementation, you were plotting a course to the ocean floor. Running a business means being careful which advice you let in and which you don't.
My only nit here is: between this and your other comment on the thread, I don't think Patrick's advice here was unwelcome. I have a vague recollection of Colin being asked first. Also: if anyone was going to ask before publishing unsolicited advice about someone else's business, it would be Patrick. So the Bingo Card thing is a bit of a low blow. Also, easy to make fun of when you exclude the context that he lived off that Bingo thingy for years.
Unlike you I don't know either person in this discussion personally and you should construe my comment strictly as beating my favorite dead horse, "be wary of smart-sounding advice from people whose livelihood is built on sounding convincing to people like yourself."
Mostly I was just moved to comment because I got the vibe that Patrick was coming across to you as condescending, and I know he was working hard not to come across that way.
The OP alluded to tarsnap being the best, but didn't really explain why that is. Rsync too sounds like it'd be fun to use, but it's 3x the price of backblaze b2 and has a minimum order size of ~$10/month
FWIW I (eventually) moved to rsync.net from BB because of that time BB put some JS on their web interface that sent... was it filenames to Facebook? Something like that. I'm using Restic so it was completely painless.
They do for Windows: https://www.rsync.net/resources/howto/windows_backup_agent.h...
A question: The article is from 2014. So almost a decade has passed. How, if at all, would it be different if it was written today?
If anyone knows of changes that were made, please share.
IMO Tarsnap is about being the perfect lifestyle business for an extremely technical engineer, it's something I aspire to build one day also. Billing? Just the most simple model possible. Sales? Nah. Marketing? Nah. Fancy website even? Nah.
Just customers that know what they are buying, are happy to pay for it and software that is exemplary.
https://news.ycombinator.com/item?id=7523953 (311 comments)
But for me tarsnap is convenient as extra offsite backup for valuable stuff. Price per GB is not the only factor and different people have different priorities.
Tarsnap is perfect tool for geek2geek backup.
If you position yourself in a new category you get crushed by big companies with lots of money, marketing teams, etc.
Not only that, you will become miserable because you compromise yourself for money. Not everyone wants to build multimillion company.
Reaction #1: Whoa, that's a lot of assumption that serious professionals operate within incorporated businesses, anybody else is frivolous.
Here's an alternative framing. Incorporated businesses don't prevent catastrophe, they just make sure there's nobody to blame when things go catastrophically wrong (e.g. https://prospect.org/health/2023-07-29-shock-treatment-emerg...). They also go off on strategic misadventures, destabilizing the product for existing customers while chasing more lucrative ones.
A caricature, yes. But a more accurate caricature I think than the one in OP.
---
Reaction #2: the Opticon page he glowingly links to no longer works. They probably switched hosting providers at some point, and that didn't affect their ability to do business even if it caused them to forget some of their past.
Backups as "ability to continue to operate in the present" are a very different thing from backups as "providing the future access to the past." It's absolutely true that any company will see more profit in the former over the latter. That seems like a weakness of capitalism more than one of Tarsnap.
I might agree with about half of the points in them, but it still grates somehow.