37signals Says Goodbye to AWS: Full S3 Migration and $10M in Projected Savings | Hacker News Reader

37signals Says Goodbye to AWS: Full S3 Migration and $10M in Projected Savings

systemadministration.net

101 pointsbluedino54 comments

Something an older and wiser programmer taught me, is to think of the infrastructure costs as a per-user cost. These numbers look enormous next to my bank balance, but if you save 500k per year to service 1 million users, it's nothing to your bottom line.

Meanwhile there's the opportunity cost of moving. All of the people who put effort into this migration, who could otherwise be building something revenue-generating. I think that's why in many companies cloud costs are a problem, but never enough to make it high up the backlog.

Building revenue generating things is hard (you have to figure out new things that people will pay for). Writing new features also requires generally different skill sets than operating servers. Running software in a datacenter is a more defined problem with straightforward ROI.

> building something revenue-generating

"a pound saved is a pound earned"

Saving 2 million a year for a company their size is guaranteed more money in their pocket. Revenue generating features may or may not do that.

More money in the company pocket. More work for the people who did the migration and now maintaining their own cloud. Then when the people who put this together leave, the rest are stuck with one of a kind cloud.

Maybe a good idea, but if it's only saving money that can only happen 1 time, then people expect the same cost going forward ...

They already had a team of people that handled this stuff. AWS requires a lot of maintenance on it's own sort.

I can’t speak to 37signals specifically, but most on prem tech is pretty standardized.

There’s nothing particularly bespoke about throwing RHEL on Dell servers running in a 3rd party managed white glove data center.

They open sourced the entire deployment apparatus that they built to do all of this - it is simple to use. They'll have no issues maintaining it or finding new people if/when needed.

And, if you bothered reading, there's significant annual savings, into perpetuity. Because aws is a racket.

They've talked about this at length in both blog posts and podcasts. The gist is that all the Amazon cloud stuff still took effort, their ops team is essentially the same size and they're actually saving all this money.

LOL, this has been my experience working with AWS. "We can fire 3 sysadmins, because Amazon will handle all this ops work for us!"

But then also: "we need to hire 3 AWS Cloud Engineers, because administering AWS requires a bunch of ops work."

Decisions made by the uninformed aka idiots.

There’s a good case for aws, but if that’s the narrative… it’s time to be replaced by someone or something else

Also the AWS cloud engineers are so much more expensive because the AWS learning treadmill runs at a 10x speed compared to the on-prem one.

Our cloud engineers spend so much of their time simply keeping up with all the things they need to keep up with, even for workflows that have been running just fine, for years.

The on-prem engineers also need to be up to date, but the changes are more measured and easier to manage.

If you’re comfortable where you are as a business, and not looking for more than steady revenue growth in your core market, then finding efficiencies is a fine use of time.

I know that mindset is antithetical to the VC “make every chart a hockey stick” culture, which is well-represented here on HN. But actually tons of businesses are run that way.

I agree.

Tech is seen in this very biased lens because Tech seems to be the only place imo that VC's are genuinely interested in seeing very high growth in.

They want to know that they have bet on the next google and they want to know it fast.

Actually, it is real funny that we are talking about this on a forum which is funded by y-combinator/ a lot of this VC nuance.

I don't know man, if I can be honest or not, but I was thinking for a moment and the whole idea of tech seems to be a lot more overflated man... , as a dev, I was just thinking about the influence of money and I was genuinely just wondering what is the fastest way I can make money and It was probably creating a proprietory product and selling the startup / getting VC fund-ed and not organic growth.

I am still very nuanced though, on one hand, both of these go against my personal philosophy, I myself would "love" to open source (preferably MIT but hey AGPL would also cut the deal like the recent redis...) my product and probably have no 0 VC funding.

But on the other hand, I am not sure, A lot of what I feel like building, is probably immature, its better off hidden. I am using Ai to build a lot of stuff for now, and I am not quite proud of it.

I feel, that as someone who has made very meaningless contributions (I guess?) in my past and while I was writing this comment, I was getting this vague sense of an issue being popped up on whatever software I create with "WHY DOESN'T IT WORK" and no issues or nothing.... I don't know, it makes it less lucrative to open source, quite frankly any product.

Maybe I want an AI which can genuinely resolve such very meaningless issues but I also don't trust AI, what if there are some issues which are actually good and AI filters it wrong.

I don't know man. I was thinking about this yesterday and I just realized that I am not thinking this much, I would do whatever I think in the heat of the moment.

Somehow no one was ever making this argument when companies were spending tens if not hundreds of millions of dollars to move their on prem workflows to the cloud.

At that point it was all “you will save the money because you will need such a smaller ops team”.

But once people actually did make the move and noticed that their ops teams didn’t get any smaller, then we started being gaslit into “well the cloud was never about saving money…it was about <fill in the blank>”.

Thenper user calculation is a terrible one. It essentially justifies any inefficiency as long as it’s not some arbitrary multiple of your user base, but inefficiencies add up.

The correct approach, that you even allude to, is to do a complete cost-benefit analysis. And the costs and benefits should include non dollar factors such as time, risk, control, reputational risk, data being available to 3rd parties, etc.

There’s no reason to divide your cost/benefit analysis by your user base at all. You can simply compare the absolute and total values against other potential initiatives and stack rank them based on their net expected benefit.

Just goes to show how greedy AWS has become with pricing now that everyone is locked in. If anything they should be able to use their scale to make it cheaper than doing it on-prem and all the costs that go with that. Strangely at the time it did appear cheaper when we all moved to cloud for these so called savings a decade ago.

it does appear cheaper because you can handle the highest workloads for just a fraction of the total cost if you had to host the highest workload yourself.

Like lets say my company gets a very huge spike and lets say I wanted to be prepared for it for all times in baremetal, then I had to had a lot of vacant free metal.

But I personally believe that a dual strategy should be used, a minimum amount of stuff should be baremetal for the average traffic and only for huge spikes should cloud be used.

Except, clouds seem to be a lock-in, so people prefered the clouds untill the cloud started raining and asking them for a big ton of money.

> just a fraction of the total cost if you had to host the highest workload yourself. Like lets say my company gets a very huge spike and lets say I wanted to be prepared for it for all times in baremetal, then I had to had a lot of vacant free metal

Huh? Autoscaling turned out to be a myth. Even in k8, you cant make users wait for nodes to come online, so you have to always have spare nodes waiting. And the spare nodes must be in proportion to regular spikes you expect and additionally any unexpected spike that you estimate. Why would that be different from having extra free metal for much cheaper and simpler? Along with an easier time finding infra people who can manage that as opposed to finding expensive talent with the expensive bloat of technical knowledge that AWS today requires?

Lets face it - this is the enshittification of infra brought by the lock-in Amazon was able to lure the orgs into. First, they locked-in everyone. Now they are squeezing everyone dry.

I am not sure but aren't lambda functions literally autoscaled? maybe you are mentioning that not everything can/should be a lambda function...

and even forget about lambda function, cloudflare workers also do the same thing. I have hosted many websites on it, and the timing is literally negligible / even faster imo than trying to self host it / bare metal.

I know It feels really shitty to move to JS just for such a huge boost IMO to get cloudflare workers but boy I am in love of cloudflare workers and I know cf has some bad sales tactics but it was just this one off instance or very rarely and I think cf was in the right on that one, all be it, they miscommunicated.

Seperate the art from the artist. Seperate the sales from the tech team (for cloudflare), and you would see that cloudflare is literally great.

Though I used to believe that cf workers + r2 was best but now thinking storage should probably be done r2 + wasabi

This old tale gets always told, and it is still a lie. Since AWS exists, every five years someone calculated the cost of AWS for our PHP stuff. And the results were always the same: we could have 4 to 7 times more CPU, ram, storage and bandwidth when we just rent servers somewhere. AWS is ridiculously expensive and renting servers was always the cheaper and better option (yes, we calculated the price for our system admins). Never was any spike of usage, or growth, a problem for our servers and software.

wait what?

I think you might've misunderstood me.

By what you mean, calculated the cost of AWS for your php stuff, and saying that it was 4-7x more expensive.

Let's say you are using AWS, and your website suddenly gets 1000% spike or even more suddenly, then AWS can still host it but if you were using bare metal, you would've had to manually scale.

Other than this nice benefit of AWS, there is none other benefits aside from maybe not managing servers but with things like coolify, there isn't much managing servers I guess...

I am genuinely interested how you are quoting 4-7x, I know cloud is expensive but sheesh.

Also, one other aspect I have always considered cloud to be cheap is probably storage, maybe please elaborate on that as well.

And also what hyperscaling techniques are you currently using without AWS in case you get some really really huge unexpected traffic because that is the stuff AWS was meant for tbh

The really huge traffic spike is the lie.

Most of the time you have a pretty good idea how much traffic you will get. Only once in 20 years we noticed that our servers are getting too much traffic, because too much people clicked on our advertising. So we stopped the ads, rented more servers and started the ads again. This took half a day and costed us a few hundred euros and that was that.

We aren't doing any hyper scaling and almost nobody needs this. We start PHP workers on a few servers and put the database on a beefy Server. What a beefy Server is, changes over time. In 2005 this was like 4 cores and 16 GB of RAM. Today you get like 24 cores and 128 GB RAM for about 200 € per month. On machines like that you can serve data for millions of users. AWS charges ridiculous amounts of money for servers like that. Also bandwidth pricing is a joke and always was.

For pricing on non-aws "clouds", look at a hetzner, ionos, maybe Strato. There is also digital ocean, vultr, ovh and many others.

> The company estimates it will save more than $10 million over the next five years, reducing its annual infrastructure bill from $3.2 million to less than $1 million — all managed by the same technical team that previously handled the cloud.

It would be cool to get more into this. One of the reasons my (much smaller) org insists on S3 (or possibly R2) is that they feel they are paying rent for the infrastructure management that they would otherwise have to hire for. If there was data out there that showed that an org could switch to a less managed infrastructure without pulling people off of other projects to fill in the gaps it would be an awesome selling point.

This is exactly what we do (but now you mention this I think we should spell this out much more clearly to our potential customers)

We move companies onto bare metal, effectively giving them their own private cloud. This about halves their costs, doesn’t impact their engineering team as we handle the migration, and then we also include ongoing monthly engineering time to support their infra team. Overall it is a win-win for everyone (well, except AWS)

(https://lithus.eu)

by bare metal, are we saying things like hetzner, ovh? or full on, renting servers spaces themselves like railway?(I am not sure if it was railway or render) did a while back

It’s both, we can rack up hardware if there is the need. But building, racking, and financing servers is a fairly well solved problem (especially here in the EU). So, in general, if we can avoid solving that problem ourselves we will.

I’ve been on a team managing on-prem data centers for nearly 2 decades at a larger place.

At one point they did a study to see if it would be cheaper to go to AWS, and the conclusion was that it was cheaper to run our own. We have some stuff in the public cloud now, but it’s still more expensive than running our own.

One of the biggest things I see, at least on the surface, is that there are certain sunk costs with on-prem. If you can do more with the same hardware (licensing costs aside), it really doesn’t cost much to add a VM or container. Whereas with the public cloud everything comes with a price tag and there is a much greater cost to leaving your dev server running when not actively using it, or that pet project that is helpful, but not in a way that’s quantifiable in dollars and cents. This is how companies end up with old laptops in closets and under desks running critical tools.

Honestly when you’re a small company or starting up, when you’re unsure of how much, if any, usage your product will receive, it makes sense to start on the cloud.

And then as you grow, it makes sense to periodically check in as to whether the cloud vs on prem calculus still holds for you or not.

There is a huge place for the cloud in our eco-system. And the cloud companies have been an incredible enabler for startups especially.

The problem lies with the lie that has been peddled by the cloud companies and many others who know about little other than the cloud, that it’s always the best solution, when in reality there are many scenarios where it’s not the best solution even when you consider the cost of migrating from the cloud to on-prem.

TLDR: Whether you’re on prem, or in the cloud. Regularly assess whether your workflows will work better and/or cheaper in one or the other. With the cloud you do have the added caveat that costs have been raised significantly whereas on-prem costs have been fairly stable (and actually dropping) over time, but again, that can be a factor in your analysis.

18 PB is not a trivial amount of storage. I think what most people always underestimate is how much engineering effort it takes in the background to keep such big infrastructure up and running reliably.

I would like to have a detailed total cost of ownership calculation. Of course, renting hardware - let's say in Hetzner or some other cheap provider - is always cheaper than AWS or any other major cloud provider in simple 1:1 pricing comparison. So that is nothing new. But the pure CPU/RAM/SSD costs is just a fraction of what it really costs to run such infrastructure.

It’s 2000 8tb SSDs ($1m), or a thousand 18TB disks ($350k)

Times something for redundancy / backup.

The hardware investment is back within a year. This is about what I would say many aws services cost. After a year, you’re just giving away your margin to Amazon.

The original story for EC2 was ephemeral, short lived/on-demand compute. This was during the time when map-reduce was all the hype.

Related It's five grand a day to miss our S3 exit (187 points, 1 month ago, 176 points) https://news.ycombinator.com/item?id=43489698

Do note that it took them ~2 years to move off of S3. The ROI of getting off of compute and hosted services is often there, but S3 is still pretty great from a business perspective.

I’ve become increasingly frustrated with AWS because of this. They used to have a culture of providing constant price performance improvements. Not anymore. Every release has questionable improvements (for example, switching from r6 to r7 family of instances is more expensive with theoretically better performance but you probably can’t actually switch and save money). S3 costs haven’t gone done in a long time despite plummeting storage costs.

Very excited about the work done by 37Signals to encourage moving off.

Good for them. It is becoming tiring paying rent seekers to run our own software.

>>all managed by the same technical team that previously handled the cloud.

Does this imply they haven’t hired any additional personnel? I would’ve thought moving everything in-house would need more hands on deck—for stuff like security, keeping things up and running, and all the behind-the-scenes stuff AWS usually takes care of.

If they managed with the same team, that’s darned impressive.

Much of the work "security, keeping things up and running" that AWS does behinds the scenes is bolstered due to the nature of running a massively complicated multi-tenant cloud. Doing exactly what you need in-house is orders of magnitude simpler, but has a deeper knowledge requirement. The cloud is a powerful way for companies to commoditize skill.

I’m not sure where this idea that the cloud doesn’t need skill to manage comes from.

Cloud engineers are extremely highly paid because they need to be extremely skilled.

No one running stuff on prem is compiling Linux from the kernel and then building a few packages onto the hardware they built manually after designing a networking infrastructure in their office building’s basement.

You’re paying for enterprise software with support and best practices and sensible defaults provided to you, running on enterprise level hardware with support, best practices, sensible defaults configured, running in professional data centers where the data center firms manage maintenance, repair, etc while also managing physical security and other risks.

If anything, the support you can get from all the on prem providers is far superior to what the cloud providers are giving you.

I see something analogous happening in about 5 years with LLMs when people realize they can get 80% of the value for 20% of the cost by using an on-site model rather than paying out the nose to OpenAI.

I'm guessing they probably could not have scaled as far and fast as they did without AWS. To me, this scalability is the cost of doing business as a younger growth company. As your workload becomes more predictable, this additional cost becomes less beneficial. I've come to see that often companies say they are saving money by moving off AWS, but much of these saving come as part of rearchitecting or optimizing their systems.

I remember when the discussion was how to escape the lock-in by hardware vendors such as EMC and Dell. I believe a lot of initial desire to go cloud was the idea it was easier to migrate from AWS to Azure to GCP if necessary. Now we know that such migrations are multi year efforts. Exactly like in the bad old days.

> I remember when the discussion was how to escape the lock-in by hardware vendors such as EMC and Dell.

Somehow I've never heard about EMC/Dell in a context of the lock-in.

All the times I heard about lock-in was a vendor trying to get me to avoid locking into their competitor and lock into them.

Twenty years ago if you bought a multi million dollar SAN from EMC, the rep would tell you the included support would be free if you also bought X Y or Z. If you bought a data center widget from another vendor, it may void your support agreement etc.

Why was not the original post posted by DHH, instead of the advertorial?

https://www.linkedin.com/posts/david-heinemeier-hansson-374b...

The advertorial with weird misspellings in the graphics...

Image generation with text instructions are notoriously bad. It's frustrating lol

Just my two cents on this topic.

What we are looking for, are easier migrations in cloud I suppose and multi cloud strategy.

I am pretty sure that 37Signals had gotten their servers set up in a lot of countries for easier latency but a lot of companies can't.

We are then forced to use S3 or the likes and honestly I am starting to wonder about S3....

Like I was watching theo's video on everything being a wrapper and he mentioned that in some sense he basically created uploadthing because s3 was cheaper but had higher egress and cloudflare r2 was more expensive but had no egress and so he wanted a way to optimize it... thus uploadthing.

But this whole idea seems so bizarringly stupid to me, I had seen a website which compared such providers pricing and to be quite frank, cloudflare was in the lowest maybe only more expensive than backblaze or wasabi but both of these had a sort of fair use limit which might be vague...

In the meanwhile, I have found another website giving some decent comparisons and though I don't like its UI as much as the other website which had some really cool graphs.., its also well built and professional https://www.s3compare.io/

and I have to say, somebody might see the amazon 4$ per Tb compared to cloudflare 15$ per tb and could've said wow maybe theo was right... untill they saw the 90.00 USD/tb...

I mean, I get that but if that has to be an archive / very rarely accessed, then why not just use wasabi or backblaze (I was going to prefer backblaze untill I saw the 10$ per tb egress for backblaze and 0 $ for wasabi.... yeah)

wasabi/backblaze both seems to be really great options, they are just fractionally more expensive than Aws s3 glacier (4.99$ per Tb) and they don't have egregious egress fees....

For something more frequent, use cloudflare r2 and for archiving/backup, use wasabi/backblaze , maybe even the 3-2-1 strategy... I am not sure if wasabi/backblaze already follow that

> Then when the people who put this together leave

Why should they leave without creating institutional knowledge and educating those who come after them... That is how it was before the 'mainstream tech' and its 'turnover/after me, deluge' mentality took over and made shareholder capitalism made every employee a disposable cog. That is what all orgs. should return to now that the zirp ended and shareholder capitalism is kaput.

The twitter criticism is getting funnier.

https://x.com/zackkanter/status/1920284851506225498 criticizes 37signal of missing an opportunity because they focused too much on reducing COGS. An example opportunity that this guy gave is really really random. As you guess it, it's the AI opportunity cost. Because 37Signal hasn't integrated any AI. That must be a loss! Of course, it's the AI again!

https://x.com/grinich/status/1920636351139230043 (from WorkOS' CEO) criticizes 37Signal of not growing and losing to Jira and Linear because 37Signal focuses on saving a couple millions of dollars a year.

Jira isn't even profitable after all these decades of "winning". It's a project management tool. What do they even spend money on?

Linear is one of its kind startup. Is losing to Linear that bad? I too want to lose to Linear and make >$10,000,000 a year with my product.

The response from DHH is perfect tho: https://x.com/dhh/status/1921075541672612294

"Problem is that nobody seems content to merely put their dent in the universe. No, they have to fucking own the universe. It’s not enough to be in the market, they have to dominate it. It’s not enough to serve customers, they have to capture them."

Not only that, project management tool's market is fragmented. It'll never be a winner-take-all given there are hundreds of use cases / settings / industries. It's going to be extremely difficult to be Number 1. Investing in growth would kill you. Reducing COGS and staying profitable is the best way IMO.

> We migrated to cloud and saved $X per year

> 3 years later

> We migrated off the cloud and saved $Y per year

A tale as old as time.

What are good US-based dedicated server hosts? Like Hetzner but in the US?

I think the only interesting & substantive piece of info here is that AWS will sometimes waive DTO fees. (For once I'm actually thankful for GDPR!)

Their claimed savings are suspect, borderline clickbait, and even if they are correct the preconditions to realize those savings don't apply to the vast majority of other companies.

They already had datacenters and claim that there will be no increased opex for things like power, network (which must be substantial if they pay 1.5MM for 18PB) and employees, only needed a trivial amount of space they already had, and sounds like they're sacrificing redundancy too.

Tl;DR: don't use RoR in AWS because it will cost you.

Reply on news.ycombinator.com