Hacker News Reader

Comment by bastawhiz | Hacker News Reader

Knowing the prices and performance of Heroku (as a former customer) the effort probably paid for itself. Heroku is great for getting started but becomes untenably expensive very fast, and it's neither easy nor straightforward to break the vendor lock in when you decide to leave.

View on HN

I find AWS ECS with fargate to be a nice middle ground. You still have to deal with IAM, networking, etc. but once you get that sorted it’s quite easy to auto-scale a container and make it highly available.

I’ve used kubernetes as well in the past and it certainly can do the job, but ECS is my go-to currently for a new project. Kubernetes may be better for more complex scenarios, but for a new project or startup I think having a need for kubernetes vs. something simpler like ECS would tend to indicate questionable architecture choices.

I pretty strongly agree. Fargate is a great product, though it isn't quite perfect.

ECS is far, far far smoother, simpler and stable than anything else out there in cluster orchestration. It just works. Even with EC2 instances it just works. And if you opt for Fargate, then that's far more stable option.

I am saying this after bootstrapping k8s and ECS both.

The only pain point there I think is auto scaling logic. But otherwise it’s painless.

I find auto-scaling with fargate to be pretty straightforward. What's the pain point there for you?

It works, but the way it’s not part of fargate, and instead some combination of cloudwatch events and rules modifying the ‘desiredCount’ property on the service.

Just feel like it could all be done in a slightly more integrated way.

GCP Cloud Run is even better, which you don't have to configure those networking stuff, just ship and run in production

Does Cloud Run give you a private network? While configuring it is annoying, I do want one for anything serious.

How does it compare with fly.io? Last I checked, startup time is still in minutes instead of less than a second on fly, but I presume it's more reliable and you get that "nobody ever got fired for using AWS" effect

Fly.io is an unreliable piece of shit.

Fly is really cool and it's definitely an extremely quick way to get a container running in the cloud. I haven't used it in production so I can't speak to reliability, but for me the main thing that stops me from seriously considering it is the lack of an RDS equivalent that runs behind the firewall.

+1 on this. ECS Fargate is great.

From their presentation, they went from $7500/m to $500/m

I mean, $7,000 a month isn’t nothing. But it’s not a lot. Certainly not enough to justify a seven month engineering effort plus infinite ongoing maintenance.

The main engineering effort to reduce by that much was completed in 6 weeks according to their YouTube video.

7 months is presumably more like “the time it has been stable for” or so, although I am not sure the dates line up for that 100%.

Also cost reduction was apparently not the main impetus, GDPR compliance was.

That “main engineering effort” will go on forever. People neglect to note that everything is constantly changing. Just like the roof on your house, if you don’t upgrade your components regularly, eventually you will face a huge rewrite when that thing your ancient home-made infrastructure relies on is no longer supported or is no longer updated to support the latest thing you need for your SaaS.

You can’t avoid this cost. Some people refer to it as technical debt, but I think more accurately it could be called “infrastructure debt”. Platform providers maintain the infrastructure debt for you - this is what you pay them for. And they do it with tremendous economies of scale. Unless your scale is truly enormous - like Meta, for instance - it isn’t worth build your own infrastructure.

This is $7k/mo today. If they are actively growing, and their demand for compute is slated to grow 5x or even 10x in a year, they wanted to get off Heroku fast.

Would you say one person not working 100% of the time is also quite minor? ;)

Sure. We have around 10 of those. It’s a significant boon to the project for them to do nothing.

assume a dev is $100k/year... so $200k with taxes, benes, etc. That's 16,666/month, at 1.5 months is 25k. So it'll take 3.5 months to break even. And they'd save around .8 of their pay, or .4 of their total cost a year...

Generally I am hoping my devs are working a good multiplier to their pay for revenue they generate. Not sure I'd use them this way if there was other things to do.

That said sounds like it was mainly for GDPR so.

Now consider that some places are not in Silly Valley, or not even in USA, and the fully loaded cost of engineer (who, once done with on-prem or at least more "owned" stack, can take on other problems) can be way, way lower

These numbers are actually WAY LOW for silicon valley. If I was doing the ROI for an Amazon employee I'd start with around 350 for an SDE all told for entry level and half a mill for one with a few years experience.

But they're also way high for other places. And just right.

The point is how to do the math not the ballpark. Also that even at 100k for a dev it's maybe a wash depending on your time horizon.

My experience is that a lot of "simpler alternatives" ballooned costs beyond cost of someone to wrangle the more complex solution - and well, after initial pains, the workload drops so you can have them tackle other problems if not at full time.

Or as I said it few times at meetups, Heroku is what I use when I want to go bankrupt (that was before Heroku got sold)

Where are you finding capable DevOps engineers for 100k total comp? It’s hard to find someone with the skills to rebuild a SaaS production stack who’s willing to work for that little around here!

Europe, probably

Correct, mynewsdesk (that created this reclaim the stack thing) is a Swedish company.

100k Eur is high salary for a dev. A unicorn who knows they are one, won't agree to that salary, but would for 150k or 200k.

I'm picking a random salary that's not too high that lower comp countries/industries will reject, and not too low that higher comp countries / industries will reject, and then doing math on those.

You can then take my numbers or math and plug in YOUR comp rates. But the TL;DR I've seen is many people never even do napkin math like this on ROI.

We've since migrated more stuff. We're currently saving more than $400k/year.

Moving from Heroku to Render or Fly.io is very straight forward; it’s just containers.

Unless you relied on heroku build packs.

Buildpacks is opensource too [1]

[1] https://buildpacks.io/

If you use containers. If you're big enough for the cost savings to matter, you're probably also not looking for a service like Render or Fly. If your workload is really "just containers" you can save more with even managed container services from AWS or GCP.

We are talking about moving from Heroku, I don't think being too needy for the likes of Fly is at all a given. (And people will way prematurely think they're too big or needy for x.)

Technically, you don’t even need to set up containers for Render.

So is kubernetes. GKE isn't that bad.

(Except for Postgres, since Fly's solution isn't managed)

Heroku's price is a persistent annoyance for every startup that uses it.

Rebuilding Heroku's stack is an attractive problem (evidenced by the graveyard of Heroku clones on Github). There's a clear KPI ($), Salesforce's pricing feels wrong out of principle, and engineering is all about efficiency!

Unfortunately, it's also an iceberg problem. And while infrastructure is not "hard" in the comp-sci sense, custom infra always creates work when your time would be better spent elsewhere.

> Salesforce's pricing feels wrong out of principle

What do you mean exactly? If it takes multiple engineers multiple months to build an alternative on kubernetes, then it sounds like Heroku is worth it to a lot of companies. These costs are very "known" when you start using Heroku too, it's not like Salesforce hides everything from you then jump scares you 18 months down the line.

SF's CRM is also known to be expensive, and yet it's extremely widely used. Something being expensive definitely doesn't always mean it's bad and you should cheap out to avoid it.

Couldn't you move to AWS? They offer managed Postgresql. Heroku already runs on AWS, so there could be a potential saving in running AWS managed service.

It's still a lot of work obviously.

So does GCP and Azure. At least in GCP land the stuff is really quite reasonably priced, too.

I moved our entire stack from Heroku to Render in a day and pay 1/3 less. Render is what Heroku would be if they never stopped innovating. Now I’m thinking of moving to fly as they are even cheaper.

I mean this is what they recommend:

- Your current cloud / PaaS costs are north of $5,000/month - You have at least two developers who are into the idea of running Kubernetes and their own infrastructure and are willing to spend some time learning how to do so

So you will spend 150k+/year (2 senior full stake eng salaries in EU - can be much higher, esp for people up to the task) to save 60k+/y in infra costs?

Does not compute for me - is the lock-in that bad?

I understand it for very small/simple use cases - but then do you need k8s at all?

It feels like the ones who will benefit the most is orgs who spend much more on cloud costs - but they need SLAs, compliance and a dozen other enterprisy things.

So I struggle to understand who would benefit from this stack reclaim.

Creator of Reclaim the Stack here.

The idea that we're implying you need 2 full time engineers is a misunderstanding. We just mean to say that you'll want at least 2 developers to spend enough time digging in to Kubernetes etc to have a good enough idea of what you're doing. I don't think more than 2 month of messing about should be required to reach proficiency.

We currently don't spend more than ~4 days per month total working on platform related stuff (often we spend 0 days, eg. I was on parental leave during 3 months and no one touched the platform during that time).

WRT employee cost, Swedish DevOps engineers cost less than half of what you mentioned on average, but I guess YMMV depending on region.

Reply on news.ycombinator.com