Comment by movedx | Hacker News Reader

Comment by movedx | Hacker News Reader

Parent Full thread

> Build infrastructure without understanding the use case? Premature optimization.

Hence my issues with micro services and Kubernetes/containerisation (by default.)

I've always hated the fact people simply jump onto these technologies and methodologies as if they're automatically the right solution because everyone is talking about them. What they don't understand is that they're optimisations.

You build a monolith and put it on one machine to begin with. No load balancer. Just a single EC2 instance with snapshots. As the customer count grows and demand increasing you scale it out...

Now you're on two EC2 instances and might want to consider using RDS. You have an ALB and you're using ACM to offload TLS certificate management. More customers come along and the monolith begins to slow down, so you optimise the application this time...

Now you have the most successful/popular parts of your application split out into separate components but still using the same database. You're still just running Docker on a few EC2 instances though. You don't need orchestration yet. But now your customers start demanding more features and changes on a more frequent basis. Also your customer count is rising more and more. You're now ready to scale out and re-architect things again...

Now you've got 80-85% of your monolith split out into separate components, in Docker images, and you're using Kubernetes to orchestrate the whole thing because you need to iterate and deploy parts of the software on a near daily basis.

Taking it slow and keeping things simple in the beginning allows you to focus (from a systems perspective) on stability and security, which are much easier when you have a monolith and two EC2 instances. As you need to iterate faster and more often you increase the complexity of the network to meet the needs of the development team. It becomes much harder to secure and manage, but the trade off is worth it.

That's how you optimise your infrastructure over time.

The only situation in which I would contradict my self on this point is if you're developing a product that you know will need micro services and K8s to begin with AND everyone on the team has extensive experience implementing an application in that manner.

Why people start at the K8 end state is twofold, fashion and a lot of younger people don't look at highways and see they started out as either single lane tarmac or dirt roads that were remade in place. There's a lot more talk these days about refactoring complex systems. But its really a job for senior engineers or young folk who have time for it (Like the new grads who complain about being dumped on legacy systems :) ). When you're jumping from framework to framework and shipping features, sitting back and considering things is really tricky, and the inertia to build from scratch but also have 20 years of refactored improvements delivered makes jumping on K8's and the magic that brings seem like a no brainer.

I really, really like your metaphor of the dirt roads remade in place. It’s fascinating to look down at the sidewalk in NYC and know that it was farmland only a few centuries ago. We know the aphorisms: Rome wasn’t built in a day, but we don’t always think them through.

At work, when a new person joins our team, it’s fascinating (in a non-cynical way) to hear “why are we using a dirt road here instead of a highway? Wouldn’t that be more efficient?” It’s fun to imagine the implications of such a project, and it’s a great opportunity to give perspective on the traffic flow of customers, developers, operations.

Could we build a highway there? Yes. But usually, it’s a better allocation of our finite resources to do some maintenance on the existing highway. Sometimes, we actually need to take highways apart and replace them with something more informal, since their maintenance costs more than they’re worth.

Something tangential this reminds me of is the concept of “Desire Lines”: https://en.m.wikipedia.org/wiki/Desire_path. The work of an SWE can be summed up as the identification and “improvement” of desire lines. If you build a highway, but it’s not on a desire line, you’ve just wasted a lot of energy!

With many apps, you can completely bypass kubernetes entirely and just use something like elastic beanstalk or heroku for scaling (just by adding more nodes behind a load balancer).

I invested several days setting up my product's backend on Elastic Beanstalk. After several weeks, I learned that Elastic Beanstalk is not reliable.

Then I spent several days learning Kubernetes and gave up when I learned that it is overly complex and mostly undocumented.

Then I wasted a week using Packer to build EC2 AMIs and deploy with terraform and run with supervisord. I learned that Packer is slow and Linux package repositories are unreliable. Both of these things could prevent me from deploying an emergency patch. Also, there are no reliable tools to clean up old AMIs generated by Packer.

Then I spent several weeks learning Docker and migrating my product's backend to that. Docker's APIs and tooling are hard to use but at least they're reliable. And I can run the same binaries in production and on my dev machine.

Throughout this process, I frequently looked at Heroku and then remembered how much they charge. I just can't bring myself to spend $500/mo for their ops automation. Perhaps I'm being penny-wise and pound-foolish.

I find it strange that you believe that Elastic Beanstalk is not reliable... That hasn't been my experience at all.

My company has been using it to host our production infrastructure for over a year and a half now, and it has been performing excellently. Our site reliability over the past year has been somewhere around the 99.99% range (and few outages we did have were not related to Elastic Beanstalk in any way).

We have our site hosted in multiple redundant physical locations (AWS regions), with a load balancer routing requests between multiple nodes.

We do about 10-20 deploys to production on a daily basis, with zero downtime. Any time there is a deployment, the code is automatically deployed in rolling batches, and the nodes aren't added back to the load balancer until they pass their health checks.

See this comment: https://news.ycombinator.com/item?id=23136482

With many apps, you can completely bypass kubernetes entirely and just use something like elastic beanstalk or heroku for scaling (just by adding nodes).

Reply on news.ycombinator.com