Hacker News Reader

Comment by swatcoder | Hacker News Reader

> I'm an avid supporter of Kubernetes (see About me), but I would never suggest that a small team adopt it from the start. Even with expert knowledge of Kubernetes, taking it on as a dependency still creates loads of maintenance debt.

Somewhere along the way, it became fashionable to build out your devops on day one as though you’ll be Google tomorrow, but to write your code without any architectural foresight as though the only thing that matters is tomorrow’s deployment.

It’s always nice to see people advocating towards a thoughtful balance.

View on HN

This is surely resume (or consultant) driven development in action. High Scalability™ is like an epic side quest in the game of making software. The final boss at the end of the quest, Yagni, can only be defeated by those who have reached a large enough scale already. Many attempt the side quest too early and as a result get so distracted that they suffer delays or even lose the whole game.

My boss was trying to push me to use some new CICD pipeline the company built, just so he could understand what CICD meant. I kept ignoring him, because it just didn't seem needed and saw no need to complicate things.

To build the servers I was running on I wrote an Ansible job to install everything and copy the code down. All I had to do was add a couple flags to that to allow it to just deploy new changes. Instead of getting into the CICD pipeline I just took a few minute to make a little web front end for myself to kick off the job and watch the job status (we have Tower, but it's painfully slow). I can now deploy in 18 seconds with 1 click. Seems good enough. I'm sure I could set up a web hook to eliminate the click, but it's not a big deal.

Ive been banging this drum on here for a while, but we use AWS with ECS. I can spin up an ecs cluster with one runner in less than 10 minutes, and setting up a CICD pipeline is a handful of clicks on GitHub actions, or about 10 lines of pipeline .yml

I went through a React course that used Github and some online service to setup CICD in a few clicks with some service. It was painless.

At my company, it would involve going to weekly meetings to get all the updates and changes happening as they keep developing it, one-off meetings to get onboarded and setup... I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years. Those weekly meetings also mean constant changes so things keep working (which I'm not dealing with on a different site that is going though CICD). All of this in the midst of a lot of bureaucracy. It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.

It likely would have taken several months for little to no tangible benefit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited. It took 8 months for someone to bother to tell me where I could see build status/errors, at first they just told me to "wait about 10 minutes". There is no documentation anywhere. It's all tribal knowledge and no one seems to have the time to share it.

The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.

> I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years.

Yeah - this is key. If you're already set up to _not_ be containerised, then it's likely not going to be worthwhile unless you are actively feeling some of the pain points around running on VMs.

> It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.

One saving grace is that a huge amount of what k8s does is abstracts the aws-ness from underneath. Running a live application on k8s on aws is three separate skills - running a live app, running apps on k8s and running k8s on aws. Two of three of those still transfer!

> fit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited.

That's fair, and it's up to you to make the call as to whether it's worth it or not. If your team aren't willing then the use cases are very limited, and that's a political fight!

> The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.

I've been a driver for improving developer best practices on every team I've worked on. Not everything has always stuck, but I've found that processes that can be automated tend to have high rates of success, and processes that rely on humans tend to be sidestepped. I spoke to a coworker from a previous job last night who told me that some of the process fixes I made as a cog in the wheel have survived 2 years without me there. All is not lost!

>I've been a driver for improving developer best practices on every team I've worked on. Not everything has always stuck, but I've found that processes that can be automated tend to have high rates of success, and processes that rely on humans tend to be sidestepped. I spoke to a coworker from a previous job last night who told me that some of the process fixes I made as a cog in the wheel have survived 2 years without me there. All is not lost!

Yeah, it's just a matter of trying to pick those horses that seem likely to stick around, or that actually make like easier. Ansible is one of those; the CICD process and everything around it is not. We've changed out container platform 3 times in 3 years, and just kicked off a project to change it a 4th time, while also opening up some public cloud access if it can be justified. It's chaos.

There was also a push to build a lot of home-grown tools over the last 3-5 years. All of that is being ripped out in favor of vendor provided solutions.

The primary things that seem to stick around are the snowflakes, as sad as that is. The only teams that don't seem stressed by all that change are the ones who just do their own thing. I fought the good fight, jumping the latest and greatest stuff for over a decade and trying to put systems in place to get others to buy in as well, but I'm tired and this latest crop of tools seemed like they were destine for failure from day one.

So instead of testing your companies CICD pipeline, contributing to it and making it overall better for everyone you now have a snow flake system only you use and know how to maintain?

I try to align when possible. I think your solution is hacky and costs the company much more in the long run.

While my system is admittedly hacky, anyone I would hand development off to would be able to pick it up much easier than the company's pipeline, as it uses the tools they know and is pretty basic... none of them know the CICD pipeline.

I used to try to align when possible, but lately it hasn't made a lot of sense. Keeping up with the tooling ends up being the whole job, as they change it every 6 months. This site is a side project to help out the team, not my primary job. I can't be dropping everything each time they decide to switch vendors, change up their variables names, or whatever else they decide to do.

> for myself

Sounds like your system is way more efficient than the CICD... in the short term, provided you're the only person that ever needs to use it (and/or nothing ever changes)

I'm the only person who ever uses it. If for some reason I left, what I have would be easier for the people who it would be handed off to, as they know Ansible, but have no idea how the CICD pipelines work. It's primarily an automation team, not an app development team.

> what I have would be easier for the people who it would be handed off to

This seems like a very large assumption.

> as they know Ansible, but have no idea how the CICD pipelines work

I know Ansible very well. I have never set up a CICD system and would much prefer to write a bunch of Ansible playbooks for myself than to do so. My company has a CICD system already set up (I have no idea how it works internally). Setting up a pipeline and understanding how that pipeline works as a user of the CICD system is still a lot easier for me than figuring out someone else's Ansible playbook repo. Adding jobs to that pipeline is still easier than writing new Ansible tasks. CICD is almost always easier for dev teams regardless of their experience once the initial complexity of CICD setup/hosting/maintenance is already handled.

If you don't already have a CICD set up, then setting up one is likely overkill for small teams. If it's already there (and maintained by a well-resourced team), use it, unless the CICD platform being used is absurdly bad for some reason (I've used a few and they have been largely fine / interchangeable).

It's a really basic playbook. There is a server build block that basically says install Apache, PHP, etc. and edit a few config files. And another block that just does a pull from GIT. As far as playbooks go, it's pretty basic and easy to understand. I'm not looking at it right now, but it's probably fewer than 10 tasks. It's not not one of those with thousand of lines referencing dozens of roles with complex when conditions and jinja2 on everything, like I often see.

As mentioned in another comment, the site has been running on VMs for almost 10 years and not containerized. The CICD pipeline is built around Kubernetes and I have no real reason to move to Kubernetes other than to say it's running in containers. The VMs have never had issues, it's a relatively small internal site, so I don't need to worry about scaling issues. K8 is also in the process of being replaced, so moving to it now seems silly. I use the pipeline for another project I'm working on with another team and while it's needed for the stack they're using, it's bad. No one seems to know how it works, I haven't spoke to a single developer who has a positive thing to say about it, it's very poorly documented, and any time there is an issue it requires messaging a bunch of random people until you find someone with the right blend of tribal knowledge and time to help. I've had a half dozen people come to me just to find out where to see the deployment status or to know how to promote something to upper environments... and I wasn't given that information until 8 months in. Seems like it would be day 1 stuff that would be easily documented.

For a hackathon they made something to do a one-click setup of a project that I would likely use, but outside of the hackathon, that feature isn't available and it would require months of work when factoring in meetings, waiting on other teams, etc to get setup without that. It's just not worth it for my use case.

I'm only a light k8s user, but isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure? Like, there's no point spinning up EC2 nodes just to run microk8s or k0s on them— have ansible scripts or whatever that just run your application directly.

But the calculus changes if instead of EC2 nodes you can have a helm chart that deploys directly to a managed kubernetes services like EKS or GKE?

That's still a bunch more complicated than other hosted solutions. I once worked for a 4 year old company whose entire infrastructure consisted of a managed postgres instance + a moderately sized single VM (and a duplicated setup as a staging environment).

Eh, I've found kubernetes more complicated than the interface of a cloud provider, you always have to do the cloud provider as well for some things, and the benefits of the pre-baked helm charts or whatever else just aren't there until the relatively small difference in efficiency in each way of doing things makes a real difference as a result of scale.

If you have a lot of experience with any interface, you can do it really efficiently, but if you don't, kubernetes can make your experience worse than anything you'd come up with on a cloud provider.

Can't you run K8 on Fargate so you don't have to manage EC2 nodes themselves?

Even with ECS which is way simpler than K8 for small teams using Fargate has been incredibly beneficial in terms of debt/time.

Then next step would be to move the entire application onto Lambda so we're not paying for downtime, but that's a whole headache and a half.

Fun with k8s starts when you realize it has no LTS and gets updated on a fixed schedule. Any given version is only officialy supported for about 1 year and a half.

Then you'll need non trivial amount of knowledge of how it works to avoid many pitfalls (the helm chart will probably only go so far, it will still need small tweaks for your specific use case)

So it's a combination of upfront time investment, ongoing tuning, and recurring maintenance events.

Honestly, I find Helm to be a trap that is going to reduce your capability to run service reliably - or understand it. It's probably one of the most annoying bits in kubernetes ecosystem

What are the benefits of k8s over docker-compose for small/medium business?

All of the cloud providers offer k8s as a service, for one.

I should specify I was thinking about VPS. Coming from AWS to Linode VPS with Caddy + Docker felt great, Im wondering at what point (N requests per second ?) it becomes tech dept

Yeah that's definitely something that needs to be specified.

> Coming from AWS to Linode VPS with Caddy + Docker felt great

I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow.

> Im wondering at what point (N requests per second ?) it becomes tech dept

I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose?

The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`

thanks for detailed response! how big is your org? My concern is that there is a cost to magic of adding deps like that. Complexity goes up and it is harder to understand what went wrong and where are the performance bottlenecks. I guess my question is at what org scale the tradeoffs are right

> how big is your org?

My last team was the same, there was ~100 developers on the main app, ~50 on the ancillary services, and a ballpark of ~10 on the "platform". I was just a team member on that team. Current org is 15 devs, 30 people total.

> My concern is that there is a cost to magic of adding deps like that.

Of course there is, there's such thing as a free lunch. It's up to you to decide which is right for your team/org. The cost of doing it manually is developer time, tech debt and the bus factor. The cost of using helm is introducing an abstraction. Profesionally, the bus factor and developer time of home rolling a solution make relying on existing solutions a very appealing proposition. Having to maintain a set of half baked shell scripts is exactly what tools exist to solve, and I've had more issues with deployments where someone has bunged together a bash script that makes varios assumptions about tooling, versions, installation paths, working directories, path formats, etc. than I have with problems related to our actual deployment infrastructure.

> I guess my question is at what org scale the tradeoffs are right

The cost of switching is huge, and likely not worth it unless you have genuine problems that would be solved by moving. For a greenfield project, frankly I'd think you're wrong to use docker-compose on any new project. Either you're managing the underlying infrastructure (ec2/vps style) or you're not (fargate/DO App Platform style), and in either case docker compose requires roughly the same amount of investment for a smaller return on a less actively developed product objectively solves less problems and requires more work to integrate.

[0] https://github.com/marketplace/actions/deploy-to-kubernetes-...

thanks for your insights! I had an impression that k8s is more for enterprise stuff, I will definatly give it a try!

While the specifications are more complex, I find much more sensible in management than docker-compose, and allowing easier expansion (not scaling the service, expansion of underlying hardware) which means keeping costs lower can be easier.

Could you specify what you mean by "management"

One of the systems I set up recently I decided to go the docker-compose way. Quite quickly I found out that either the tool doesn't support (or at least isn't exactly interested in supporting) multiple compose files, or I am too stupid to find out the right combo of options, which resulted in a growing yaml file as the setup of dockerized components rose on that specific server.

Whatever claims about user friendliness, I didn't figure a stable way to do deploy separate docker-compose.yml's from same user at the same time.

Not using Compose in production, but our development setup is split across multiple files - you can include them by passing -f flag to Docker Compose (I wrote a small wrapper script that does that automatically)

> isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure?

That's a big part of many k8s pitches I've seen but in my experience it hasn't been representative of reality (or at least for every part of infrastructure you no longer need to run there's a new part of k8s architecture/config to learn & manage).

For me the selling points are:

1. For the time-cost of upfront complexity and setup, you get extremely easy and quick subsequent addition of infra/tooling & scaling thereof. 2. There are certain patterns of tooling that k8s enables (relatively easily), that would have been difficult/impossible with more traditional setups.

I don't think it makes infra management overhead less, it just reorganises it so it's more upfront and then less later while scaling.

It really depends on what you're going to run and what experience you have.

Personally, I would run k8s from the day I expect to run the code as separate deployment to running it from my IDE. I have the skills to do so, and I found I lose way less time doing so than trying to do "classic" server setup, even if it's just one server.

A small team should use the central k8s platform if it exists and it's the company base infrastructure layer.

Everything else would only mean less golden path support, less synergies etc.

By "small team" I assume they meant a start-up - i.e the team is the company's entire dev.

As someone who works in a small team within a large corp where we use k8s for everything large and small, I did not read that statement as applying to me.

Reply on news.ycombinator.com