Kubernetes is a red flag signalling premature optimisation
jeremybrown.tech
jeremybrown.tech
Also he talks about all of this and then gives example of WhatsApp at the end. WhatsApp chose Erlang for the backend and their front end was written in Java and Objective-C. They could have chosen Java for backend to keep frontend language same but they didn't. They used Erlang because they based their architecture on Ejabberd which was open source and was built with Erlang. Also WhatsApp managed all their servers by themselves and didn't even move to managed cloud services when they became available. They were self hosting till FB acquired them and moved them to FB data centres later on (Source: http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...).
Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
I know of such a case, where a single engineer could leverage the helm chart open source community, and set up a scalable infrastructure, with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated through nginx and cert-manager, do full integration tests, etc.
I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've seen places hire a dev that write all the OPS stuff and they scaled awesomely.. I mean if they had purchased 100servers full time on amazon, they would have spent a fraction of the cost to scale, but they could scale. In 5 years I think they've never once had to reach even near the 100servers.
At the same time. I can scale heroku to 500 servers, and still be under the cost of one ops person. I can make that change and leave it there. I can do that all in under 30 seconds. Oh. And CICD is built in as a github hook. Even with blue-green deploys.
I think his point was most start-ups don't need to scale more than a site like heroku can offer. If you need more than 500 servers running full time then it's time to start looking to "scale"
And then Heroku shuts down.
If you're building something that needs to scale up rapidly if it succeeds, k8s is worth thinking about. Either you don't succeed, in which case it doesn't matter what your stack was, or you do, in which case you'll be glad that you can scale up easily, you'll be glad you are using a common platform which is easy to hire competent people in, and, if you were smart about how you used k8s, you'll be glad that you can relatively easily move between clouds or move to bare metal.
I agree, but so what? K8s isn't magic, it won't make all your problems go away, but if you have people who are genuinely skilled with it, it solves a lot of problems and generally makes scaling (especially if you need to move between clouds or move onto bare metal) much smoother. Of course you'll still have other problems to solve.
Given that most startups never need to scale up much, it's not surprising that k8s is mostly used where it's not needed. But people usually prefer not to plan for failure, so it's also not surprising that people keep using it.
"Why no k8s?", you ask, only to be met with incredulity: "We don't have those skills", says the profiteering web agency. Sure, k8s is hard… Not. Nevermind that it's pretty much the only important part of your job as of 2022.
That's because 'the norm' for web agencies is a site that does basically zero traffic. If a company hires a 'web agency' that's by definition because the company's business model does not revolve around either a web property or app.
Whether that's a gas station company or a charity or whatever, the website is not key to their business success and won't be used by most customers apart from incidentally.
With that in mind most agencies know only how to implement a CMS and simple deployment perhaps using Cloudflare or a similar automated traffic handling system. They don't know anything about actual infrastructure that's capable of handling traffic, and why would they?
A lot of agencies are 100% nontechnical (i.e. purely designers) and use a MSP to configure their desktop environment and backups and a hosting agency to manage their deployed sites.
In asking if "Kubernetes is a red flag signalling premature optimisation", you correctly explain why we're yet on the "yes" side for the typical web agency category.
[Although FWIW I was hinting at a non-trivial category who should know better than not to setup a scale-ready infra for some potentially explosive clients; which is what we do in the entertainment industry for instance, by pooling resources (strong hint that k8s fits): we may not know which site will eventually be a massive hit, but we know x% of them will be, because we assess from the global demand side which is very predictable YoY. It's pretty much the same thing for all few-hits-but-big-hits industries (adjust for ad hoc cycles), and yes gov websites are typically part of those (you never know when a big head shares some domain that's going to get 1000x more hits over the next few days/weeks), it's unthinkable they're not designed to scale properly. Anyway, I'm ranting now ^^; ]
My unspoken contention was that eventually, we move to a world where k8s-like infra is the de facto norm for 99% of infrastructure out there, and on that road we move to the "no" side of the initial question for e.g. web agencies (meaning, we've moved one notch comparable to the move from old-school SysAdmin to DevOps maybe, you know those 10 years circa 2007-2018 or so).
[Sorry for a too-terse initial comment, I try not to be needlessly verbose on HN.]
This is very very hard to parse BTW. I don't want to reply to what you've written because I can't determine for sure what it is that you're saying.
Essentially I mean: scalable infra may be premature optimization today in a lot of cases, but eventually it becomes the norm for pretty much all systems.
You could similarly parse the early signs of a "devops" paradigm in the mid-2000's. I sure did see the inception of the paradigm we eventually reached by 2018 or so. Most of it would have been premature optimization back then, but ten-ish years later the landscape has changed such that a devops culture fits in many (most?) organizations. Devops being just one example of such historical shifts.
I anticipate the general k8s-like paradigm (generic abstractions on the dev side, a full 'DSL' so to speak, scalable on the ops side) will be a fit for many (most?) organizations by 2030 or so.
I hope that makes sense.
It's also worth considering that appropriate use of k8s can quite likely save you time and money early on as well. It standardises things, making it very easy for new ops people to onboard, and you might otherwise end up spending time reinventing half-baked solutions to orchestration problems anyway.
Well, precisely what I said is that 99.9% of startups won't find themselves in a situation where they need to scale quickly and the only scale problems they find can be solved with Kubernetes.
> It's also worth considering that appropriate use of k8s can quite likely save you time and money early on as well. It standardises things, making it very easy for new ops people to onboard, and you might otherwise end up spending time reinventing half-baked solutions to orchestration problems anyway.
The point is that you might not even need orchestration from the start. Instead of thinking how to solve an imagined scenario where you don't even know the constraints, go simple and iterate from that when you need it with the actual requirements in hand. And also, "make it easier for new ops people to onboard" doesn't matter if you don't have a viable product to support new hires.
There are a lot of companies in the middle, and following dogma like "you don't need k8s" leads them to reinvent the wheel, usually badly, and consequently waste enormous amounts of time and money as they grow.
Knowing when is the right time to think about architecture is a skill; dogmatic "never do it" or "always do it" helps nobody.
I'd argue you should definitely consider multi-cloud strategy from the get-go indeed in 2022. Something like Terraform helps statically setting k8s clusters on most clouds. Especially for startups, it's better to default to vanilla stuff and only complicate on a need-to basis.
A multi-cloud paradigm is also a great way to teach you about your application and about those clouds themselves. A good reminder that "implementation is where it's at", and "the devil is in the details".
No one is advocating for a single server running in your closet, but a large and well funded PaaS can handle any realistic amount of growth at least temporarily, and something like Heroku is big enough (and more importantly, owned by a company big enough) that shutting down without notice is not a possibility worth considering.
This take brushes right past the causes of success and failure. Early stage success depends on relentless focus on the right things. There will be 1000 things you could do for every 1 that you should do. Early on this is going to tend to be product-market fit stuff. If things are going very well then scalability could become a concern, but it would be a huge red flag for me as an investor if an early stage company was focusing on multi-cloud.
Kubernetes is basically an industry standard at this point. It's easy to hire ops people competent in it, and if you do hire competent people, it will save you time and money even while you are small. As an investor "we use this product for orchestration rather than trying to roll our own solutions to the same problems, so that we can focus on $PRODUCT rather than reinventing half-baked solutions to mundane ops problems" should be music to your ears.
As many problems as containerization solves, it's not without significant overhead. Personally I'm not convinced the value is there unless you have multiple services which might not be the case for a long time. You can get huge mileage out of RDS + ELB/EC2 using a thinner tooling stack like Terraform + Ansible.
If you have such a team, it can of course make sense to delay or avoid containerisation if you don't see obvious major technical benefits.
But those teams will get rarer as time goes on, and since we're talking about startups, honestly it would be questionable to build a new ops team from people with no containers knowledge in 2022.
Yes, if your application looks like a web application that is cache friendly, k8s can really take you a long way.
The "web application" / cache-friendly part of your comment doesn't make much sense to me; k8s is pretty well agnostic to those kinds of details. You can orchestrate a database-bound system just as well as you can anything else, of course.
I do consider this impressive. And, to be clear, I wouldn't say this is because of a "super-developer". In fact, he had no prior k8s experience. But rather that there are thousand upon thousand of infrastructure hours devoted to the helm charts, often maintained by the people who develop the services themselves. It is almost mind boggling how much you get for almost free. Usually with very good and sensible default configurations.
In my precious work place, we had a team of 5 good engineers purely devoted to infrastructure, and I honestly believe that all five would be able to spend their time doing much more valuable things, if k8s had existed.
As for whether or not such devops solutions could be done in a day. Hm. I don't know. These things should be tailored to the problem. If you've done all of this a few times, then maybe you can adjust a bunch of charts that you are already familiar with and do what took a couple of months and impressed me, in a couple of weeks. It's a lot more than just "helm install. Done", that goes into architecting a scalable solution. Implementing monitoring, alerting and logging. Load testing stuff. Etc.
That does remind me that when that employee started, the existing "simple operational tech" was in fact to SSH into a VM and kill the process, git pull the latest changes, and start the service.
The only way you can solve the actual problem (not a simplified one) would in my opinion either be k8s or terraform of some kind. The latter would mostly define the resources in the cloud provider system, most of which would map to k8s resources anyways. So, I honestly just consider k8s to better solve what terraform was made for.
I'm sure the "simpler operational tech" meets few requirements for short disaster recovery. Unless you have infrastructure as code, I don't think that is possible.
Only if "scaling" is the problem that your startup is solving.
and ... you might be justified in that concern. However... after having been in the web space for 25+ years, it's surprising to me how many people have this as a primary concern ("we gotta scale!") while simultaneously never coming close to having this concern be justified.
I'm not saying it should be an either/or situation, but... I've lost count of how many "can it scale?" discussions I've had where "is it tested?" and "does it work?" almost never cross anyone's lips. One might say "it's assumed it's tested" or "that's a baseline requirement" but there's rarely verification of the tests, nor any effort put in to maintaining the tests as the system evolves.
EDIT: so... when I hear/read "scaling is a main concern" my spidey-sense tingles a bit. It may not be wrong, but it's often not the right questions to be focused on during many of the conversations I have.
Also, discussions about rewrites to scale up service capacity, but nobody has actually load tested the current solution to know what it can do.
It's a trap many of us fall into - I've done it myself. But next time I'll chuck money at the problem, using whatever services I can buy to get to market as fast as possible to test the idea. Only when it's proven will I go back and rebuild a better product. I'll either run a monolith or 1-2 services on VPSs, or something like Google cloud run or the AWS equivalent.
Scaling something no one wants is pointless.
yes by simple I mean covering high availability requirements, continuous deployment, good DORA measures - not simple as in half-baked non-functional operations (such as manually sshing to a server to deploy)
In our case, the workers were both quite heavy in size (around 1 GB), and heavy in number crunching. For this reason alone (and there are plenty more), lambdas would be a poor fit. If you start hacking them to keep them alive because of long cold starts, you would lose me at the simple part.
this is a few years out of date of platform capability, just fyi
And scaling is the same that VPS I could just scale the same way. Run a resize in my hosting company panel. (I dont use autorescal atm)
Only if I would hit about 100x times the nrs I would get the advantage of k8s, but even then I could just split up customers into different VPS.
CI / CD can be done good and bad with both.
And in practice K8S's a lot less stable. Maybe because I'm less experienced with K8S. But also because I think its more complex.
To be honest k8s is one of those dev tools that has to reinvent every concept again, so it has it's own jargon. And then there are these ever changing tools on top of it. It reminds me of JS a few years ago.
Premature optimization is a top problem in startup engineering. You have no idea what your startup will scale to.
If you have 1,000 users today and 5 year goal of 2,000,000 users, then spending a year building infrastructure that can scale to 100,000,000 is an atrociously terrible idea. A good principal can setup a working git hook, circleci integration, etc capable of automated integration testing and rather close to ci/cd in about a weekend. Like you can go from an empty repo to serving a web app as a startup in a matter of days. A whole year is just wasteful insanity for a startup.
The reality for start-ups running on investor money with very specific plans full of OKRs and sales targets is very different: you need to be building product as fast as possible and not giving any fuck about scale. Your business may pivot 5 times before you get to a million users. Your product may be completely different and green-fielded two times before you hit a million users.
I can't imagine any investor being ok with wasting a quarter of a million+ and a year+ on a principal engineer derping around with k8s while the product stagnated and sales had nothing to drive business -- about as useful as burning money in a pit.
You hire that person in the scale-up phase during like the third greenfield to take you from the poorly-performing 2,000,000 user 'grew-out-of-it' stack to that 100,000,000+ stack, and at that point, you are probably hiring a talented devops team and they do it MUCH faster than a year
Why are so many micro-services needed? Could the software be deployed in a more concise manner?
Not getting into the whole monolith-vs-services arguments. In both cases, complexity of deployment is part of the cost of each option.
I am not privy to the details of the case, but a rule-of-thumb I heard once is that if it's far enough from your core, a SaaS can be used (obviating the whole question), and if it's part of the core, start by developing it as a separate functionality before moving it to another service.
So far, we're up at 6 services, yet still at almost zero developer overhead cost. Then add the SaaS stack for each environment (api, worker, redis) and you're up at 15.
Not because I'm amazing, but there's a frankly ridiculous amount of information out there, and good chunks of it are high quality too. I think I started the job early January, and by April I had CI/CD, K8s for backend/frontend/DBs, Nginx (server and k8s cluster), auto-renewing certs, Sentry monitoring, Slack alerts for ops issues, K8s node rollback on failures, etc.
The best way to learn, is to do. Cliche, but that's what it really comes down to. There's a fair few new concepts to grasp, and you probably have picked some of these up almost by osmosis. It sounds more overwhelming than it is, truly.
- initial nodes setup: networking configuration (private and public network), sshd setup (disallow password login), setting up docker, prepping an NFS share accessible on every nodes via private network
- install RKE and deploy the cluster, deploy nginx ingress controller
- (optional) install rancher to get the rest of the goodies (graphana, istio, etc). These ate a lot of resources though, so I usually don't do this for small clusters
Done in a single afternoon.
My otherhot take is to not use helm but rather something like jsonnet or even cue to generate your yaml. My preference is jsonnet because you can very easily make a nice OO interface for the yaml schemas with it. Helm's approach to templating makes for a bit of a mess to try and read, and the values.yml files _really_ leak the details.
These often get developed side by side. GitLab, unit tests, api-server, nginx, cert-manager, deployments, integration tests, prometheus, metrics in services, grafana, alert-manager, log consolidation, work services and scaling, etc.
Just spinning up a cluster, nodepool, nginx, cert-manager w/let's encrypt cluster issuer, prometheus, grafana, can easily be done in a day. So, time estimates kinda depend entirely on what you mean by it.
Spinning up promerheus and grafana with automatic service discovery: one day. Making good metrics, visualizations, and alerts: everything from a week to a month or two. So, take the time estimates with a grain salt.
Also we bought us immediately an easy approach to build gitlab ci/cd pipelines + different environments (dev, staging, production) on the same cluster. Took me a week to set everything up completely and saved our team developing rapidly features really a lot of time and headache since then. But the point is, I knew how to do it, focus on the essentials and deliver quick reasonable results with large leverage down the route.
Bad culture alert! No one needs Elastic "as quickly as possible" unless their business, or the business they work for, is being very poorly run.
I would also argue that you might have got it running quickly, but how are you patching it? Maintaining it? Securing it? Backing it up? Have you got a full D/R plan in place? Can you bring it back to life if I delete everything within 3-6 hours? Doubt it.
> maintaining nodes with some clunky shell scripts.
Puppet Bolt, Ansible, Chef, ...
There are so many tools that are easy to understand that solve this issue.
Setting it up is easy, but not operations.
Ive been managing multiple clusters on AWS and on Azure.
I would take a managed EKS/OS in a heartbeat if I was a startup.
Elastic usecase is for crawling crazy amount of data and make it searchable and aggregatable and historically available, don’t know any other solution than elastic who has reasonable response times and easy-to-use access (plus we can add some application logging and APM).
> Puppet Bolt, Ansible, Chef, ...
Helm chart values.yaml and you’re all set, security + easy version bump included.
But that's the thing though: they didn't do it alone. You literally pointed out that this wasn't true almost immediately: "leverage the helm chart open source community". They used the work of others to get to the result.
Also, I highly doubt they could debug something if it went wrong.
I simply cannot believe anyone would advocate, or believe, that because a Helm chart makes it simple to create a complex piece of infrastructure it must also be true that maintaining it is also simple. It's really.
Once you understand the concepts it's not hard to debug. It's fair to acknowledge that kubernetes is complex, but also we should not ignore the real work that has been done in the past few years to make this stuff more accessible to the developers that want to learn it.
Also, saying it's not "alone" in this example I think is not fair. What would you count as "alone"? Setting up the kubernetes cluster from scratch and writing your own helm charts? Using that same logic, I can't say that because someone else designed and built the hardware it's running on. I think it's fair to say that if someone, independent of coaching, regardless of the underlying infrastructure produced some production grade infrastructure by themselves, they certainly did it alone.
As for debugging. You do have a point that it becomes more difficult. But, this also holds true for any of the alternatives discussed her (lambdas, terraform). I'd argue that when it all comes down to it, that because you can spin up the entire infrastructure locally on things like minikube, that it makes it many times more easy to debug than other cloud-provider-only solutions.
Should everyone go back to assembly on bare metal, then?
I think you mean silicon though.
> I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've always found this interesting about web based development. I have no idea what Prometheus, grana, etc do. Ive never used k8s.
And yet, as a solo dev, I've written auto-scaling architecture using, for example, the AWS ec2 apis that let you launch configure and shutdown instances. I don't know what else you need.
Really the only advantage I see to morass of services is you get a common language so other devs can have a slightly easier time of picking up where someone left off. As long as they know all the latest bs.
This is fine, if you’re on AWS and can use AWS APIs. If you’re not (especially if you’re on bare metal), something like K8s can be nice.
In short. Prometheus is a worker that knows about all your other services. Each service of interest can expose an endpoint that prometheus scrapes periodically. So the services just say what the current state is, and prometheus, since it keeps asking, knows and stores what happens over time. Grafana is a web service that uses prometheus as a data source and can visualize it very nicely.
Prometheus also comes with an Alert Manager, where you can set up rules for when to trigger an alert, that can end up as an email or slack integration.
They are all very useful, and gives a much needed insight into how things are going.
Users don't really care if you have a really impressive stack with cool technologies if it doesn't offer anything more than a couple of web servers and a DB server.
Personally I would take managing "a couple of webservers and a db" any day over k8s.
You can find plenty of auto mechanics though. Cars are cheaper and ubiquitous. Maybe they can't drive to the moon, but they can get most things done.
Unless your business is flying to the moon, stick to cars and trucks over spaceships.
New lead came in, got rid of that crap by using 3rd party services to spin up infrastructure. Got a load balancer, a few VMs + DB. Reduced the cost down by 85% and greatly simplified the entire stack.
I learned a really valuable lesson without having to make that mistake myself.
I understand why people get excited about tooling. It's cool to learn new things and automate stuff away. I'm prone to that myself and do this on my own server when I get that itch.
Having said that, it's wrong to foist this stuff onto an unsuspecting company where the owners don't know any better about tech, that's why they hire other people to do that for them and seeing that just left a bad taste in my mouth for overcomplicated setups.
I get that SV is different, that's why tools like K8 are made and I would jump on those tools on a heartbeat as needed.
But for other smaller businesses, the truth is they just need a boring monolothic load balanced app with a few VMs and a DB sprinkled with 3rd party services for logging or searching or other stuff not core to the business.
> They used Ansible scripts to spin up various AWS services
This seems less about using the "cool/new" tech... rather it's about using the "right" tech. Config management tools like Ansible/Chef/Puppet are very much previous-generation when it comes to cloud infrastructure.
They... can manage cloud infrastructure, but they were created prior to the ubiquity of cloud deployments, and the features are glued on. Not choosing a more modern IaC framework tells me they(those devs) were going to be making sub-optimal implementation decisions regardless.
I'm sure there are plenty of use cases where that makes sense, and is a better approach. But, I disagree that k8s suggests a NIH-mindset.
I will always recommend using whatever container / function as a service e.g. ECS, GCF, Lambda any day over K8s for a startup. With these services its back to more similar models of security such as networking rules, dependency scanning, authorization and access...
Realistically, on all flash, you are very unlikely to need to maintain anything on a server for a few years after deployment.
If and only if your business model depends on it. A startup's job is mostly to find product market fit; if being decoupled from AWS isn't part of your market, you are spending money on a non-problem.
The reality is unless you’re some rich dude who can borrow dad’s datacenter (And that’s cool if so), you’re either going to be renting colo space, virtual servers, etc.
It’s always a challenge in business to avoid the trap of spending dollars to save pennies.
IMO, you’re better off working in AWS/GCP/Azure and engineering around the strengths of those platforms. That’s all about team and engineering discipline. I’m not in the startup world, but I’ve seen people lift and shift on-prem architecture and business process to cloud and set money on fire. Likewise, I’ve seen systems that reduced 5 year TCO by 80% by building to the platform strengths.
I'm aware that no man is an island in some sense, but I'm not comfortable with locking myself into one of 3 companies who need to increase their revenue by double digits year over year. And as you say, a lift and shift is basically setting money on fire. Currently I run sort of a hybrid approach with a small IaaS provider and a colo. It seems to work well for us both technically and financially though that seems to go contrary to what is considered conventional wisdom these days.
Where I work, we can deliver most services cheaper on-prem due to our relative scale and cloud margins. But… we’re finding that vendors in the hardware space struggle to meet their SLAs. (Look at HPE — they literally sold off their field services teams and only have 1-2 engineers covering huge geographic regions. So increasingly critical workloads make the most sense in the cloud.
Google is not evil. They're just big, and define some practices which we might think should change in the future.
Once you have the thing up and running, you can start to think about hosting your own.
Also, you don't need to use fancy services because most startups can run just fine on a single instance of whatever, meaning, there are a lot of cloud providers out there.
As a rule, for anything in an startup, adding that "scalable" adjective is a waste.
Of course, exceptions exist, but it's one of those problems that if you have them, you probably know it, and if you have them and don't know it, you won't succeed anyway. So any useful advice has the form of "don't".
So if you already know Kubernetes, then great, use it. Leverage your existing skills. If you don't, then just use Heroku or fly.io or whatever, or go with AWS if that's your competence. Maybe revisit in a year or two and maybe then you'll have funding to hire a devops person or time to spend a week or two learning to do it yourself. Right now you want to get your SAAS MVP in front of customers and focus on the inevitable product churn of building something people want to pay for. The same advice goes for anything else in your stack. Do you know it well enough? Do you need it right now? Or is there a "good enough" alternative you can use instead?
Taking human resource information into consideration sounds very wise. Although, learning a new language is generally not that a huge barrier, while changing your whole stack once the minimum viable product cap is passed can be very expensive. And if you need to scale the team, the available developer pool is not the same depending on which technology you have to stick with.
It doesn’t invalidate your point, but maybe it brings some relevant nuances.
Kubernetes is not 'harder' if all you need is to host a webapp. Where it falls on the hardness spectrum depends on what you are trying to do, and what is the alternative. I am very fluent with Kubernetes but have no skills in managing traditional virtual machines.
And you're also forgetting Android and macOS and Linux.
That's why cross-platform frameworks like Electron and React Native are so popular. The time wasted in going native for every single platform is just infeasible for most non-huge companies.
Here is an example of a team that is doing great work in mobile, frontend and backend:
3 people doing native iOS, 2 people doing native Android, 3 backend engineers, 1 frontend and 1 QA.
Two engineering managers/team leaders: one for mobile and one for web.
Of course this is one single product offering native mobile apps and some limited web app functionalities.
The apps are great, smooth, nice UX, works fast, offers native experience.
Is this a huge company? I don't think so.
Rather than the author "not knowing" what they're talking about, I suspect they're taking narrow experience and generalising it to the entire industry. Their background is selling k8s as a solution to small/medium enterprises: it strikes me that there may be a strong correlation between startups interested in that offering and those deploying failed overengineered multilang micro-architectures. Suspect the author has seen their fair share of bad multilang stacks and not a lot of counter examples.
The sole exception to this rule I would make is javascript which is more or less required for frontend stuff and should be avoided like the plague for any other development. As soon as you can get your frontend done in Rust, though, you should also switch.
Of course there's some balance needed. If your team is familiar with some niche language then long term that might not be a good strategy if you intend to bring more devs on board later.
One side of this which I don't think is discussed often is the fun of choosing new technology. How do you balance having fun and being realistic at the same time?
Fun meaning trying new technology, learning as you go, setting up systems that make you feel proud, etc. It can lead to failure, but I think having fun is important too.
Being charitable, what I think they are getting at is maybe more about having fully separated frontend and backend applications (since the front-end examples he gives are not languages but frameworks / libraries). Otherwise it seems really backwards - I'm definitely an advocate of not always needing SPA-type libraries, but using literally zero Javascript unless your backend is also JS seems like it goes to a too-far extreme.
Er.
I read this as him saying "one of the things I've seen as a bad reason to use Kubernetes is that there are multiple languages in use."
I've seen people do this. Frontend in one container, backend in another, Kube to manage it.
If that's what author meant, author is right, that's a profoundly stupid (and common) reason to involve Kube.
Kubernetes is a perfectly good platform for any size operation, but until you are a large org, just use a managed service from Google/Amazon/DigitalOcean/whoever. Kubernetes, the data plane, is really no more complex that eg Docker Compose, and with managed services, the control plane won't bother you.
K8s allows composability of apps/services/authentication/monitoring/logging/etc in a standardised way, much more so than any roll-your-own or 3rd-party alternative IMO, the OSS ecosystem around it is large and getting larger, and the "StackOverflowability" is strong too (ie you can look up the answers to most questions easily).
So, TLDR, just use a managed K8s until you properly need your own cluster.
My k8s "cluster" is a single node that doesn't cost me any more than a VM would. I don't need the scalability at the moment. But I do need different environments for dev, qa, and prod. And all three are running identically next to each other using Namespaces. Saved us a ton of maintenance and cost.
Any project that grows has its needs change. GKE gives you a ton of integrated tools right from the start including logging, alerting, metrics, easy access to hosted databases, pub/sub, object storage, easy and automatic network setup, easy firewall setup, dns management, and a lot more. k8s is no different than using any other hosted service. It provides a great set of features that you configure using fairly consistent yaml configuration files. And it is all accessible from the web based "Google Console" as well.
Learning the k8s yaml format and some basic kubectl commands is all you need to get going and it saves a TON of time that can go back into developing your application rather than dealing with configuring disparate pieces with their own configuration methods.
I was fairly early to k8s while they were still competing with other similar solutions and other tools like Puppet and Chef. I tested all of them and truthfully, k8s was the easiest to learn, implement, and maintain my app with. Using GKE of course. I would NEVER as a one man or even small team of developers take on managing an installation of k8s myself.
This was my first thought, and I was to comment so, but saw you already did. The only reason we see this comment is because HN has an irrational hate of K8s, for us that do run things in production at scale, k8s is the best option. The rest is either wrapped in licenses or lacks basic functionality.
I like to call what the author is referring to as, "What-If Engineering". It's the science of thinking you'll be Google next week, so you build for that level of scale today. It involves picking extremely complicated, expensive (both in compute and skilled labour) technologies to deploy a Rails app that has two features. And it all boils down to, "But what if..." pre-optimising.
It happens at all levels.
At the individual unit level: "I'll make these four lines of code a function in case I need to call it more than once later on - you know, what if that's needed?"
It also happens at the database level: "What if we need to change the schema later on? Do we really want to be restricted to a hard schema in MySQL? Let's use MongoDB".
What's even worse, is Helm and the likes make it possible to spin up these kinds of solutions in a heart beat. And, as witnessed and evidenced by several comments below, developers think that's that... all done. It's a perfect solution. It won't fail because K8s will manage it. Oh boy.
Start with a monolith on two VMs and a load balancer. Chips and networks are cheaper than labour, and right now, anyone with K8s experience is demanding $150k + 10% Superannuation here in Australia... minimum!
Good luck to them, but they're not going to occupy time and space in my mind.
We seem to mostly manage to avoid that in my department, but we have a very low turnover.
We'd like that (I'd like that), but resume-driven choices are a very large driver of technology direction, unfortunately.
It means those of who want to build something very maintainable and very stable using the most boring (stable, secure) technology possible are often outnumbered.
I couldn't help but rewrite the entire thing, and now it's a single 8K SLOC server in App Engine, down from about 70K SLOC.
Also, there's no inherent lock-in, you can basically just deploy it somewhere else.
Data is where the lock-in lies. Moving can be hard if you use proprietary databases. Can still be worth it.
That means very little, I hope you realize. Reader, Voice, Chat, etc.[0] were all around a long time.
>Also, there's no inherent lock-in
GAE has plenty of proprietary APIs you can depend on. Whether or not you do is up to the programmer.
0 - A comment below notes that voice and chat aren't deprecated yet. Voice was announced deprecated, and google has had so many chat apps I'm not sure which ones are gone. Anyway, here is a more complete list of things Google has abandoned: https://killedbygoogle.com/
Someone convinced the right person to “put stuff on kubernetes” and then booked it for another job/greener pastures with barely any production services, or networking configured.
Thus an opening for a new SRE and once again I find myself looking at a five headed monster of confusing ingress rules, unnamed/unlabeled pods, unclaimed volume specs…
Sigh.
I agree entirely too.
>> Start with a monolith on two VMs and a load balancer. Chips and networks are cheaper than labour,
Kudos to you! You are a dangerous man for you opine the truth.
My advice is generally, "Build something. Then see if you can sell it." or "Sell something and then go build it." Either way, it all starts soooo small that the infrastructure is hardly a problem.
If you "get lucky" and things really take off. Sure. Yeah. Then get a DevOpps superstar to build what you need. In reality, your business will very probably fail.
>> You can’t just hire 1 DevOps superstar
Secondly, that assertion is not necessarily true. Obviously, you should just hire 1 DevOps superstar... in some cases.
Don't nitpick and don't argue foolishly.
You don't go from zero to needing global 24x7 support overnight.
Hiring 1 DevOps superstar is exactly what we did a few startups back and it worked great. Of course there was no after-hours support, it's a small startup. Eventually the team grew.
I’d just caution that when it comes to the mental health of the one person in this role, even if they seem like they are doing ok, check in frequently.
No. DevOps is about the development and operations disciplines working together in a cross functional way to deliver software faster and more reliably.
In a small enough startup both disciplines may be represented by a single person, though.
News to me. I work at one the biggest companies around, and our DevOps posture is pretty great, and that was achieved by cross functional teams from the historic dev and historic ops teams doing all the things that Accelerate codified.
Genuine question, why is this bad? Is it because k8s can spin it up but becomes unreliable later? I think the industry wants something like k8s - define a deployment in a file and have that work across cloud providers and even on premise. Why can't we have that? It's just machines on a network after all. Maybe k8s itself is just buggy and unreliable but I'm hopeful that something like it becomes ubiquitous eventually.
It's everything that comes afterwards that gets neglected. Not by everyone, granted, but by most.
There's other reasons to use K8s than just thinking of massive scale.
Setting up environments becomes a massive PITA when working directly with VMs. The end result is either custom scripts, which is a messier version of terraform, which ends up being messier than just writing a couple of manifest files for a managed k8s.
> anyone with K8s experience is demanding $150k + 10% Superannuation here in Australia... minimum!
sheds a tear for CAD salaries and poor career decisions
The author advocates using a high-level PaaS. For sure working directly with VMs is the wrong answer to premature optimization, but as an early stage startup, there's plenty of services around that you basically just need to point your Git repo at and you'll have a reasonable infrastructure set up for you.
I guess I've just never found this to be true.
My main goal when engineering a solution is always, "Keep It Super Simple (KISS), so that a junior engineer can maintain and evolve it."
Working directly with operating systems, VMs, networking, etc. (purely in Cloud - never on-prem... come on it's 2022!) is the simplest form of engineering and is much easier than most claim.
The code example is not always right.
Beware, if you know it will be needed, you might as well make it a function now. Likewise if you think probably it will be needed, why not make it a function now?
It’s not a good review comment or rejection to say “yeah but I don’t want to do that because it’s not yet needed”. Sure, but what if you are just being lazy and you don’t appreciate what it should look like long term?
The “I don’t want to write a function yet not needed” is not a clear cut example.
I wasn't aware that some devs have a side hustle as fortune tellers?
On a more serious note, you should take a look at Sandi Metz's "the wrong abstraction". https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction
A simple example. If you are making a multi-page website that contains a header on all pages, you can separate the header into a component from the get go, instead of duplicating the header for each page and then abstracting it in the future (where it starts to become more work).
I would very likely agree in all cases.
However, they only represent 4-5 users out of all 3,400. And that's the issue - only a small fraction of the industry operates at that scale :-)
Level 2) you pack your software into an ami and use the update the asg config. You can periodically "drain" the asg of old instances
Level 3) you deploy your stack again with the new stack having the ami that you've build at level 2 referenced. You start shifting traffic between the old stack and the new stack. You monitor and rollback if something is wrong.
But to answer the question, VMs have been a thing on the desktop for a very long time.
we also live in a world that does not appreciate well crafted software and a lot of things are driven by the desire to build a resume. I've maintained code that was decades old and was amazing to work with and was still generating ridiculous amounts of money. I've also worked on code that was just written and used all the possible bells and whistles and development speed grinded to a halt once the it's been around for more than a couple of months.
My worst case scenario is having to work on code where the original developer didn't understand what they were doing and they just wanted to use X. Double the trouble if they didn't master X when the thing was put together.
Now you've got automatic, constantly updating VMs every night if you want them. And a new deployment is just commiting code to master and pushing and that whole pipeline triggers for you.
People like to overcomplicate things, Mirceal. You're on the right path :-)
If you need a service to manage K8s for you, then that's a red flag already (regarding K8s, not you personally.) If the service is so complicated that experienced engineers tell me constantly that only managed K8s is the way to do it, that tells me enough about why it's going to be a rough journey that should probably be avoided with IaaS or PaaS.
> ... and wildly a better pattern than just running VMs.
I've never had an issue running VMs, personally. And when I join a firm and begin helping them, I find I can very quickly and easily come up to speed with their infrastructure when it's based on straight IaaS/SaasS/PaaS. If it's K8s, it's often way more complicated ("Hey! We should template our YAML configs and then use Ansible to produce the Helm charts and then run those against the infra!" - ha ha!)
It's really not, k8s does a ton for you, trying to do the same with VMs would be unbelievably complex
> that should probably be avoided with IaaS or PaaS.
PaaS is great till it isn't, seen plenty of companies hit the edges of PaaS and then need to move to k8s
> infrastructure when it's based on straight IaaS/SaasS/PaaS
Again this is great till it isn't (see heroku) and then people move to k8s. Having control and understanding of the underlying infrastructure is important unless your just running some basic web app
The main issue I have seen in startups is premature architecture complexity. Lambda soup, multiple databases, self-managed message brokers, unnecessary caching, microservices etc. Whether you use K8s or not, architectural complexity will bite your head off at small scales. K8s is an enabler for overly complicated architectures but it is not problematic with simple ones.
>Did users ask for this?
Not an argument. Users don't ask for implementation details. They don't ask us to use Git or build automation or React. But if you always opt for less sophisticated workflows and technologies in the name of "just getting stuff done right now" you will end up bogged down really quickly. As in, weeks or months. I've worked with teams who wanted to email source archives around because Git was "too complicated." At some point you have to make the call of what is and isn't worth it. And that depends on the product, the team, projected future decisions and so on.
I started out with Iaas namely Google App Engine and we suffered a ton with huge bills especially from our managed db instance. Once the costs were too high we moved to VMs. Doing deployments was fine but complicated enough that only seasoned team members could do it safely. We needed to build a lot of checks, monitoring etc to do this safely. A bunch of random scripts existed to set things up and migrating base operating system etc required a ton of time. Moving to kubernetes was a breath of fresh air and I wish we'd done it earlier. We now have an easy repeatable process . Infra is easier to understand. Rollouts are safer and honestly, the system is safer too. We know exactly what ports can allow ingress, what service boundaries exist. What cronjobs are configured, their state etc with simple kubectl commands.
Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge
Kubernetes isn't just about global scale that most people will never need, which would agree with the article. It is about deploying new apps to an existing production system really quickly and easily. We can deploy a new app alongside an old app and proxy between them. Setting up a new application on IIS or a new web server to scale is a mare, doing the same on AKS (managed!) is a breeze. It is also really good value for money, because we can scale relatively quickly compared to dedicated servers.
It is also harder to break something existing with a new deployment because of the container isolation. We might not need 1000 email services now but we could very quickly need that kind of scale and I don't want to be running up 100s of VMs at short notice as the business starts taking off when I can simply scale out the K8S deployment and add a few nodes. There is relatively little extra work (a dockerfile?) compared to hosting the same services on a web server.
I disagree.
With octopus deploy I can add a step, set a package and hostname, and press a button, and have a new deployment of a new api or website in IIS pushed out to how ever many servers currently exist in a few minutes.
There are many ways to manage deploying services, and scaling, without K8S or containers in general.
While I could setup a new Octopus Deploy quickly, it would be like you setting up something in AKS. We are both good at the tools we know. But saying my way or your way is wrong - is the thing thats wrong.
- How does it decide which servers to deploy to?
- Does it scale up the number of servers if the deployment is too large?
- What happens when a server dies?
Servers get have a tentacle which has an environment and tag. So you could say a server is test environment and tagged with main-web-app. While another server is production with the same tag. You promote a package from test to production. You configure your setup to say a package is deployed to a server tagged main-web-app.
Octopus is an orchestration tool so isn’t responsible for scaling up. But in AWS with an autoscale group you can configure Octopus to auto detect a new server tag it and deploy. Part of the deployment would add it to the load balancer.
As a side note you can deploy containers using octopus too. Tho I’ve never found a reason to use containers in production yet.
I would say they will be.
> ton with huge bills especially from our managed db instance
This doesn't have much to do with App Engine, right? Last time I used it, we were using a PostgresQL instance on AWS and had no problems with that.
> Doing deployments was fine but complicated enough that only seasoned team members could do it safely
I just plain don't believe this. I bet you were doing something wrong. How is it possible that the team find too difficult to do an App Engine deployment but then they're able to setup a full kubernetes cluster with all the stuff surrounding it? It's like saying I'm using React because JavaScript is too difficult.
> Using kubernetes forces you to write configurable code and is very similar to testing: it sounds like it'll slow you down and shouldn't be invested in until the codebase is at a certain size but we've all learned from experience how is actually speeds everything up, makes larger changes faster, cheaper customer support and saves you from explaining why a certain feature has been broken for 10 without anyone's knowledge
This is far, far, far from my own experience.
Some other questions:
How did you implement canary deployment?
How much time are you investing in upgrading Kubernetes, and the underlying nodes operating systems?
How did kubernetes solve the large database bills issue? How are you doing backups and restoration of the database now?
If I were to found a company, specially not VC founded, dealing with kubernetes would be definitely far below on my list of priorities. But that's just me.
No more load balancers, no more VMs, no more scaling up or down to match demands.
This is a huge redflag.
It's basically admitting that you expect most later employees to not understand k8s or how its being used. You may think they don't need because it works, but you have to think about what happens when it doesn't work.
The shops I've been to all had the same mindset: the docker/k8s infra was setup by one guy (or two) and no one else on the team understands what's going on, let alone have the ability to debug the configuration or fix problems with it.
Another thing that might happen is some team members understand just barely enough to be able to "add" things to the config files. Over time the config files accumulate so much cruft, no one knows what configuration is used for what anymore.
Custom VM/cloud/$randomSaaS deployments are much worse when it comes to "the one guy who understands the intricate details is on vacation".
The underlying assumption behind my comment is that you really really want to simplify your deployment as much as you can.
Unfortunately this is nearly impossible with the currently accept set of standard best practices for developing web applications, where you use a scripting language and like six different database systems (one for source of truth on data, one for caching, one for full text search, and who knows what else everyone is using these days; I honestly can't keep track).
There are many ways to make the deployment process simple. My rule of thumb is to use a compiled language (Go) and an embedded database engine.
The levels.io guy famously did it even with php https://twitter.com/levelsio/status/1102487697220820994
That's also exactly what would happen if you homebrewed your own system. You need to centralize some part of expertize around infra at some point, but hopefully around more than two people.
It is realistic to expect a small team to build the same in an industry standard way and hand of a clear, well documented API to the rest of the team.
Bespoke solutions need to deal with this complexity somehow, k8s provides a standard way to do so.
The difference is, kubernetes is pretty standardized and therefore learnable in a repeatable way, unlike the Frankenstein of tooling you might otherwise cobble together.
Sure. Though the standard changes weekly.
v1alpha1
And the good thing is the graduation process means that this can actually be relied upon not to break compatibility.
One guy (okay it was two guys) set up all the infrastructure and as soon as it was working bounced to new jobs with their newfound experience. The result is that dozens of engineers have no idea what the heck is going on and are lost navigating the numerous repos that hold various information related to deploying your feature.
In my opinion (and I’m sure my opinion has flaws!), unless you have global customers and your actual users are pushing 1k requests per second load on your application servers/services, there is no reason to have these levels of abstractions. However once this becomes reality I think everyone working on that responsibility needs to learn k8s and whatever else. Otherwise you are screwed once the dude who set this up left for another job.
And honestly.. I’ve built software using node for the application services and managed Postgres/cache instances with basic replication to handle heavy traffic (10-20k rps) within 100ms. It requires heavy use of YAGNI and a bit of creativity tho which engineers seem to hate because they may not get to use the latest and shiniest tech. Totally understand but if you want money printer to go brrr you need to use the right tool for the job at the right time.
Again, I really see this the result of VC money chasing large valuations, and business decisions influencing technical archietcture, a sign of our times, of exuberance and senselessness
Engineering has to raise the cost of engineering to match it (280 character limit crud app on AWS lambda with 2 full stack developers vs 2000 devs in an expensive office).
We never had an issue with multiple languages across tier-n architectures.
Suddenly with the uptake of HTML 5, it became an issue not being able to use JavaScript everywhere.
That does ignore the fact that an experienced frontend JS dev is not necessarily also a good productive backend JS dev, but at least they know the language basics, can use the same IDE, etc.
If thats worth it is something that depends on what you try to achive, I guess. I personally would not pick JS (nor TS) for the backend.
In practice though the main reason is that to have decent DOM bindings you need to stabilize many other specs first (unless you do a ultra-specific DOM-only extension, but nobody wants that)
Rust, for example, has a huge package with interfaces for every DOM function.
But I wouldn't recommend it in general, because JS (and for slightly more complex things, TS) is much easier to create because of all the old Rust features that surface every time you mention it. And most GUI code is simple enough that you only have to write once, maybe fix a few details, and forget about it.
Wasm would be much more compelling if it was target by higher level languages.
The rise of the "fullstack developer" has mostly reduced quality across the board. When you hire a "fullstack developer with 5 years experience" you aren't getting someone who is as good as a frontend developer with 5 years AND a backend developer with 5 years but someone that adds up to 5 years split between those 2 endeavors but probably with less depth as a result of switching.
(as a side note I also think it's contributed to developer title inflation)
Learning a new language and its tooling is comparatively easy compared to learning the required domain knowledge to be effective in a new area. i.e transition from frontend -> backend or visa versa has very little to do with the language or tooling.
Your average frontend dev doesn't know squat about RDBMS schema design or query optimisation, probably aren't familar with backend observability patterns like logging, metrics and tracing, most likely have very little experience thinking about consistency and concurrency in distributed systems, etc, etc.
Just like the backend dev is going to struggle with the constraints of working in a frontend context, understanding UI interactions, optimizing for browser paint performance, dealing with layout and responsiveness, etc.
Meanwhile if you know say Java and some scripting language, say Python and you end up a new job doing backend in JS it's not going to take long for you to pick up JS and hit the ground running because you are going to encounter exactly the same stuff just different syntax and runtime.
Backend being substantially divorced from frontend isn't a bad thing, it's generally a good thing that results in nice clean separation of concerns and healthy push-pull in design decisions about where certain logic and responsibilities should lie etc.
The bar is generally just a lot lower now across the board. We have people with the title "Senior Engineer" that still regularly need mentorship and peer-programming sessions.
The title I have now (Principal Engineer) I feel like I don't deserve, it was reserved for folks I looked up to when I was starting and I don't think I have reached their level. Yet at the same time it's essentially necessary to distinguish myself from what is now considered "Senior".
I have a separate rant for the lack of proper juniors and lack of structured mentoring etc but that is for another day.
Having a solo developer that can do it all well enough also allows useful products to reach users faster (and for me it's really about solving user problems, not just making money), without getting derailed by communication overhead, bureaucracy, fighting within the team, etc. Just make sure you don't pick a developer who cares too much about things that mostly matter to other nerds, because then they might get derailed with premature optimization. Yeah, I've been there.
The chunking unit is programming languages which means as long as you can call the task the specific programming language, people will believe it's the same thing.
In reality you have a hammer expert and are in the business of making everything look like a nail.
So now we have complicated bloated applications of JavaScript in places where it's absolutely inappropriate and you need far more skill to navigate those waters then someone who uses more appropriate tools.
It's a perversion in the name of simplicity because we're forcing too coarse of a model on too fine of a problem and as a result everything explodes in complexity, takes too long, is too expensive and comes out working like trash.
We could do better but first we have to eat crow and admit how foundationally wrong we were and frankly things aren't bad enough to make that mandatory so the circus continues just like it did when we tried to make everything OOP and it made the same smell.
They're useful tools, but that's their perimeter; they aren't zeitgeists... well they shouldn't be.
It's a very effective way to make sure your team is only composed by junior developers.
I write elixir that smells like rails, but as the months move on, I change.
You also see it when people who've mostly done imperative programming tries their hands with a lisp or some ml based language. I've been there myself. You still find yourself using the equivalent of variable bindings, trying to write in an imperative rather than declarative style.
I guess when trying to learn a new language in a different paradigm you also need to unlearn a lot of the concepts from the former language.
Someone learning a first language isn't just learning a new language: they're learning how to program. It's a new profession, a new hobby, a new superpower.
The rest of the stuff (standard library, ecosystem, buildsystem and all kinds of intricacies) is just a mix of trivia and bureaucratic garbage you gotta fill your brain with but will all be replaced in 10 years anyway. Sure it takes time but it's nowhere near as important as actually knowing how to program.
Even changing paradigms (imperative to functional) isn't as hard as learning the first language.
x = 1
for i in [1, 2, 3] {
x = i
}
A lot of beginning programmers struggle with this kind of stuff, as did I. Reading this fluently and keeping track of what variables are set to takes quite a bit of practice.
I've been hired to work on languages I had no prior experience on, and while there was of course some ramp-up time and such, overall I managed pretty well because variables are variables, ifs are ifs, loops are loops, etc.
The argument that learning a second language is as difficult as learning the first doesn’t really hold water in practice as well, lots of people have done so.
IME a lot of the conceptual stuff transfers pretty well there too.
Most job offers are for a mythical full-stack developper that'll master web design, CSS/HTML, front-end interactions and code, networking, back-end architectures, security,... You end-up with people who don't have time to get enough expertise and write and build clean stuff. Hacking poor JavaScript code everywhere.
With the same language, you may think you can somehow reuse stuff between front and back. A bad idea in most project, will typically create more problem than it'll solve.
If you do server-side rendering with something like nextjs, then its quite a lot of code.
With trpc you can share types without going through an intermediary schema language (https://trpc.io/) although I think it would've still benefitted from separating the schema from the implementation (hope you don't forget the type specifier and import your backend into your frontend by accident)
For business logic as always the anwwer is "it depends". Does your app benefit from zero-latency business logic updates? Do you need rich "work offline" functionality combined with a rich "run automation while I'm away" for the same processes? etc.
I'm a big Nextjs fan; I just think it's useful to emphasize that using it doesn't necessarily have to mean "only javascript/typescript on the backend".
> While Remix can serve as your fullstack application,
Isn't that where most teams should start, and where they should stay unless they have a really good reason to get more complicated? This is what I was thinking of while reading the GP comment about Next.js.
So, if you can keep all of your templates on a single place, and don't have a lot of data diversity, you won't have a problem. But if that's not your case, you do have a problem.
Personally, I think the frontend code should be more generic on its data and organized around operations that are different from the ones the backend sees. But I never could turn that ideal into a concrete useful style either.
Better to have a full stack developer that can concentrate on becoming an expert and fluent in just one language rather than being kinda-ok in 3 or 4 IMHO.
The real expertise is being a front-end expert, authoring efficient architecture around user interactions, browsing the MDN without effort, mastering the DOM... Then, being a back-end expert, knowledgeable on scaling, could architectures, security issues, being able to author good API design...
If you can be considered an expert in all of this, and also edge computing, I don't think switching language would be an issue for you. Language is a tiny fraction of required expertise and you might be more productive by switching to an appropriate one.
And they don't really have time to learn new stuff, as they spend too much of their time on Hacker News complaining there's too many new frontend frameworks or something like that.
It is extremely easier to find more affordable talent that has come out of a boot camps knowing javascript but more specifically "react/nodejs" which lets them work both frontend and backend. They rarely know best practices, how to properly debug & troubleshoot a problem that isn't a Google search away, etc, but they will be hungry and work their butts off to ship their CRUD style features.
Also I've seen UX research that established some guidelines for how long something can take while still feeling responsive allowing a user to remain focused.
Knowing you can achieve the required/desired performance in any given language is mostly a matter of experience solving similar problems.
But the overwhelming majority of the time you don't need it, at least not yet. I would say that unless you have actual evidence that your other language would not be adequate - i.e. an implementation of your system or some representative subset of it, in your main language, that you spent a reasonable amount of time profiling and optimizing, and that still proved to have inadequate performance - then it is indeed premature.
This I very different to say starting a with a micro service architecture which imposes relatively high overheads, with little benefit as splitting up a well designed monolith is easy.
The cases where Go is significantly faster than JavaScript are vanishingly small.
> whereas rewriting it later would be a massive undertaking.
This is vastly overstated IME. Porting existing code as-is between languages is actually pretty easy.
> This I very different to say starting a with a micro service architecture which imposes relatively high overheads,
Disagree. You're imposing a huge overhead on hiring (if you want people with both languages) or on people's ability to work on the whole system (if you're happy hiring people who only cover one side or the other). Debugging also gets a lot harder. There's essentially twice as many tools to learn.
Yep. We always implemented (parts of) embedded software in Java first and later in JS and then port it. If no additional functionality is added, this is trivial and saves a lot of work and errors as you already tested, debugged and fixed the logic parts.
It's not only about performance but also energy and resource efficiency.
Which is to say ... while JS developers do know how to profile code, people who are routinely exposed to this problem are not going to be JS developers. The people who are good at identifying when a system needs to be re-written in Go for performance reasons are probably already Go developers and people who have lots of experience writing performant code.
Plus writing in that sort of language from the start means there is a chance to segue into performant code without needing to rewrite things in a new language.
Sure they are. The skills aren't really language-dependent, and nowadays machine code is so far away from the actual hardware behaviour that it doesn't actually help a lot. Besides, the biggest speedups still come from finding errors or inappropriate algorithms or datastructures, and that's if anything easier to spot in a higher-level language where there's less ceremony to get in your way.
when I interview developers, folks coding in Java can usually tell me what an appropriate datastructure would be, while half the JS developers can't explain how an array is different from a linked list. Because most of the time JS developers. dont have to think about it much.
Well said: this should be the only rule!
"Using different languages" is just Tuesday in most places. Sure, don't use languages needlessly, but it's not a big hurdle unless you're just a "nodejs bro"
Given that probably most developers support the push to tackle the climate change, they seem to be making no effort to ensure their apps execute in as short time as possible using as little resources as possible. You would expect that people would actually embrace doing things in C or Go to save energy. Maybe cloud providers should think of showing the carbon footprint as a first class performance metric.
When you want to save the climate there are many, many low-hanging fruits. The choice of programming language is likely not one of them, as much as I like efficient code.
Like so many others, the author appears to be latching on to the phrase "premature optimization" as a popular buzzword (buzz...phrase?). This is so far from what Knuth actually wrote in his book that it hurts.
That's fine.
Premature optimization would be saying:
I don't know Go/C++/C... but I know its fast, so instead of using what I know to get up and running quickly, i'll waste time building it in something I don't know which probably wont be as fast as doing it in something I know well.
The thing is, when you're building it, no one is using it! If it's slower in RoR than Go, who cares, get it up and running, and fix it later when people are actually using it.
Setting up a managed kubernetes cluster for a single containerized application is currently no more complicated than setting up AWS Lambda.
What you get out of it for free is amazing though. The main one for me is simplicity - each deployment is a single command, which can be (but doesn't have to be) triggered by CI. I can compare this to the previous situation of running "docker-compose up" on multiple hosts. Then, if what you're just deploying is broken, Kubernetes will tell you and will not route traffic to the new pods. Nothing else comes close to this. Zero-downtime deployments is a nice bonus. Simple scaling, just add or remove a node, and you're set.
Oh, and finally, you can take your setup to a different provider, and only need some tweaks on the Ingress.
You pay AWS/GCP for that these days and just use the API.
I have used Managed K8s for 4 years and literally never had any problems with CNI. My clusters runs with no problems.
Things get much simpler if you try to limit CNI complexity by going towards at least conceptually simpler tooling that matches original, pre-CNI design of k8s, IMHO.
I don’t sleep these days.
The second is a nice, clean EKS app. Developers build containers and I bind them with configuration drop'em like they're hot right where they belong. The builds are simple. The deployments are simple. Most importantly there are clear expectations for both operations/scaling and development. This makes both groups move quickly and with little need for coordination.
You were able to do it, but what happens to them when you're not around? Does their team have the required experience to handle it? That's the difference in cost. It's like DYI - yes, if I have all the skills and experience I can do everything myself incredibly cheaply, but... I don't. So I gotta pay.
But if your team does have the skills and experience, it's definitely worth looking into.
I do think people deeply underestimate what can be achieved with a single (or two, if you need a standby) dedicated Linux server these days. A single server can easily go up to 2TB of RAM and 128+ cores. Long before you ever get to a scale where that's a limitation, you'll have more than enough resources to figure things out.
Exactly - $950/mo is nowhere near enough to pay for those skills if you don't have them. It's good value for money.
My first deployment involved virtual machines, but we were essentially packing as much as possible into lowest amount of VMs, then doubled it up so that we had failover. This way we had clear visibility of how many resources we were using and could allocate them according to how much we had.
So disaster recovery is spin a new box, pull the most recent db backup in, and update dns - all scripted should I be gone.
This is why you pay someone. Saving $950/month means it's well worth spending $500 for a day of someone's time occasionally. You don't have to do everything internally when you run a startup. Buying in services that you only need occasionally is well worth the money.
Besides, there are people out there who offer on-call services for a small retainer.
And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.
And you'd be surprised how little problems you might have - I've had many VMs with literally years of uptime running without issue.
Most people focus on "devops" as a job - and they never bother to teach the rest of the team anything about how stuff works. Worse than that, modern clouds encourage you to build opaque and complicated systems which probably only someone working on them full-time has a hope to understand...
Culture doesn't mean they'll know how to fix <some weird edge case> at 3am in the morning. The SRE with constant ops exposure is likely to have a much better chance, if only because they (ought to) really know how to debug it.
I hope they all have fully patched base libraries and kernels, security auditing is getting to a much more common requirement these days even among very small companies. For example, anybody using Facebook Login.
This is why we document things at the company I work for. If you're serious about a project and want it to exist for a long time there will be things that only come up every few years, and you won't remember them. You can either write things down or you have to relearn how things work every time. Wroting things down is much easier.
But this applies to everything. You also need Heroku or Kubernetes or whatever experience to maintain those systems, right?
LOL - are you f'ing kidding?
Running VMs is much closer to running your app locally than containers and k8s - it's not that hard to do badly, and only marginally harder to do well.
We've lost that, and now everyone thinks we need "sysadmin/devops" for simple and moderately complicated deploys. Heck, most of the guides are out there, follow them. Also, ready-made images and docker containers are amazing these days with config built in. If you look-for or hire devops, you get K8s and all the "formal" items they'll bring with them. You don't need CI/CD for the first 6 months, just PoC the damn thing, get an MVP out and start getting users/customers, the rest will follow.
These systems rarely were constrained in performance by scaling issues that would have be solved by scaling horizontally, and in many cases the added latency of doing so would have caused more problems than it would have solved.
And as you say, having everything on one or two VMs is not just orders of magnitude cheaper hosting, but also comes with benefits of much more easy debugging and performance monitoring.
These weren't tiny start-ups either, these were long running services contracted out to clients including the government and other major companies.
It doesn't scale infinitely, but I'd wager that the traffic just isn't there to justify these kinds of setups in 99% of cases, and that some of the scaling needed is because all the added latency and service discovery etc is adding overhead that wouldn't be needed without it.
I've often found it odd that people who strive for YAGNI at the code level don't apply the same to the system architecture level.
Kubernetes is the right solution to a difficult problem you may or may not have in your future.
I guess redundancy is not really a thing then?
With serverless offerings you can get rather good deals, I don't think you need your own K8S cluster if you can get away with a single Linux host, but a single Linux host is pretty pricey maintenance wise compared to google Cloud Spanner and cloud run.
Most of the time it really doesn't need to be. In the end what you care about is uptime and cost. A redundant solution doesn't have a perfect uptime just because it's redundant, in fact sometimes it might have even less uptime because of failures in the redundancy mechanism. Of course if you need to be always up it might be worth it. But for a lot of situations some downtime is acceptable and most of the time it's better to be on a simpler setup that's easier to debug and less costly to maintain, than to be on a complex one that requires more maintenance and still doesn't have perfect uptime.
Also, I wouldn't underestimate the cost of maintenance in managed solutions compared to self-hosted. It wouldn't be the first time that the managed solutions screw something up or apply some changes and you don't know whether it's your fault or theirs. You also have to add the cost of adapting your solution to their infrastructure, which is not trivial either.
I'm fairly sure that google does a better job keeping cloud spanner and cloud run working, and their redundancy mechanisms working, than whoever runs your single linux box will do.
> most of the time it's better to be on a simpler setup that's easier to debug and less costly to maintain
Keeping a whole linux box running is more complicated and requires more maintenance than Cloud Run and Cloud Spanner.
> Also, I wouldn't underestimate the cost of maintenance in managed solutions compared to self-hosted.
It is not 0, it is just that if you manage cloud run and cloud spanner you don't manage all the other things you have to manage when you self host, and managing cloud run and spanner is really not a lot of effort, it is a lot less effort than managing a standalone database at least.
> You also have to add the cost of adapting your solution to their infrastructure, which is not trivial either.
Cloud run can run stock standard docker containers, you will have a bad time if your processes are not stateless though, and you will have the best time if you have a 12-factor app, but I would not count that as adapting to infrastructure.
And you can use AlloyDB if you want Postgres.
Most of the uptime loss won't come from your provider but from your applications and configuration. If you use Cloud Run and mess up the configuration for the redundancy, you'll still have downtime. If your application doesn't work well with multiple instances, you'll still have downtime.
> Keeping a whole linux box running is more complicated and requires more maintenance than Cloud Run and Cloud Spanner.
Is it? I keep quite some linux boxes running and they don't really require too much maintenance. Not to mention that when things do not work, I have complete visibility on everything. I doubt Cloud Run provides you with full visibility.
> It is not 0, it is just that if you manage cloud run and cloud spanner you don't manage all the other things you have to manage when you self host, and managing cloud run and spanner is really not a lot of effort, it is a lot less effort than managing a standalone database at least.
Managing a simple standalone database is not a lot of effort. For most cases, specially the ones that can get away with running production on a single box, you'll be ok with "sudo apt install postgresql". That's how much database management you'll have to do.
> Cloud run can run stock standard docker containers, you will have a bad time if your processes are not stateless though, and you will have the best time if you have a 12-factor app, but I would not count that as adapting to infrastructure.
That definitely counts as adapting to infrastructure. For example, if I want to use Cloud Run my container should start fairly quickly, if it doesn't I need an instance running all the time which increases costs.
I'm not saying Cloud Run/Spanner are bad. They'll have their use cases. But for simple deployments it's more complexity and more things to manage, and also more expensive. If doing "apt install postgres; git pull; systemctl start my-service" works and serves its purpose, why would I overcomplicate it with going to redundant systems, managed environments and complex distributed platforms? What do I stand to gain and at what cost?
You'd be surprised - the major cloud providers have outages all the time.
Google in particular will have some random backing service firing 502's seemingly randomly while their dashboards say "all good".
There's always 1-2 comments in these threads that advise using a Linode VM or Hetzner dedicated server in order to save money; but they are really skipping over the headaches that come with building and maintaining your own servers.
- Are the server provisioning scripts source controlled?
- Is there a pipeline if the server needs to be recreated or upgraded?
- How is the server secured?
- Are the server logs shipped somewhere? Is there monitoring in place to see if that is working?
- Does the server support zero-downtime deployments?
- Are backups configured? Are they tested?
I imagine the answer to a lot of these questions is no and to be fair not all PaaS systems provide all of these features (at least not for free).
The server becomes a pet, issues inevitably arise and that $950 starts to look like a false-economy.
If you do it correctly, it is loads easier to reason about and understand than a complex k8s deployment.
But for pete's sakes man, if you have used K8s before, know what you're doing and you're running on cloud, just shove a couple off-the-shelf Terraform modules at the problem and there's your execution environment needs solved, with easy and reliable automations available for handling certs, external access, load balancing, etc, all of it reasonably easy to get going once you have done it before.
Stop pretending Kubernetes is this humongous infrastructure investment that mandates a full time job to keep up at low scale. Of course if you have done this multiple times before you don't need to be told this, but people new to it shouldn't be fed a barrage of exaggerations.
It is a full time job. The cost of using something is not just the setup cost. The same way that software development is not just the cost of writing the software.
"It's easy to setup" speaks nothing to how much effort it takes to maintain.
I've set up EKS multiple times across different companies with few nodes and everything runs smoothly all of the time.
Can you go into detail?
I've seen projects fail or take 10x as long due to choosing to go with k8s day 1 instead of starting with even a basic VM.
It may be easy enough to get k8s itself setup. However, without an in-house expert sat within the dev team, all the access/permissions to resources to/from the cluster were hell. NFS disk Read/Write, S3 Read/Write/List, RDS DB access, Kafka access, corp network access to web server within k8s.. and then all the daemons you need to actually be able to monitor your logs/health of apps/etc.
Each of these things in isolation would be no big deal, except that they'd take a different subset of 10 different infra/devops team members 1-2 weeks to sort the first time, in some fragile manner that frequently fell over.
All of these for an application that was replacing something basically owned by 1 dude as 10% of his job, running on a 10 year old server under cron, with a big disk attached.
Well, with Kubernetes this became container images versions and operator versions that can't be upgraded. Though I agree with your in general.
For example NFS you need to make sure the underlying hardware the k8s is running on has the mount, and then that the pods within have that access.
Whatever host level config you need to do on the VM needs to be basically replicated within each of your container images for the pods you want to run, no?
Then you have the same update cycle except across N container images instead of 1 VM?
Honestly I feel like the original blog post shouldn't be talking about k8s at all. The real lesson, as far as I can tell, is just "when you're starting out, choose simple tech that you already know how to use."
If you already know how to use Kubernetes, then feel free to use it. If not then don't.
Most folks building software at startups and scale-ups should avoid Kubernetes and other premature optimisations. If your company uses Kubernetes, you are likely expending energy on something that doesn't take you towards your mission. You have probably fallen into the trap of premature optimisation.
Please don't take this post to be only aimed against Kubernetes. It is not. I am directing this post at every possible bit of premature optimisation engineers make in the course of building software.
For me at least, Kubernetes has become something like an universal standard: if you're already running Docker or other containerization, "using Kubernetes" is nothing more than a clear, "single source of truth" documentation on what infrastructure your application expects. Certainly beats that Confluence document from 2018, last updated 2019, on how the production server is set up that doesn't even match up with what was reality in 2020 much less today.
All of these much more annoying and harder than `helm update --install`ing some charts to your own cluster.
If you have a bunch of people who know it, you can deploy a cluster into gcloud or aws with a few clicks or lines of IaC.
Would I recommend a startup team learn kube while trying to ship a product? No.
Would I think it's a red flag if a team who already know it choose it as their preferred platform? Also no.
Personally I don't see too much difference between kube yamls and systemd units and cloudformation or whatever and the "cluster maintenance" itself is not the burden it used to be if you stay on the "paved road" provided by your cloud.
On a project a couple of years ago my cofounder and I opted to use Kubernetes after running into constant frustration after following the advice to keep it as simple as possible and just use VMs.
Kubernetes makes many complicated things very easy and lets you stop worrying about your infra (provided you’re using a managed provider).
On our next project we used a PaaS offering because we had a bunch of credits for it and it was painful in comparison to what we had with Kubernetes. It was way more expensive (if we didn’t have credits), the deployments were slow, and we had less flexibility.
Kubernetes isn’t perfect, far from it. But for those who know it well it is usually a good option.
Hiring people with k8s experience in 2022 is not a difficult task, it's not an obscure technology anymore, and when your initial crop of devops decides to leave, the new guys coming in can scan your deployments and namespaces and pretty much hit the ground running within 48-72 hours. That's a big, important part of running a business. Being able to hire people with the right skill set, and being able to keep the lights on.
Bespoke systems built on top of EC2 require time, extra documentation and a steep learning curve, not to mention the fact that the bespoke system probably isn't getting security audits or being built to modern standards or best practices, tooling isn't being kept up to date. I can build a mechanical wristwatch in my garage, but if I had full access to the Rolex factory floor for free, I'd probably take that option.
I just came out of a project where Kubernetes performance issues involved wild guess and blind tuning until the so called experts actually found out why the cloud cluster was behaving strangely, including support from Cloud vendor.
And good luck making sense of all the YAML spaghetti available for bootstrapping the whole cluster from scratch.
72 hours? They better be 10x DevOps team.
The experience pushed me to decide "never again" and to plonk k3s or at least something like podman from the start on the server.
Haha, this is exactly what “hobby” means to a lot of people. Less judgmental: thinking and dreaming about the right tools in disproportion to the need is something people do a lot, presumably because it is a source of joy.
I feel that Kubernetes is exactly that for a large number of developers, operations people less so, they want to be part of a professional, world class, trendy environment/community. So they try leverage Kubernetes to elevate them to this Silicon Valley type tech company.
There's absolutely value to be had from Kubernetes, but I agree with the article, it's not for the majority. Except I think there is one valid reasoning: If you purely see Kubernetes as tooling for deployment, it's hard to present any good alternatives for deploying to VMs or physical hardware.
I see it the same type of folks as ones using FAANG tooling :) just so they can feel better about their job or more important when they work on business line CRUD.
I am working on business line CRUD and I like it with boring servers :)
I can add services, cronjobs and whatnot as I see fit in a standardized manner and don't worry about. A couple YML's get you started and you can scale as you see fit. Anything you would like to deploy can share 1-2 vCPU. The whole thing is also relatively portable across clouds or even on premise if you want it. Since everything is in containers, I have an escape hatch to just deploy on fly.io, Vercel or whatnot.
I get the criticism, that people over complicate projects that maybe don't even have traction yet. Bashing k8s is the wrong conclusion here IMHO. It feels a bit like people bashing React and SPAs just because what they have built so far never needed it. Just stick to your guns and stop evangelizing your way of doing things.
I've seen cases where we started off as simply as possible with no k8's. We built the initial product really quickly using a ton of managed services. Whilst it was great to get us going, once we hit "growth" things just didn't scale. (1) The cost of cloud was getting astronomical for us (and growing with each new deployment) and (2) it was totally inflexible (whether that be wanting to deploy in a different cloud, or extend the platform's featureset) because we didn't own the underlying service.
We ended up porting everything to k8's. That was a long & arduous process but it gave us total flexibility at significant cost savings. The benefits were great, but not everyone has access to the engineers/skillset needed to be successful with k8's.
That's why we built Plural.sh – it takes the hard work out of k8's deployments. I've seen people go from zero to a full production deployment of a datastack on k8's in just 2 weeks. It deploys in your cloud, and you own the underlying infra and conf so you have total control of it. And because we believe in being open, you can eject your stack out of plural if you don't like it and keep everything running.
Great post, and hope all is well with you!
The night and day difference is hard to explain when people haven't had to deal with all these issues on top of scaling, uptime, performance issues etc. I just can't recommend k8s enough
I think the argument from most people advocating against using k8s from the get go isn't so much that you'll never need it, but more that it's better to pay this cost later when you have a demonstrated problem that needs to be solved, even if it's a bit more expensive (and I would bet that the total time expenditure isn't really all that much more moving to k8s later, it's just in one chunk instead of spread over your history).
By deferring the cost of building more complex infrastructure:
- You can use those resources to advance your product (arguably some companies that have hit a wall with PaaS vendors may never have gotten to the point where they outgrew those vendors if they spent more of their time on infrastructure vs. company value) - You'll have a better idea of what you'll need and where your scaling hotspots are going to be if you've already encountered them. Building infrastructure that allows for flexibility in a few key areas is infinitely easier than building infrastructure that can scale in every way imaginable. - You can potentially take advantage of new technologies if you defer the decision to when you need it. It would be silly to assume k8s is the final word. If you wait a few years, things will have evolved and you gain the advantage of using those few years of advancement rather than cementing a choice early on.
Kubernetes is a layer of complexity that just isn't warranted for most companies. Hell even Amazon still sticks with VMs. Autoscaling and firecracker solve most things.
Its nice to have cluster (pod?) management as a first-class concept, but you get that with tagging of instances just as well for most use cases.
In short - i think the author has a point.
Meanwhile, I was chugging around with k8s because it was balance between "easy to deploy our workloads with minimal time spent" vs "increasing the cost by few pizzas will cause noticeable loss so pack those VMs tight".
It's about having a standard API for deployment artifacts.
The k8s manifests are trivial (if verbose) the complexity comes from running the underlying layer which when you are small you simply outsource to AWS or GCP.
There is some k8s know-how that is table stakes for a good experience, namely knowing what extra stuff you need on top of a base k8s cluster. i.e certmanager, external-dns.
Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes.
Before anyone starts with "but serverless!" you need to consider the spagetti you end up with if you go down that route, API Gateway, 100x Lambdas and thousands upon thousands of lines of Terraform boilterplate does not a happy infra team make.
My experience is even mediocre engineers are capable of understanding k8s concepts with minimal assistance assuming things are either sufficiently standard (i.e google-able) or well documented with any company specific practices.
K8s == hard to run, k8s != hard to use.
I did get everything running but also saw so many settings and features I had to take on faith as being handled without understanding it. I just chose not to bother further because this seems like a minefield of problems in the future as we on boarded other engineers. The number of times I’ve seen production deployments go down due to k8s misconfiguration on other teams has only validated my concerns.
You aren't meant to fully understand it in a week, anymore than you are expected to fully understand all of sysadmin in a week.
Just because you don't know what every single directive in nginx config does doesn't mean you can't use it effectively and learn what they mean when the time comes.
k8s isn't much different. You don't need to know what a liveliness probe is the first time you use it, you can learn about it as you go (most likely when you run into a badly behaved program that needs forced restarts when it hangs).
Ofcourse if you are running it yourself that is entirely different, you really do need to know how it works to do that but you should be using hosted k8s unless you have hardcore systems folk and actually need that.
The solution I went with, given my limited knowledge of some of these things, was to use elastic beanstalk. You write your flask application, upload the zip and that’s it pretty much. You get a UI to do all the configurations and for the most part nothing there is hard to decipher or google. The only hiccup might be when you’re trying to connect it to RDS and to the external internet but even that is straightforward as long as you follow the right instructions. We run apps that power entire SaaS organizations and this system seems to be more than sufficient. Why complicate further? We have other fish to fry anyway.
On the other hand, with docker swarm, I built a cluster in less than a day. So yeah setting up k8 isn't as trivial as you are making it out to be.
For comparison, you can just as easily expose/hide your VM as your managed k8s cluster.
I do not agree with this. I have tried to start Kubernetes and it's far more confusing to set up than just setting up something on a standalone box, mainly because it looks like the set of knowledge I need to set up anything on Kubernetes is a superset of what I need to set up the same thing in bare Linux.
Lets see to get something even half-reasonable on a bare box you need the minimum:
process monitor: For this you can use systemd, supervisord, etc.
logging: rsyslog or similar.
http reverse proxy: nginx or haproxy
deployment mechansim: probably scp if you are going this ghetto but git pull + build and/or pull from s3 are all common at this level plus a bunch of bash to properly restart things.
backups: depends on persistence, litestream for sqllite these days, pgdump/whatever for RDBMS, wal shipping if you are running the DB on your own boxes.
access control: some mechanism to manage either multiple users + homes or shared user account with shared authorized_keys.
security: lock down sshd, reverse proxy, etc. stay on top of patching.
So are bare minimum there is a ton of concepts you need to know, you are essentially regressing back into the days of hardcore sysadmins... just without all the hardcore sysadmins to do the job correctly (because they are now all busy running k8s setups).
Meanwhile to deploy a similarly simply app to k8s (assuming you are buying managed k8s) you need to know the following things:
container + registry: i.e Dockerfile + docker build + docker push gcr.io/mything.
deployment: Describes how to run your docker image, what args, how much resources, does it need volumes, etc. You can go ghetto and not use service accounts/workload-identity and still be in better shape than bare boxes.
service: how your service exposes itself, ports etc.
ingress: how to route traffic from internet into your service, usually HTTP host, paths etc.
Generally this is equivalent but with way less moving parts.
There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you. You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc.
K8s reputation as being hard to run is deserved.
K8s reputation as being hard to use is grossly undeserved.
And by the looks of it, ingress doesn't seem trivial, not only you need to understand how NGINX works for reverse proxy but also understand how Kubernetes interacts with NGINX. You also ignored backups for Kubernetes, if you're just doing full disk backups you can do that with regular boxes, specially if they're just VMs.
> There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you.
No SSH server might be an advantage for you, but for me it means it's far more difficult to know what's happening when Kubernetes doesn't do what I want it to do.
> You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc.
But I still need to know how to set up all applications I use, which is the most important part. And understanding systemd service files, sshd_config, nginx.conf and rsync is far, far easier than understanding Kubernetes. Kubernetes manages those things so you actually need to understand both the underlying concepts and also the abstractions Kubernetes is making.
You are also comparing self-hosted with managed, and then only mentioning the disadvantages of self-hosted. Of course with a managed setup you don't need to worry about SSH, but you have to worry about actually dealing with the manager service. If you were to set up self-hosted Kubernetes you'd still need to learn about SSH, or about account management. That's not something about Kubernetes but about self-hosted or managed.
Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.
You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).
The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.
Given that you haven't managed to identify any of k8s real downsides here they are:
It's a bunch of yaml. Yeah I don't like that either but there are tools like Tanka that make that suck less.
It's complex under the hood. When your process runs its in a network, fs and pid namespaces and you have cgroup managing resources. There is probably some sort of overlay network or some other means of granting each container an IP and making that routable (BGP, OSPF, etc). Interaction between ingress, service and pods is through an indirection layer called endpoints that most people don't even realize exists. Scheduling is complex, there are tunables for affinity, anti-affinity, it also needs to interact with cluster autoscaling. Autoscaling itself is a complex topic where things like hot capacity etc aren't exactly sorted out yet. QoS, i.e pod priority isn't well understood by most. When will a pod be pre-empted? What is the difference between eviction/pre-emption etc? Most people won't be able to tell you. This means when things go wrong there are a lot of things that could be wrong, thankfully if you are paying someone else for your cluster a) it's their problem b) it's probably happening to -all- of their customers so they have a lot of incentive to fix it for you.
Multi-cluster is a mess, no reasonable federation options in sight. Overlay networking makes managing network architecture when involving multiple clusters more difficult (or external services outside of said cluster).
Ecosystem has some poor quality solutions. Namely helm, kustomize, Pulumi, etc. Hopefully these will die out some day and make way for better solutions.
Yet for all of these downsides it's still clearly a ton better than managing your own boxes manually. Especially because you are -exceedingly- unlikely to encounter any of these problems above at any scale where self-hosting would have been tennable.
I think the best way to think about k8s is it's the modern distributed kernel. Just like Linux you aren't expected to understand every layer of it, merely the interface (i.e resource API for k8s, syscalls/ioctl/dev/proc/sysfs for Linux). The fact everyone is using it is what grants it the stability necessary to obviate the need for that internal knowledge.
Dockerfiles aren't trivial. I've helped migrate some services to docker and it's not "just put it in Docker and that's it". Specially because most applications aren't perfectly isolated.
> Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.
Literal thousands of teams also use Nginx and still doesn't mean that there aren't configuration errors, issues and other things that are to debug. Not to mention that a lot of applications can run without requiring a reverse proxy.
> You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).
And that's cool if you need that, but "ssh user@machine" looks far easier.
> The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.
Look, the other day I had to set up a simple machine to monitor some hosts in a network. I've Ansible to automate it, but ultimately it boils down to "sudo apt install grafana prometheus; scp provisioning-dashboards; scp prometheus-host-config; scp unattended-upgrades-config" plus some hardening configs. I don't need redundancy, uptime is good enough with that. If Prometheus can't connect to the hosts, I can test from the machine, capture traffic or whatever and know that there isn't anything in the middle. If Grafana doesn't respond I don't need to worry whether the ingress controller is working well or if I missed some configuration.
Could I run that with Kubernetes? Well, managed k8s is already out of the window because the service is internal to an enterprise network. And self-hosted kubernetes? I still need to do the same base linux box configuration, plus setting up k8s, plus setting up the infra and networking with k8s plus the tools I actually want to use.
And that's the point. I am not trying to identify any k8s downsides, what I said that setting up k8s looks far more confusing than setting up a standard Linux box, and by the things you say I am even more convinced than before than it is indeed. I still need to understand and run the programs I want (which is most of the cost of deployment), but I also need to understand Kubernetes and, if I am using a managed platform, I need to understand how the managed platform works, configure it, link it with my system... In other words, not only do I need the general knowledge for application deployment but also the specific implementation in Kubernetes.
What if you dont need multiple servers? Well go the single node approach and have a flexible, true and tested way to spin up containers, which can and should be able to crash whenever they have to.
Using Containers is pre mature optimization too? maybe I should get my typewriter.
More like grab a damn pen and scrawl out what you actually need. It's flexible and fast, which in many cases suits the situation best.
You seem to have missed the alternative "get a computer" in between there.
I have just provisioned another bare metal k8s cluster, taking software of dedicated servers and.... well if I dont have to, im not running bare metal anymore in 2022.
If you want to just "get a computer in there" have a look at harvester, its the best of the two worlds
> Solve problems as they arise, not in advance.
while this does make sense for supporting varying requirements the lean way, it fails to address the increased costs that rearchitecting a solution mid-lifecycle incurs.
> Do more with less.
goddamn slogan driven blogging. what is the proposed solution? doesn't say. are we supposed to log in every prod/test/dev machine and maintain the deps by hand with yum/apt? write our own chef/puppet scripts? how is that better than docker images running kubernetes? The comparison between solutions is the interesting part.
op never say. guess "works on his pc" is enough for him, we can only assume he envision a battery of mac laptops serving pages from a basement, with devs cautiously tipotoing around network cables to deliver patches via external usb drives
There are a lot of valid reasons why a start-up might prefer Kubernetes to other solutions.
A simple and good enough reason is the need for variable compute:
You might need more computing sporadically (say for example you're doing software in the sports industry; your demand changes with the number and size of events, as well as on weekends).
Another reason might be a start up that allows their customers to execute some type of arbitrary code (eg a no-code product). This can vary from customer to customer, and it can also vary within the customers use cases.
Imagine having to manage all these storage, networking and compute resources manually... Or going full circle and managing it with a bunch of puppet/ansible/shell scripts. Now you slap some automated scale triggers on top that fires off these scripts.
Congratulations! We've built something that looks like Kubernetes; if we squint. There's some smoke coming out of it. Documentation? Lol, we don't have time for that. we are a company that gets shit done, we don't toy around with Shcubernetes and documentation! Error handling/monitoring/reporting? Eh, just read the code if something fails. Need to add cert issuance? Yeah let's implement our own ACME integration. Network ingress? Let's just have a 500line haproxy infront of an 2000line nginx config; no big deal. DNS? just append to named; who cares?
The name for the "we get shit done" crowd should probably be "we don't give a shit about anything and have others solve problems we created because of our lack of thinking and foresight", but it doesn't sound quite as memorable.
It's just people who are comfortable cutting corners at other people's expenses. When they have to own the shit they made, they start blaming others and leave the company.
Sorry for the rant.
During development of the scripts, the developers/data scientists wrote and tested everything outside of the cluster since they were simple scripts. They had no problem grinding through a day's worth of data in their testing. But going into prod, had to shove it into the cluster. So now we had to maintain the scripts AND the fscking cluster.
Why?
"What if the data volume increases or we need to run lots of analytics at once?"
"You'll still be dominated by I/O overhead and your expected rate of growth in data volume is <3% per year. You can just move to faster disks to more than keep up. Also there's some indexing techniques that would help..."
Nope, had to have the cluster. So we had the cluster. At the expense of 10x the hardware, another rack of equipment, a networking guy, and a dedicated cluster admin (with associated service contracts from a support vendor). It literally all ran fine on a single system with lots of RAM and SSDs -- which we proved by replicating all of the tasks the cluster was doing.
Argh...
…and if you don’t understand how to use and deploy kubernetes just what in the fresh hell are you doing? Stick with technology you know and then move to kubernetes later if or when you need it.
We’re a relatively small shop, and we’re using k8s. All of our senior stuff know and understand it. The pipeline is fully automated. If we prematurely optimizing anything it’s developer output over saving money.
In a quest to get closer to the metal, Kubernetes keeps you far away, which is the opposite of what any production service should want.
What is the purpose of adding layers when uni-kernels and eco-kernels give you better isolation and better performance?
Your cloud provider already runs your virtual machines OS on a hardware hypervisor. Then running Kubernetes on top of the OS and then a zillion containers on it is a recipe for poor performance.
What is the logic here that I am clearly missing?
Cloud providers native Kubernetes stacks don’t improve performance or pricing, compared to their cloud compute instances virtual machines and a dedicated virtual machine per would-be-container that thankfully doesn’t need to now share processing resources with others.
What gives? Why on earth would anyone run production processes in Kubernetes?
*i have used kubernetes and understand how to use it as intended. I feel like I am missing the motivation for it’s current widespread usage in web apps
The minimum virtual machine size for a Windows server that is at all useful for anything is 4 GB of memory. Okay, okay, so you can technically boot it up on 2 GB and some roles will work fine, this will last only until some dingbat remotes to it with RDP with a 4K monitor and it starts swapping to disk.
Even if you use Server Core and block port 3389, it still needs a ton of memory just to start.
Running in a container it uses a few hundred megabytes.
Similarly, the minimum system disk size you can get away with is 32 GB if it is a discardable / ephemeral instance. You need 64 GB minimum if you ever intend to run Windows Update on it.
With containers, the unique parts of the image might be just a few hundred megabytes, even for complex apps.
My experience is with Windows, but from what I hear Linux VMs vs Linux containers have vaguely similar ratios.
So with containers, a single host can run dozens of applications, all sharing the same base disk, and all sharing the same OS kernel. The savings can be staggering.
At $dayjob, the admins are very much stuck in the dedicated VMs for every role mentality, and they're burning through enormous piles of taxpayer money to run them at literally 0.1% load.
Having said that, Kubernetes has its own problems. As you said, layering it on top of cloud VMs is a bit silly, and can easily result in the container running in a nested hypervisor at molasses speeds. Similarly, every single typical process changes dramatically: Deployment, updates, monitoring, auditing, etc...
Combine the above with the incompatible underlying cloud layer and things get really messy really quickly.
In my experience 90% of the world just isn't ready for the learning curve. Windows as an operating system isn't ready, certainly. Microsoft Azure isn't really ready either. Their AKS managed offering is still undergoing massive churn and seems to have more preview features than stable features. Even in the Linux world I hear more horror stories than success stories. It seems that everyone who says they love Kubernetes is using it on like... one machine. Come back and tell me how you feel after troubleshooting a failed upgrade on a cluster managing $100M of finance transactions.
What I would like to see is "native Kubernetes clouds" where the hosts are bare metal and there is no impedance mismatch between K8s and the cloud provider APIs because K8s is the API. Instead of the Azure portal or the AWS console you literally log into a Kubernetes console.
IMHO that would allow a true commoditisation of the public cloud and start to erode the near-duopoly of AWS and Azure.
They run on hardware.
Ultimately, there needs to be a singular scheduling system running on hardware and a singular HAL-like driver layer, and exo or iso kernels deliver just that, vs lxe containers provided by os services.
The sizes are also quite impressive.
Example of thing that is pretty straightforward in K8s and much less straightforward outside of it.
1. For compliance reasons, you need to make sure that your underlying OS is patched with security updates.
2. This means you need to reboot the OS every X time.
3. You want to do this without downtime.
4. You have a replicated architecture, so you know you have at least two copies of each service you run (and each can handle the required traffic).
In K8s, this operation can be as simple as:
1. Mark your old nodes as unschedulable.
2. Drain your old nodes (which will grab new nodes from your cloud provider, installing the updated machine image).
3. Delete your old nodes.
The exact steps will differ based on your use case, but that's basically it.
Steps you didn't need to think about here:
1. If I'm about to take a node down, do I have enough redundancy to handle its loss? K8s understands the health of your software, and you have a policy configured so it understands if taking down a container will cause an outage (and avoid this). Note: third party tech will be naturally compatible - if you use Elastic's cloud-on-k8s operator to run ES, it'll appropriately migrate from host to host too, without downtime. Likewise, the same script will run on AWS, Azure, GCP.
2. How fast can I run this? If building this logic yourself, you'll probably run the upgrade one node at a time so as to not have to think about the different services you run. But if it takes 15 minutes to run a full upgrade, you can now only upgrade 100 hosts each day. K8s will run whatever it can, as soon as it can without you having to think about it.
3. What happens if concurrent operations need to be run (e.g. scale-up, scale-down)? With K8s, this is a perfectly reasonable thing to do.
4. Does this need to be monitored? This is a fairly standard K8s workflow, with most components identical to standard scale-up/scale-down operations. Most components will be exercised all the time.
Generally I've been impressed by how straightforward it's been to remove the edge cases, to make complex tech fit well with other complex tech.
A while back we upgraded between two CentOS versions. In such a case it's recommended to reinstall the OS - there's not a clear upgrade path. In K8s, this would have been the same set of steps as the above. In many orgs, this would be a far more manual process.
- kube-state-metrics
- Coredns
- Prometheus
- Alert manager
- Node Exporter
- Spot instance termination handler
- Karpenter/cluster-autoscaler
- Event Exporter
- Kubernetes Dashboard
- Ingress
- Cert-Manager
- Custom CNI?
- Istio (not always)
- ArgoCD (not always)
But for startups and/or simpler setups, ECS/Lambda is so much less work, while usually being powerful enough.
Which one(s) run your API/server, your database, etc is basically arbitrary. You’re free to let the K8s scheduler do as it see fits, or you can get increasing degrees of control by adjusting things like node and pod affinity/anti-affinity, etc.
* You describe what type of state you need (Deployment = one or more Pods running your container, Service = expose the deployment) * Kubernetes runs a control loop that reconciles the current cluster's state with your desired state
Nodes are the virtual machines on which everything runs, you don't need to interact with them unless you have special requirements (e.g. GPU instances). In that case, you annotate GPU instances with some type of label (like `has-gpu=true`), then in your deployment you add a node affinity saying it needs to run on nodes that have that label. Kubernetes will schedule it for you if there's any node matching.
K8s signals the team values infra and sees it as an important part of delivering their product.
It's an investment like any other and for some teams it makes sense to set infra on a solid path sooner rather than later.
You can move fast on things like serverless but the limitations of those systems can force architecture decisions that are sub-optimal whilst control of your own infrastructure can allow broader interpretation of problems and enablement of more wholistic solutions.
i.e it opens up building things "the right way" sooner in many cases and that can save a ton of time and frustration.
Also in the case where it's obvious you will outgrow serverless it also saves on a sometimes painful migration effort.
But isn't "infra[structure]" what you run your product on, and not part of the product itself? If your "product" is so big it needs stuff like that built into it, then it feels to me that by defintion that stuff isn't "infra".
All in all, your sentence feels like "The Excel-Dell Humongo-XPS bundle signals that Microsoft values infra and sees it as an important part of delivering their product" to me. No, that only signals that A) Dell and Microsoft are using the dominant marketing position of Excel to milk money from naive customers, and B) If Excel can't run on more modest hardware, it's an unnecessarily bloated and inefficient product; look around for another spreadsheet.
First of all k8s isn't bloated, it's actually surprisingly compact if you think about what you are getting.
Second it doesn't imply the product -needs- it. Anything you can run on k8s can be run on bare boxes.
Just doing so probably means you either lack the scale to need k8s (i.e my spreadsheet doesn't have many users) or you lack the competence to understand you have reached the scale that it's required (i.e your spreadsheet is run by n00bs). So it's much more of a positive than negative signal if you are seeking a vendor. I think the product relationship thing is a poor analogy from that side though, it -mostly- matters for people that want to work there on/in/around the product itself.
As a primarily infra guy I have a preference for k8s because it allows me to have some reasonable set of services I can rely on being there if a place is "using k8s". If not I just have to hope either the existing infra team is competent enough to have things like service discovery, secret management, process monitoring, etc under control or that they are amenable to efforts to get it under control.
If I am approaching it from a backend developer perspective where most of that isn't my problem then it gives me some confidence the infra team has some idea what they are doing and that I won't need to deal with some sort of bespoke deployment system that has endless quirks and company-specific bugs that I shouldn't need to concern myself with.
Generally k8s just eliminates uncertainties by increasing standardisation so I don't need to watch people solve the same problem poorly over and over again for no reason at all.
[1] https://twitter.com/thockin/status/1539987108521054208
[2] https://twitter.com/the_thagomizer/status/153975704907180032...
The author mentions other things he considers red flags such as using a different language for backend and frontend development with no additional context.
Is the author talking about a startup in the context of one person who just knows JavaScript working on their own building a prototype? Is he talking about a series B company with 500k MAUs?
Some additional context would improve the article a lot. I think the author should have had a few people read over the article and given feedback before publication.
In my experience, a well-functioning new team will include performance in their selection process. Doing so allows a project to defer optimizations such as higher-complexity deployment (e.g., cluster orchestration, highly sophisticated caching, and so on) for much longer than those who build on low-performance platforms and frameworks.
Choosing a low-performance platform and framework sets an artificially low performance ceiling. Developers dealing with a low performance ceiling will either instinctively (through received or learned patterns) or reactively (through user complaints, issue reports, troubleshooting, and tuning) deal with performance challenges they should not need to deal with so early in a product's lifespan.
Ironically, some would say that including performance in your technology selection criteria is "premature optimization," but I argue the opposite: considering performance a feature early allows you to defer costly optimizations (such as the matter at hand, Kubernetes deployment), perhaps even indefinitely.
BTW Google open-sourced Kubernetes not for charity (like all businesses they also want to make money), they knew they had lost the cloud war with Amazon/Azure gulping up almost 80-90% market share. So they wanted a platform on which they can lure users back to Google Cloud when they start providing kick-ass services (to avoid the much famed vendor lock-in). And since docker was able to solve the dependency management in a reasonable way (not in a very esoteric way like nix) and were dipping their toes into distributed orchestration, they took it as a open fundamental unit to solve orchestration of distributed systems.
But yes Google succeeded in convincing dev/ops to use k8s by frightening them with vendor lock-in. But the most ironic thing that I see about k8s is that all these HPA, Restart on crash all those things are being touted as great new features. These features have existed in Erlang for decades (supervisors and spawning actors). I'm not sure why Google did not try to leverage the Erlang ecosystem - it might have been much faster to market (may be NIH).
Unless you want to embed another language in-process of some k8s components, but the need to do that is disappearing as things are moved out of process.
This is incorrect. It's a common mistake to pre-optimise for a future that may never come. I call it "What-If Engineering".
You should always just build a monolith and scale it out as and when you understand what it is that needs scaling. Choosing the right languages and frameworks is where you need to put your effort. Not K8s.
BLUF always have both a current state architecture as well as a future state architecture in mind and a plan for how to gradually evolve from one to the other over time.
Teams should be able to benefit from Kubernetes and the surrounding cloud-native ecosystem without directly consuming or modifying it. You don’t need to reinvent the wheel, and that’s what you could say a lot of platform teams are doing today.
That’s what we’re working on at https://northflank.com a next-generation deployment platform using Kubernetes either in a secure multi-tenant environment, traditional PaaS or deployed directly into your GKE, EKS and AKS clusters. (disclaimer: Northflank co-founder)
To me, kubernetes is the kind of giant, complicated, hard to understand software that is ripe for HN disdain.
Same with k8s. People conflated running k8s from scratch, with administrating. The former is very hard, you should buy it from a cloud provider. The latter is IMO not much harder than VMs at small scale and much simpler at you grow bigger.
We don't want to run k8s from scratch? Let's buy the managed version in the cloud?
It sounds like an argument for containers.
I mean, at that point why not use a simpler abstraction, like a VM. Or just use functions.
Few other Kube goodies:
1. Plenty of apps are available as helm charts. It's like pre-build docker image, but even better and easier to install and manage (!).
2. There are some marvelous operator like Postgres operators which will do the hard work of managing Postgres database for you, including replication, backups, upgrading. On the down side is the fact that when it broke, you need an expertise... So may be managed database still is better for small scale.
I'm coming to kube as someone who spend some time dealing with docker-compose servers. And I feel like kube will be better.
For new projects, even on small scale, it's likely that I'll choose kube.
The only question is which is which.
Also something like karpenter.sh is much nicer than ec2 autoscaling group.
Yes, you can build an amazon machine image for ec2, update your cluster with the new image, without scaling from zero.
> As others have already pointed out, it is also important for us to remain independent from Google Cloud / AWS.
This seems like a good reason to use k8s, although as the original article touches on, it seems like switch cloud vendor rarely ever happens.
A lot of data tools are starting to target Kubernetes directly as a runtime so using them with GKE/EKS is a bit simpler as it’s officially supported, allows to run locally and on the cloud with no vendor lock in.
ECS in a scaling group works well if your app is stateless but as soon as you scale workers dynamically, do service discovery, orchestration, you end up building some of the features Kubernetes provides.
This is a superpower that you want to have individually as an engineer but you need organisationally as a culture too. If you have that power then you when you need to write some similar code you can copy and paste, because you can trust that when you have a third use you will actually refactor properly. It means that you can get the service up and running quickly on a box you've got, because you can trust that if the service actually gets used your org will allow you the time to put it on the cloud properly.
It's the most important part of agile - make the right decision for what you've got now, and then make a different one later.
it beats the pants off the the random rubricks of ansible and puppet projects I've run into the past anyway
I write code for human, to serve real benefit/profit from business standpoint. I build value and trust based on what i delivered, not from what i "want to learn".
Currently, i go all in for unidirectional architecture. Think redux all the way, not just for frontend.
The point here is, it's the architecture that's drive business profit, not Kubernetes. Kubernetes is not your architecture.
on-prem: procuring hardware, installing hardware, installing the host OS/deploy the host virtual machines, networking, deploying K8s, CNI, CSI, etc
if cloud: increasing quotas, verifying that CPU sizes are available for more nodes or new node pools, performing virtual networking and subnetting for the K8s clusters, etc.
both: security everything from RBAC to networking to image validation, upgrades - you made sure to have a scalable app right? if not fingers crossed on availability.
Would the chore exist outside of Kubernetes, of course there would be a chore. I'm not sure so great though. While deploying to Kubernetes is simple. Setting up an on-prem platform is not what-so-ever and if you with public cloud you now have to be aware of all the caveats and gotchas that aren't handcuffs with on-prem.
Nothing I've ever done has taken off at all so automating DevOps would have been a total waste of time. Then again the projects themselves have been a complete waste of time.
It's very easy to say something could be improved, but much harder to propose something to replace it that doesn't have its own shortcomings.
It's a pity because this article actually makes a good point, that we should think about what the actual user/business goal is before making technical decisions, and be careful lest we spend all our time on technology. I've heard this idea described as "innovation tokens". It's disappointing that the article chose to wrap this idea in clickbait, but I guess we wouldn't be talking about it otherwise!
Disclaimer: I have no end of personal biases in favor of kubernetes!
The article explicitly advocates for high-level PaaS tools as an alternative to Kubernetes.
I quite liked the actual message of the article (which I take as "be mindful of your technology choices") - but I think that it is really devalued by the less coherent but more click-baity arguments (e.g. "javascript is the only rational language choice")
Edit: Actually, re-reading the article yet again, I think I can see your reading (though I wish you had written the article, as your writing is much clearer). As I understand it, the article says you should use a PaaS-style solution until you outgrow it, when you should move to something from the cloud providers. For me, that would be CloudRun -> GKE-Autopilot -> GKE (skipping CloudRun if you don't want to learn two systems), but there may be some google-bias there!
"Another time our Data Science team told us that they needed an orchestration tool for their data pipelines. I steered the selection process towards Argo Workflows (which runs inside Kubernetes) instead of Prefect (a SaaS offering), which they already used for a PoC. There were all sorts of justifications in my head for this decision. Unfortunately, they were all based on premature optimisations. In the end, our team needed to build a new set of Terraform and Helm charts to automate the deployment of Argo Workflows and integrate it into our SSO etc. I regret this decision. I think we lost weeks or even months shipping something to end-users because of this decision. Premature optimisation!"
I frequently need to re-provision new servers and close existing ones.
I've been doing all this with the help of docker-machine (now deprecated), and a collection of handy bash scripts. While this has worked, it's become more and more fragile and I need something better.
For years I've resisted Kubernetes because I keep hearing "premature optimisation", but at this point I'm not sure I have any other options. So before I dive head-long into Kuburnetes by what feels like necessity, how else could I possibly: - Push button provisioning of new hosts across a number of cloud providers - Monitoring so I can restart services under certain conditions. - Notification of failures etc. - Automatic / continuous deployment across all servers and regions.
But argocd hooked up to each provider along with provider specific overrides should work with enough tweaking.
It will be simpler, more straightforward and most likely use less computing resources. It’s also a decades old technique at this point that is extremely battle hardened.
You don’t say it outright but you imply that you are doing this as a single operator. That removes giant swathes of the advantages K8s brings to the table. Further your implication that there are multiple edge clusters is a case that is fairly new to K8s and the best practice would be to run n K8s clusters.
If you are looking for a more managed/turn key option I’m a huge fly.io fan boy.
> Solve problems as they arise, not in advance.
It strikes me that this particular philosophy, that I associate for no other reason than a gut feeling with US-American culture, is one of the biggest reasons for our shitty software development quality. Every successful team that I have seen so far had solved the worst problems in advance and thanks to this investment ended up with a much higher capacity for times of crisis. The assumption that problems can be solved (really, solved, not somehow worked around until things will explode in your face Tuesday next week) when they show up in practice looks like fake optimisim.
Of course it takes good judgement to distinguish actual future problems from mere inconveniences and phantom issues. But that's a skill that people should try to cultivate not bullshitting their way out of responsible management with terms like "Pareto principle" and "agile".
I honestly couldn't give a toss if some guy thinks using some < language / tech / library > is <blah>, it's a technology, a tool, try it, use it, if it works for you good, if not use something else. Move on.
The only time it's a allowed is if it's bashing JIRA :)
To set up complete envs for dev, test and prod with GIT, full CI/CD and everything you need like databases and storage is less than 20 hours work in a modern cloud. This is something that comes with low maintenance as well. If you think containerization and Kubernetes is a better option, you are basically incompetent when it comes startups.
Much, much less work to then move the deployments elsewhere, including building and teardown of integration testing environments.
Are you kidding me that's my favorite part!
The basic idea is that of "be like this spec" via a yaml/json file - a controller is continuously monitoring an object and making adjustments until spec matches actual.
I've seen many startups have one person who sets up and maintains their k8s cluster. If you know what you're doing, it has pretty solid ROI.
I think the issue with people that use k8s first is that they go for super trendy architectures before really nailing down the business logic first. This leads to technical debt and a lot of confusion around the tool.
As far as the advice goes. No not really, lots of people are familiar enough with Kubernetes that it makes it the fastest choice possible for them. Engineers like to pretend that tech choices are oh so important and can be discussed in a vacuum, but really most tech choices are made once you know the people who are working on the project.
Pick stuff that is familiar to you unless you have a good reason to do otherwise.
It no longer makes sense to spend so much resources on orchestration or distributing across multiple hardware.
Not when you can simply write code and have it running globally distributed on edge. You would be doing yourself a disservice to not follow this new paradigm shift.
I am not a Kubernetes advocate but what else is there that handles all of the issues faced when deploying containers? Such as scaling, deployment, configuration etc?
There are alternatives such as Hashicorps Nomad, but I don't see how this is any better/worse that K8s.
It's basically a higher level of managed Kubernetes ( EKS or GKE ). You give your containers, and they take care of running it.
Tim Hockin (one of kubernetes creator) support the idea to use something as much managed and automatic as possible : https://twitter.com/thockin/status/1539987108521054208
I need to run some services on premises and have set up a self hosted Kubernetes instance on a physical server in a rack.
It could be overkill and maybe I could use something like Docker Swarm. Apart from this I am unsure what I can use that isn't K8s to orchestrate my containers on site.
This includes small / medium size organizations (like when you are not Google or Amazon), startups, one-man projects, and "hobby" projects. " https://dockerswarm.rocks/
The other day I spun up a new eks cluster, created some Iam roles, installed the cert-manager, external dns, and aws alb ingress controller.
Then I pushed a deployment, a service, and an ingress and BOOM. Log aggregation, automatic dns, secret handling, ...I can go on.
It is simpler than normal aws services in my experience.
“More than one language for your application. For example, a backend in Golang, Ruby, PHP etc. and a Frontend Web App in React, Vue etc.”
When would your front end and backend ever be in the same language? I guess JavaScript would be your only option?
You can use elastic beanstalk, and RDS. You get scaling, reliability and backups. And it’s not complicated and you can set up easy CICD with just GitHub actions.
If you don’t need scaling, just use lightsail and litestream.
There are many many options to do things simply.
What is the shortest route to a sustainable, defensible revenue stream?
In the 80's IBM built a super quick computer out of pieces you could buy at Radio Shack. The ultimate solution! Why incur all that upfront manufacturing cost when you can assemble it out of existing parts? They learned the answer really quickly. The lower barriers of entry meant anyone could recreate the product. And so a million clones popped up using the same parts. For a decade Apple ate their lunch by investing in a proprietary platform(which a lot people said was madness using the exact reasoning on display in this article)
SaaS is great but you have to have some sort of moat for whatever you are building. Sometimes that moat can be as simple the idiosyncrasies of your over-engineered, prematurely optimized product. I had a hand in the development of one of the major VPS players during the 2010's. There were things we could do just because of the weird decisions people had made in the past. That translated directly to competitive advantage.
Let’s see just how long it takes for participants using their preferred tools to build and deploy a Todo app with user accounts!
And then we see how long it takes to go from the initial application to one that supports 1000 requests a second.
Now we are restarting lean (but wiser for the experience). 10x less people and technology accomplishes more and faster.
Most companies with that kind of problem DO NOT need Kubernetes because they don't have problems for which Kubernetes is the cheapest solution.
Most companies do need focus on keeping things simple, minimise amount of stuff to make it possible to hire a development team that has a shot at understanding what is happening.
For example, we made decision to restrict ourselves to only one cloud provider (AWS), to only one backend language, one frontend language, etc.
Our prod environment is simple, too. No effing microservices. One application on one application server easily serves the needs of a company of over 200 employees and our clients.
Simplifying the environment made wonders to productivity in many different ways, the most important being that the discussion now focuses on what the company needs and the functionality that would solve the problem rather than on technical trivia.
All this makes possible following things:
* having one development team where everybody can contribute to everything (no need to Zoom with local K8s guru to accomplish anything or spend time in endless meetings with "frontend people".)
* hiring a relatively simple developer profile. This does not mean hiring bad developers. But it means I am looking for people who know X and Y very well rather than X, Y, Z, A, B, C each a little bit... It is stupid to assume a person will know well every one of 50 different technologies listed on their LinkedIn profile -- most likely they only really know couple of things and everything else only superficially.
* hiring from talent pool that has a lot of experience but not necessarily in the stuff that other companies require. Other companies may hive overlooked them because they don't know a bunch of new tech, but we don't care. We only care that they are great people that get shit done, fit our company, and know how to design and implement functionality in their core programming language.
* engineers being able to accomplish their individual tasks from start to end, right after joining the company. No need for a long learning period.
* company needs main center of attention rather than spending large part of the effort on technical trivia.
My take? If you need to add something, first ask yourself what are the costs and better understand what you get in return for that cost.
More than one container then cloud hosted kubernetes makes sense. It's a transferable skill a kind of learn once apply to any cloud technology.
> Kubernetes is a red flag signalling premature optimisation
And even that presupposes that it's an optimisation at all in the first place.
a day to three days for an experienced system engineer to package the components,
and a week to three weeks to test and package the configuration (by automating in %pre and %post of the RPM's).
That's far less than one year, and far less than it takes to set up Kubernetes, Terraform, Docker and a complete CI / CD pipeline.
My ultimate MVP starter pack is probably a big fat VM running a single node K3s instance and a cloud provider managed database. Software wise nginx-ingress / cert-manager / argo-cd is a pretty good default set.
let's say devs push to gitlab, which starts a pipeline, tests run, container images get built, then in the last step in the gitops repo a version bump happens and then argocd will pick it up automatically?
(and if one wants to be able to rollback then that's the same workflow just instead of version "bump" the pipeline tags images with the commit hash and at the last step sets the version to the right tag?)
are there any gotchas, best practices? :)
thanks!
For rollbacks, you have a button in argo that lists previous deployments and you can just choose which previous deployment you want to go back to. It works pretty well those (thankfully few) times i have had to rollback any deployments.
I think when you look at the entire argo suite you start seeing something that could really disrupt the way we use products like gitlab, particularly for startups.
But that's going past an MVP/startup toolset.
You're also right, rolling back just includes changing the hash back to whatever you had, or to a new hash which was a result of a git revert or whatever. Also the good thing is that if that newly deployed service is very broken (that it doesn't pass the k8s health check), ArgoCD will hold on to the old ReplicaSet and will not let your service die because of it.
What's also nice about ArgoCD is that you can play a bit with some service/application in a branch. Say you have some live service, and you want to adjust some configuration of it. Usually third party services have a lot of options to set. The problem is that you're not 100% sure how to get what you want, so doing a pull request for every small change can be very slow and exhausting. To work around that, you can point your ArgoCD's application chart to a chart which is in a branch, test/dev/fiddle with just pushing to that remote branch, and when you're satisfied, you merge your branch, and at the same time point your ArgoCD application manifest to point to the master/HEAD for that chart. In effect, at this step, only your Git repo will be updated, your service already has all the changes so ArgoCD will do nothing. That way you can iterate faster, and undo whatever regression you've introduced just by pointing ArgoCD to watch the master, not your branch (or you can just reset your branch to be indentical to master).
I will say I think what the fly.io guys are doing with VMs on the edge and in-process distributed databases is pretty exciting.
Plan9 is the ultimate distributed OS. Running programs on different computers is as easy as mounting a remote /dev/cpu file to your process' namespace. Virtual desktop is as easy as mounting /dev/draw. And so on. Many such complicated things that require ad-hoc solutions on Linux are solved on a fundamental level in Plan9.
I wouldn't say that it's going full steam ahead, since its use in the modern world is fairly limited, but the ideas were there, and they have influenced the development of Linux.
I think the more believable scenario is "hey let's see what we can do with the existing Linux systems and sockets and stuff" and then importing the network into the model because it was the simplest solution.
Then again, whether or not should applications be network-transparent-by-default is a discussion for itself. I believe they should. Every program (that does not drive hardware directly) is just a piece of code that glues various OS APIs together. In the case of Plan9, the APIs are the filesystem, and the filesystem can be modified with 9P protocol. If we can use the same local program to perform an operation on a remote machine without any modification, why the hell shouldn't we.
Which operator did you go with out of curiousity?
It is not a bad choice if you want to do the ops yourself for some reason. If you don't then use a BaaS or PaaS, that might be easier and probably not much more expensive, but will lock you in.
What about something that does the subset of what K8s does that is the simple bit and covers most use-cases outside of complex clusters?
I get the strong sense people are just misusing it for the wrong types of task. I've seen this happen countless times in tech. Development by CV.
The problem is, while this is in theory good, the weight of knowledge and support for React (and git) make it worth putting up with a bit more complexity. That support covers: Stackoverflow, Colleagues, New Hires, Cloud Support of these things (Vercel for example using both Github and React!)
You learn the tool, then you are set for 10+ years, probably, in both cases.
In the Kubernetes case, the killer thing is it very easy to set up a cluster on cloud platforms. I am not so sure about say Docker compose or the other ones. I am in the Azure world and Azure dropped support for direct container running, and now you need to use k8s if you want something cloud managed. (As far as I know).
From my brief glances at it, it seems like a great solution for very complex use-cases. But "vast majority"? The cost-benefit seems way off unless you have a very different sense of "typical use-cases" to me.
Most people are building simple web sites and apps that run on a single server, surely?
I’m invested in Cycle.io, which is a company that simplifies containerization and infrastructure (major oversimplification of the company). But a lot of customers are refugees from K8s, whose appetite exceeded common sense. Their infrastructure ops teams are costing them 6-7 figs that they can shed with something like Cycle, because K8s is simply really complex.
This is going to become really really important in the next 3-4 lean years we have coming our way. If you’re a K8s user, and spending money you can’t afford on an ops team to manage, you owe it to your company, investors, and customers to explore options to reduce this type of burn.
P.S. Intentionally not making a hyperlink for Cycle.io because I’m honestly not trying to shill here.
I believe so, which makes a comparison between k3s and docker-compose very interesting.
Its a pain in the arse, and you're on the hook for looking after it.
I'm captain infra, so when Managed DBs came along, I felt threatened. But then when I used it, I realised that, yes, they are expensive, but I don't have to fucking worry about them. One touch deploy, proper backups are an option away, as is proper role based auth. K8s is like having a DB admin. Everything revolves around them, and it can be limiting, unless you have a specific use case.
K8s is useful for a specific role, but for most people who've outgrown a single machine/bunch of machines, ECS is good enough. Yes, there are fancy things you can't natively do, but you probably shouldn't do those just yet anyway.
Where K8s comes in useful is probably limited to:
1) mixture of real steel and cloud
2) A cluster shared with a large number of team, each deploying services that rely on other people's stuff
I've been managing clusters in one for or another for many years. K8s is certainly better than swarm, but its really not that great as a general purpose "Datacentre OS".
in short, K8s is your generation's XML.
I am curious. How much work would it take to get on this train in 2022? Sounds like more of a cult than a valuable technology proposition when presented in this manner.
Kibernetes brings me overhead and wastes system resources.
I already have infrastructure as code.
Why would I want to containerize systems I already have YAML to describe?
People forget they real hardware has to run this stuff, and layers os schedulers are not helpful.
Look at the disaster of fibers threads and processes already!
> Companies using Kubernetes for an application that is a web app.
There is probably a whole spectrum, with manually deploying code to shared hosting through SFTP being on one end and having full Infrastructure as Code running on Kubernetes with CRDs and other functionality on the other.
Personally, I think that you need to find the sweet spot:
- use containers, otherwise your app environments will rot over time, or at the least become inconsistent
- consider something like Ansible or another GitOps approach for managing the app config, consider the same for server seetup
- use some sort of container orchestration, whatever is easier and whatever you're more familiar with (Docker Swarm, Hashicorp Nomad, K3s)
- if you do go for Kubernetes (K3s or otherwise) make it as simple as possible, don't go for fancy service meshes or bunches of operators etc.
- document what you can: with Markdown or whatever for describing things that need to be known and done manually, code for the rest
Many of those advantages also apply to monolithic apps, like some garbage that may only run on JDK $OLD_VERSION or $OLD_DISTRO, but should still have its surface area for attacks limited as much as possible. If done right, developing apps like this will be a breeze without getting too overcomplicated. If done wrong, you will get nothing done and will have to suffer with figuring out how to get your Kubernetes cluster up and running properly, then will have problems with resource usage etc.
> More than one language for your application. For example, a backend in Golang, Ruby, PHP etc. and a Frontend Web App in React, Vue etc.
This is probably the most controversial argument: I will admit that using something like Ruby on Rails or Laravel, or any other server side rendering option will probably be a pretty decent and fast way to get up and running.
Yet, nowadays people like developing APIs so that other apps can be connected to those relatively easily, which is somewhat handled by most of these options as well, though developing a separate front end and dogfooding the API yourselves will perhaps be the better way to ensure that everything works as expected.
More so, in my personal experience trying to develop both the front end and back end as a single bundle can sometimes be a major pain in the behind, which gets infinitely worse when you do have some sort of a full web app that you try to embed and serve from the back end container/process/deployment as static files, because of issues with permissions, threads/performance etc. If anyone is curious, doing that with Spring (not Boot) was really annoying and kind of brittle, having to always deploy them together and having unreasonably long startup times aside.
So I can't help but to lean towards a split setup (in most cases):
- RESTful API for the back end, can be used by the front end or other integrations (I don't think that GraphQL is all that good, but it's also an option)
- some sort of a separate web app front end (Vue, React, Angular) that is served by Nginx/Caddy/Apache and may later be replaced/migrated over to another solution etc.
This probably depends on corporate policies or whatever people are comfortable with. Using those cloud services will result in vendor lock which, as Heroku showed, isn't always cost effective or even that good of an idea long term. Currently I pay around 260 Euros/year for my servers (5 VPSes) on which I run my containers, some people pay more than that per month on certain platforms. Because I settled on OCI containers as the common runtime format and the aforementioned approaches to orchestration, I can switch between whatever hosts that I want.
Previously, I've used DigitalOcean, Vultr, Scaleway, Hetzner, Contabo and Time4VPS (my current one) and can easily move, whenever a better offer comes along. Of course, for many startups with SV funding that doesn't matter much, but there is definitely a lot of merit to avoiding vendor lock and focusing on open source standards. Or, you know, you decide to work in an industry where the law book gets thrown at you and you have to do on-prem.