To those who have used K8s extensively...
1. Is it really so complicated?
2. Is that complexity incidental or essential?
3. Could we get away with a simpler set of abstractions for 90% of applications?
1. Is it really so complicated?
2. Is that complexity incidental or essential?
3. Could we get away with a simpler set of abstractions for 90% of applications?
1. You read the O'Reilly book first (or another good book). There are a few unexpected abstractions (replicas, services, deployments, etc) which the book explains nicely.
2. You pay for a hosted Kubernetes. Google's is great. EKS is workable, but you may need to spend more time configuring it.
3. You don't mess with the networking system, and nothing goes horribly wrong.
Our clusters peak out at close to 400 CPUs, and Kubernetes generally does what it says it will do.
One caveat: If your app can be deployed using a "platform as a service" (Heroku, Render, etc), that's usually a better idea than Kubernetes. Kubernetes makes sense when a PaaS starts feeling too limited.
Most questions seem to revolve around a tiny part of the puzzle, or a small "just starting out" phase and completely forgets about the lifecycle of the business process that it is built for, and the existing systems it needs to interact with. Even a startup will have that problem considering most are trying to get bought which essentially means being absorbed into a legacy company. So even starting out with no legacy to worry about is just a stay of execution.
2. Essential. K8s solves a problem that's quite complex. You can't really solve it in a simple manner.
3. Probably, but that 10% will require the additional abstractions and complications anyway, and it'll be easier to manage one system rather than 2.
1. It's only as complicated as you make it. Kubernetes is essentially PKI (which is a must in any case), a REST API, and a scheduler. It stores some stuff somewhere, and you can add more stuff for it do have more features. I wouldn't call that complicated and it's essentially what Swarm and Mesos do as well (minus the PKI part).
2. PKI is essential. If you think that's complicated that's a whole different problem. Everything else is incidental. If adding more OpenAPIV3 schemas or REST API seem complex, again, not really a Kubernetes thing, mostly a general software development thing.
3. Yes, as 90% of applications really only exist as mediocre CRUD viewers you could run on a potato. Also, 90% of applications don't need to be as highly available or scalable as people might think. Then again, ecosystem complexity in software development combined with the lack of general knowledge (i.e. how to use an RDBMS properly) makes that while the software is simple and could be run as a single statically compiled binary, it generally is a mess, requiring more messes to make it run. But since that is cheaper (less developer time spent, more cheaper developers available to do that type of work), that is where we end up.
2- See 1.
3 - If you can containerize your app in a simple way, then yes.
Note that a stateful app that would require attention in a bare metal server or a singular VM would still require that kind of attention on K8 as well. K8 just removes the need to manage the VM infra. And makes running your infra as code much easier.
If you need to run a stateful app in a highly available manner, you can do it in K8 and it would be good - however you will spend a similar effort for maintaining the high available services like you do in other venues. Ie, if your stateful app requires a Percona cluster and a NFS cluster off of K8, you will still need to launch and maintain those services. K8 operators make these a lot easier to launch and maintain. But its still maintenance nonetheless.
Using managed, hosted databases can work for the database part. But they are expensive. So launching a database cluster via a K8 operator would be cheaper to maintain. NFS is a problematic thing across all platforms. So if you need it, you either launch a rook-ceph cluster to provide a shared filesystem or use a hosted service like Google File Store.
Even hosting redis etc.. is really straight forward.
It is funny, but the complexity starts to happen where you want kubernetes to handle other stuff: like hosting databases, or other storage resources, and if you want to for some reason I will never understand have your external services essentially communicate directly with kubernetes rather than have some middleware service you pay for do that for you (like a load balancer, etc.)
One thing I did have an issue with was setting up SSL... that was surprisingly stupid. Should have been much easier to do that with LetsEncrypt.
Then you run into a litany of issues with networking (like you mentioned SSL termination) and stateful apps or databases.
Even in this thread, someone mentioned how Redis defaults lead to a lot of issues in containers.
But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many?
See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening.
As another commentator said, there is a level of knowledge that is required to things in production reliably.
... and we're now fully back to the mainframe era with the people in white coats who "run" the computer.
The cloud truly is mainframe 2.0.
Depends. Are you a >500 Developer org with many services? Then it's easy compared to what's out there. Anything less than that I'd say it's complex and you'd be better off using a PaaS
> 2. Is that complexity incidental or essential?
Depends. If you're going to do simple things forever then it's an overkill. But if you expect to grow in unknown ways in the future and don't want to waste your time doing bunch of migrations in the future then it's essential.
> 3. Could we get away with a simpler set of abstractions for 90% of applications?
Maybe? Heroku, AppEngine, CloudFoundry tried, but didn't go too far. Let's see what new crop of PaaS offerings are able to do
Kubernetes is available as a managed service in AWS, Azure, Google etc and this is likely to be the most popular deployment model.
By any definition this is a PaaS and if you add in custom monitoring, logging, security, ingress etc. is going to be just as simple and significantly cheaper than using a managed solution.
If you're just building a basic website then sure it's an overkill but fewer people are building those these days.
> 3. Could we get away with a simpler set of abstractions for 90% of applications?
Yeah but you can do that in k8s too, check out knative serving for example. K8s encourages the creation of higher level abstractions, with the advantage that you always have the break glass to dig into the primitives, which you don’t get with a lot of other systems.
2) necessary at scale, incidental before then.
3) yes.
b) Kubernetes clusters can span multiple accounts, clouds etc.