I'm failing to see why k8s needs to be involved here - it's overkill for most model serving cases but its involvement here now adds additional overhead. So it's not really any cloud, it's any cloud where you're running your EKS/AKS etc.
So your skillset is reusable.
There is no free lunch. But if you learn k8s, moving from AWS EKS to Google GKE to DigitalOceans hosted k8s is easy.
If you are the one maintaining it it's a full time job handling all these edge cases, it's completely miserable and I wouldn't recommend it to anyone.
Are you using AKS, EKS, GKE on those providers, or deploying your own k8s on top of the compute those providers offer? It sounds to me like the former.
But I enjoy working with Kubernetes.
The post I was replying to seemed to be saying (by analogy) “Linux is hard to manage because I run into all sorts of trouble trying to support a mixed environment of SuSE, Ubuntu and RHEL, therefore Linux is just too complicated”.
For example, the consistent use of labels as a way to identifying groups of resources that need to coordinate with each other is very useful for any distributed system. I find myself looking for them in say, CI/CD systems (in the form of agent tags), or at the application level in say, matching players to game servers.
I enjoy working with Kubernetes, but forcing a complex domain into something legible is a recipe for catastrophe. There are quirks, across cloud providers, and this is just another day in Ops, with or without Kubernetes. (See: https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call... )
"Kubernetes has been essential in making 8-figure monthly cloud spend happen"
The best though was when I ran across someone in the org trying to run a single container to run a periodic job in its own cluster. They spent half the day trying to get it to work with ingress.
You can imagine how it came to a head when the company realized they were spending hundreds of thousands per month on idle clusters in AWS.
So instead of learning how to deploy on GCP, AWS, and Azure, which is only 3x more complicated than deploying to a single cloud, you should learn K8s, which is 10-15x more complicated, in addition to still having to learn about all the various ingress controllers and weird quirks that are completely different on each cloud provider. Doesn't really track for me.
You can learn k8s in a day. It's really simple.
> various ingress controllers and weird quirks that are completely different on each cloud provider
Which are thoroughly documented and not that hard to implement or understand. You'd be reading about each cloud's nonstandard ingress even without k8s.
The beauty of k8s is you can run your software locally and have a much easier time lifting and shifting to another cloud.
Fitting to the shape of a cloud provider is a great way to never leave.
Another benefit of k8s is that you treat your services as cattle you can easily spawn and kill. Adoption of k8s naturally leads to anti-fragility, anti-brittle best practices.
But I think most developers don’t care, and instead, should interact with a platform built with Kubernetes as a foundation.
Probably the biggest one is understanding you don’t ever do anything directly with Kubernetes.
I use gpt-4 through the API where you can set your own system prompt. I developed one that basically instructed it to give me kubectl commands to solve my problems and then wait for me to give it the result before continuing. Through this I learned the practical techniques and which kubectl commands you use on a daily basis, which is so much more helpful than reading the documentation which just gives all commands equal weight.
EDIT: Oh, and definitely watch a few "TechWorld with Nana" videos on YouTube. She does a great job of explaining the architecture, terminology, and philosophy of k8s which I think it very helpful to know.
That mindset of not doing anything directly comes from understanding that you are not setting up pods up, but rather, you are setting automated processes up. These processes knows what the desired state is, and if there is something that changes things, it takes actions to get back to that desired state.
So you don’t create pods. You create deployments that maintain a set of pods. You don’t assign pods to nodes. You define pod affinities, and use node selectors on nodegroups. You use pod priority. You make graceful startups and shutdowns work correctly. And so forth.
If you understand that, you then know you can add processes (such as operators or the cluster autoscaler).
It’s a mindset shift. Thinking of it as “using” kubernetes, as if it is a monolithic thing that you directly control, will greatly increase the difficulty in understanding and reasoning through what’s going on within a Kubernetes cluster.
The complaints are real, because in practice a company needs both aspects and when a small company struggles to setup and manage the kubernetes infrastructure correctly, the application operators are suffering the consequences (e.g. log collection infrastructure doesn't work, it's hard to provision nodes with the right capacity, things like that). They see the infrastructure operation team struggle and they partake in that struggle because what is advertised to be easy, it's not.
That said, K8s is a very good way to build an "API" between infrastructure and application "teams". It can work very well if the people involved set things and processes up correctly. It can be a nightmare if botched up
Agreed.
Someone has to actually build it with a product mindset — including product-market fit. I’m one of those oddballs that have set up and scaled up infra, and have put together and delivered applications before.
In one gig, I had people joke about naming what I had setup after Heroku. I didn’t realize its significance until I came to a place where it was not done this way. Many on the application team have expressed dissatisfaction… but it’s like the infra team is oblivious to that.
Switching your database, just like switching your cloud provider, rarely happens in practice.
As I understand it this new Nvidia VM image comes with Kubernetes on the inside so to speak, perhaps microk8s with nvidia extension enabled.
BTW this is how I’ve started running my own little AI experiments. Sure, there’s some overhead. But compared with constantly downloading new versions of drivers it’s quite lightweight. Also K8S is turning into the ligua franca of sodtware platforms, so well worth learning and paying the overhead on IMHO.