Tools to Run Kubernetes Locally
yankee.dev
yankee.dev
We called it localkube, and it was an all-in-one binary with features stripped out that ran the kubelet, apiserver, kube-proxy, etcd, and other components as Go routines. Eventually it had to be removed - upstream k8s was moving too fast, and there were fundamental issues that couldn't be fixed without large patches (TPRs, the predecessor to CRDs were one).
The problem with this approach is that it requires a ton of maintenance and there can be subtle bugs that come up.
The other distinction not found on this list is that only Docker for Desktop and minikube are cross-platform solutions. Running Kubernetes and docker on Linux is somewhat trivial, but managing a lightweight virtual machine and adding that extra layer of abstraction is what makes it really difficult.
The hope would be that we would stop trying to chase our tails keeping up to date with K8s, and instead we could maintain a more generic abstraction layer which could be applied to other PaaS/SaaS systems as well. You use the abstractions to separate the design from the implementation. The "desktop" implementation could be insecure, non-HA, non-robust, easy to install & maintain. The "cloud" implementation could be all the best practices. The containers would be the same, but the service definition, command-line tools, etc could all be different, and just be written to interface with the abstraction specification.
Obviously you still would need to completely test the hell out of the service in the cloud cluster before releasing to production, but the initial local development could be much faster when done locally in a simpler system. I think trying to replicate the cloud implementation locally is just asking for trouble, we know it will never match up.
Correct me if I am wrong, but I don't see trend that k8s is getting huge. In fact, I get a feeling that it is trying to be smaller. If you look at the release notes for each minor version, lots of in-tree features are being deprecated and users are being pointed to out-of-tree implementations. DNS, networking, metrics server, cloud vendor specific features(storage class, load balancers), container runtimes...just to name a few. And the community put great amount of effort to harden API machinery, to make it easier for 3rd parties to implement specific features for specific business.
This kind of gives me hope that one day I may be able to just run a vanilla k8s straight out of a release from github.com/kubernetes/kubernetes without any "batteries" as a platform, and I can later decide what I want to install, all by myself.
(2) It's one of the main products of a venture-backed company. At Google, at the time, it was tough to get engineers on products like minikube.
It takes 1 command and 1 minute to spin up a 3 node cluster from scratch, everything works the same as you would expect on a real managed Kubernetes cluster and it took a few lines of YAML to config the cluster so that the ingress works in such a way where you can write your manifests exactly how you would on a managed provider. It's also super easy to get your local Docker images into the cluster.
Did anyone else come to the same conclusion? Maybe I'm missing something but you can't really ask for much more than the above.
For k3s, it mainly comes down to wanting to use k8s not k3s. I'd like my local cluster to be as close as possible to what's running in a real managed k8s cluster on a major cloud provider. This might come off as naive but I want to use k8s in dev / local testing because I'm using k8s in production.
I also found kind's documentation much better than k3s for wanting to accomplish the task of setting up a cluster and using an ingress. It also had easy to find and follow docs for WSL 2 to configure extra port mappings in the cluster config so I could access external services (managed with an nginx ingress) over a defined port. Basically the dev experience has been flawless.
Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days.
Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about.
I have not found a modern/open-source tool that is similar, but I have to imagine it exists.
kpt: https://github.com/GoogleContainerTools/kpt
kapp: https://carvel.dev/kapp/
cnab: https://cnab.io
There are many (many) more. But these are somewhat popular recently.
(Rendered manpage at https://knot8.io/, if you like that exposition format)
kubecfg (with jsonnet) or CUE. Gives you actual composability and a powerful programming language for reducing boilerplate. Oh, and no text templating.
With kubecfg deploying/updating is:
kubecfg update foo.jsonnet
One of the issues for those running applications in Kubernetes is the complexity. Kubernetes is complex. So much so that people like Kelsey Hightower have said Kubernetes is a platform for creating platforms. For most people to learn Kubernetes will slow down their velocity which is a negative.
Most people need a simpler system on top of Kubernetes to make them effective.
For instance, when people upgraded to 1.16, where Kubernetes retired apps/v1beta1, Kubernetes automatically transparently converts these types.
Helm, having once created these resources as v1beta1 will completely refuse to touch the deployment. Without some specialized extra tooling that people created in the wake of this, resolving this requires deleting the entire namespace and reinstalling the chart. Issue on helm repo closed as wontfix.
If someone was using raw kubernetes manifests instead of Helm and charts they would need to know the schemas and details of those manifests. Then, when 1.16 came out they would need to update to the new apiVersion and modify their manifests for any differences in schema.
This requires someone who knows and understands k8s resources. This IS NOT a typical app dev or person who wants to run apps in k8s. I've been in numerous circles of app folks who have complained about this expectation from the k8s community.
On the main Helm project they have the stance that what's in a chart is up to the charts authors. Just like what's in a debian package is what's up to the debian package authors. That those folks should do a good job maintaining them. That's why it was a wontfix issue.
What we see as an underlying issue with k8s is the experience that app devs and people who want to run apps have to deal with. It's currently like expecting an app dev to write in assembly and know when assembly codes are deprecated or changed. It's in the pre-high level language phase. This comparison to assembly is one that's come out of the k8s community itself.
Helm is still functionally broken. A helm chart that was kept up to date still could've used v1beta1 early on. All it takes is a cluster that hasn't been patched in a while. This is 100% a helm issue. (Note that this bug also triggers if the chart has been updated, but helm recorded a different GVK)
What you actually need is a platform team that provides a middle layer. I was part of such a team between 2018 and 2021, and my main takeaway was to not let developers have write access at all after slicing a namespace with 25k pods out of etcd because gRPC refused to operate on it at that point (that was a misconfigured CronJob).
You need something like KubeVela, not Helm.
Here's the rule of thumb for Kubernetes use tools to manage the files not the cluster, the cluster description is the source of truth not the cluster.
It reminds me of the old article "Beating the Averages": http://www.paulgraham.com/avg.html
It's not a great API, but it certainly beats writing bash scripts that shell out to kubectl and yq.
In the same vein, I wish people knew that YAML isn't even really a native Kubernetes thing, it's just one of the many ways you can serialize k8s objects...
We did this for a while, from a container rendering out our stack in Go, when server-side apply was not a reality yet.
The main issue is that lifecycle management of resources beyond writing code for a single deployment is hard to do with the typed clients, and the dynamic client used to be even more of a mess than it is now. There's a lot of implicit magic happening, baked into kubectl in a non-reusable way you just couldn't replicate before 1.16.
The lack of this is what frustrates me so much about all of these things. Writing some bundle of yaml but just telling the management software "here go run this 'app'" with no state or version management of it makes me feel like I'm running a stack like it's a windows desktop. Just random shit everywhere and I don't know what any of it is. Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows.
All feels like a huge step backwards to me. Let me use version control as a source of truth, not just a spot to throw things in.
See Argo CD.
At my workplace, we can easily spin up a kube cluster locally but our Helm charts aren't transportable for local development. They can be modified, torn apart, extracted from, but we don't have a seamless and elegant way to say "Hey, this thing in production, that's running via Helm/k8s, I want the same thing locally ... with these environment variables adjusted"
I don't think that's a difficult thing to accomplish and there is appetite from others I work with to do so, but there's _effort_ behind it.
Maybe there are smarter ways and we just don't know it yet.
I think it may have been this: https://github.com/fkorotkov/k8s-kotlin-dsl
I have worked with Marathon and Kubernetes in the last few years and never had the need to deploy them locally, it seems strange to even need that
now your dev env is very different from how production works. Problems that show up in production won't easily be reproducible in dev. And if you are testing in this dev env, you aren't really testing, so implicitly turn to testing in production!
I have not tried k3s or any of the other attempts to produce a stripped-down version, so I'm curious if they fix this problem. But the fact remains that the Kubernetes daemons are heavily based on polling; even a Kubernetes instance that does nothing will be using a lot of CPU.
Right now, the tool of my dreams does not run anything locally at all, but transparently maintains a small remote Kubernetes cluster in the cloud of my choice. The benefit here is that I could get all the tools at my disposal, such as public load balancers, ability to effortlessly scale up my nodes when needed, etc.
What do people think is the best ‘stack’ to invest in? I’m also curious about experiences with CapRover and/or FastAPI.
Those two tools have been around for a while and you’ll find plenty of documentation and other people looking for help online.
Package your apps into containers. Then run them somewhere. DigitalOcean Apps, AWS Lambda, and Google Cloud Run are all fine. So is Kubernetes.
We've been working on something taking a different direction that makes it a simple app to run. It's called Rancher Desktop (https://rancherdesktop.io/). It uses k3s under the hood. The goal is to make running k8s locally as easy as running a normal desktop app.
Note, as the creator of Rancher Desktop I'm entirely biased and this is a bit of a shameless plug.
And do what? This is the most important question.