A better Kubernetes from the ground up
blog.dave.tf
blog.dave.tf
Every time I look at k8s networking seriously it gives me great pause on whether I should continue to run such a complex system. IPv6+EtcD would solve this matter really well.
Networking in k8s is the poorest designed aspect of Kubernetes. So much so, I'm honestly surprised it didn't kill off K8s early on. You still can't get proper service layer load balancing out of the box w/ iptables.
I'm not sure the history - it's the simplest thing possible, I guess, but doesn't perform very well at the most basic of tasks. To try and solve this, we have the current mess of service meshes, CNI, etc. It's going back to the middleware days of yore - which I think anyone with operational experience should know we really need to avoid. Having multiple layers of network proxies and masquerading between service calls, on an internal network, is just ridiculous and difficult to debug or operate at scale.
I’m being snarky but it is a long time observation of mine. When I see a network config I am more often than not shocked by its needless complexity.
K8 networking is far more complex than typical networking and with less visibility
Communication between pods running across different computers gets encapsulated in another IP packet. This is pretty standard across the industry.
Broadly speaking kubernetes saves time by providing an api for existing Linux technology. For example, iptables, ipvs, storage, self-healing, virtual ips, etc via yaml.
Kubernetes was created to give anyone "Google like, production grade" infrastructure without the vendor lock-in of the cloud. And kubernetes on physical machines is where it shines most.
EDIT: it is very easy to complain that kubernetes is complex, but try to do virtual ips with keepalived, corosync, pacemaker to understand how much time kubernetes saves by providing the same capability in a generic way which is easy to use. I have the feeling that people take kubernetes for granted without knowing how much manual work one has to go through in order to build a similar experience using the standard building blocks available.
EDIT 2: straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google" that is a lot of saved time just by using kubernetes.
https://twitter.com/copyconstruct/status/1253968037579030532
https://twitter.com/copyconstruct/status/1254273432847446016
If you happen to have the problems Kubernetes solves, it's great. But even Google doesn't have exactly those problems.
(And there is a huge difference between setting up Kubernetes and hosting Kubernetes. If you know how to sit in a chair, you can be in a plane's pilot seat just fine, but piloting is an entirely different skill.)
Also, I would disregard the "moving toward" in that post. Borg has always had mutable container limits, for 10+ years (Large-scale cluster management at Google with Borg, §5.5).
Still, kubernetes makes a wonderful job abstracting all the hard work of using the underlying tools.
Look how simple is to create a ipvs round-robin by just defining a "kind: Service" object, try to do the same using the bare tools in a way that is synchronized across N computers.
Discovery via DNS is pretty standard too.
Mind to elaborate on what you would like to see in those areas?
Discovery over DNS suffers from the TTL issue right? You’re basically dos’ing yourself with DNS updates?
That's not even remotely accurate unless you run Kubernetes on AWS, Azure, GCP.
> straight out of kubernetes home page: "Kubernetes builds upon 15 years of experience running production workloads at Google".
Sounds like you need to read that sentence carefully. They never say they use Kubernetes.
It seems to me you're a Kubernetes fanboy.
Can you please expand on this and if possible point to some references that'd help in case I want to go down this route myself? Thx.
First, background reading:
https://en.wikipedia.org/wiki/6to4
https://en.wikipedia.org/wiki/IPv6_rapid_deployment
The basic idea is you create a SIT tunnel device and assign it an IPv6 /64 composed of two parts:
1. A network prefix between 32 and 56 bits long. This prefix is the same for all machines in the network.
2. A subnet derived from the machine's IPv4 address, minus the netmask.
For example, if your IPv4 addresses are allocated from 192.168.1.0/24 and the machine has 192.168.1.155, then the network prefix should be 56 bits long (64 - (32 - 24)) and the machine's prefix is `xxxx:xxxx:xxxx:xx9B::/64`.
The Linux kernel knows how to wrap the IPv6 with IPv4 so it can route within your local network to any other machine with a similarly configured tunnel device. If you want to send packets to 192.168.1.200 then they get addressed to `xxxx:xxxx:xxxx:xxC8::1` or whatever, they'll transit the IPv4 network like normal, and on arrival the receiving machine's kernel will strip off the IPv4 wrapper and route the IPv6 locally.
How's this useful? Well, if each machine has a /64 prefix then each pod can be allocated an IPv6 within that prefix without coordinating with other machines. Let's say the pod gets `xxxx:xxxx:xxxx:xxC8::aaaa:bbbb:cccc`. Anything with a correctly configured tunnel and that pod IP can route it traffic, no proxy or iptables needed.
I’ve been meaning for a while now to experiment with this same idea in Erlang. I.e., hack up the Erlang runtime to use an IPv6 address as its PID type, such that each Erlang node running on a machine gets its own /64 subnet to hand out; and each Erlang actor-process on that node gets an IP allocated from its node’s /64 range.
This could just be a way of letting Erlang nodes talk to each-other through tunnels. Or it could be a way of having Erlang “VMs” exposed directly to the Internet as their own little machines.
https://www.cni.dev/plugins/main/macvlan/
Basically just do normal ipv4 via your dhcp server rather than an overlay.
-- Edit
For arguments sake, I just set this up:
root@nas:/opt/cni/bin# ./dhcp daemon
cat /etc/cni/net.d/01-macvlan.conf { "name": "mynet", "type": "macvlan", "master": "eno1", "ipam": { "type": "dhcp", "routes": [{ "dst": "192.168.1.0/24"}] } }
PODIP = 192.168.1.181:8096
Works in my browser; so routes correctly.
Got its dhcp from my pihole.
But in production, I'd rather my ability to launch a new pod not be dependent on a DHCP server being reachable and functional. In that case, this particular trick is rather neat, since assignment of IP addresses is fully static/local (without having to agree upfront what range of IPs each node can use for bringing pods online), while retaining the benefit of everything being directly routable. You can now also run a ridiculous amount of pods on a single node.
Though to counter your point, you don't actually need to use an external DHCP server in my example either, you can just define the block you're giving the server via the macvlan/ipvlan plugin, and I presume, again, it works with both IPV4 or IPV6.
So I guess my wider point is, k8s probably doesn't need to replaced to have the networking work how you like.
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/c...
In the end there is no ideal scenario; boils down to what works best for the use case (or what is the 'least worst' solution). Sometimes it gets you down, but those imperfections can turn a churn job into an interesting one.
We haven't noticed unusual CPU throttling, though we do have some workloads that turned out to be burstier than expected and had to adjust their CPU limits to match.
Note that when it comes to subtle Linux thread scheduling behavior, your experience will depend on which runtime you use, and if using runc then which version of the Linux kernel your workers run. We weren't affected by the CFS bug introduced in Linux v4.18 because we never ran Kubernetes workloads on a machine with the affected kernel, and if a similar bug occurs in the future it might not affect workloads running within gVisor or Firecracker.
Additionally, Stripe has historically cared more about security than efficiency. This lead to an architecture where services run on dedicated VMs, which naturally strands capacity and reduces the impact of bugs that appear at high utilization and/or high core count.
Often see quota's get exhausted through short bursts that don't show up in metrics that then causes CFS throttling to occur even though it looks like the pod is no where near its limit. Also struggled with application startup requiring far more CPU than at runtime leading to ridiculously slow startup times if you had a low limit.
So far our solution has been to just remove CPU limits, but hoping things will get better.
Removing the limits really improved our latency tail, and so far hasn't resulted in CPU saturation at the node level but your mileage may vary
https://github.com/kubernetes/kubernetes/issues/67577
https://github.com/torvalds/linux/commit/512ac999d2755d2b710...
If I recall correctly you need 4.18+ to get the fix.
Like others on this thread I completely agree that k8s networking is over-the-top complex. Still, it is hard to understand how this addresses relatively common use cases like providing intelligent load balancing for clients outside the cluster or secure multitenancy within it.
Solving this requires a step back to think through simplifying networking itself. I thought amazon did the world a favor by removing L2 networking from their model. We need a better set of primitives that abstracts out as much of the low level implementation as possible. The current situation feels like data management before the advent of the relational model.
My dream platform:
1. Single binary install on nodes, and easy to join them into a cluster.
2. Resources defined as JSON with comments in a simple format with JSON Schema URLs denoting resource types - I should be able to run 1 container with 3 lines of resource definition.
3. Everything as a CRD... No resources or funcionality pre-installed and instead available via publicly hosted HTTPS schema URLs.
4. Pluggable / auto-installed runtimes based on the schema URL or a "runtime" field: containers, vms, firecracker, wasm, maybe even bare processes, etc.
5. A standard web dashboard with a marketplace which can install multiple containers, vms, wasm or via a copy-pasted HTTPS URL, or a standard electron app which lets me connect to clusters or manage local deployments on my devbox.
6. Apps can provide a JSON schema with config options, which map to env vars and volume mounts and can be displayed in a user friendly way in the web dashboard.
I feel something like this could standardize even more than kube. I'd love if one system allowed me to manage AWS EC2 / DigitalOcean / Proxmox instances, as well as manage services on my devbox for daily use (like daemons, etc.)
With that said, I do like a standard format across service providers. While I do find kube complex at times, I like that something "won" this battle, and I also like the push towards containers vs vms.
I'd love to see kube start pairing down and innovating toward making things easier for smaller clusters. Anyone know if there are discussions about that going on, or if there are resources for managing kube in smaller teams? Anyone interested in this?
If you want to stick with k8s, the k3s project is an attempt to simplify it: https://k3s.io/
We can have multiple implementations of the same API, but what I am seeing currently from the "commercial" vendors is the base K8s with UI changes. I hope we will have multiple implementations of the spec. The specs also have to evolve with time.
Last decade, systemd had its fair share of criticism. But it provided a consistent API to run "Compute Units" locally. K8s can benefit from the same principles to manage "Compute Pods" across the cluster. The concepts of promise theory [1] provide interesting control loop co-ordination.
YES! This is exactly what we should have had been able to do in K8S swiftly, but I would like to suggest one additional goal: to get a highly-distributed, fault tolerant computer cluster over different network without a hassle (as a mean to spread risk), while also having a as-low-as-best-as-possible TCO through the use of heterogeneous architecture.
So this means I can have used AWS for their Graviton instances for high performance, near-bare-metal microservices, and can also run typical x86 workload on various cheap VPS providers such as Vultr, DO and any other cheap hosts out in the wild to handle normal stuff that typically wouldn't run well on ARM, such as GitLab, Prometheus, Keycloak, in a sense that we associate x86 to have an affinity with "heavy stuff" runner and this implies ARM to be the Mr. "lightweight stuff" guy. This is not really possible by today's shape of the k8s ecosystem, given that the majority of the images in Docker Hub is predominately x86/amd64, and my wild guess is around 90%. By the way I use ServerHunter[1] to scavenge such cheap x86 servers.
Also I'm a k3s[2] user and I have attempted to do this pretty well. Given your kernel support, you can even strap WireGuard on natively (albeit my VPS host internet bill tells me this can hit quite heavy)
While I can definitely control k8s by using kubeadm but I just liked k3s's philosophy of being battery-packed: It will install Traefik (although my recent needs to run nginx-ingress ultimately let me turn it off), CoreDNS, Flannel, a simple local storage and extra goodies all out of the box in your cluster, and you can opt them out whenever you want if you think they are insufficient for your kind of workload (whilst they are adequate for most of 90% of the thing actually). To be honest, this is how k8s should have had been this simple in the very beginning to say the least.
[1]: https://www.serverhunter.com/ [2]: https://k3s.io
One-cluster-per-region (with some tie-in into one region being its own failure domain, both at the underlying infrastructure and application level) is the way to go for reliability.
In practice this is going to be tricky unless your services are completely stateless. For 80% of people that's going to be true, but if you have customers with large datasets, (I'm thinking mostly media) you do not want to be schlepping those things between clouds or between cloud and on prem or even between on prems.
I've legitimately considered running some kind of simple SaaS and behind the scenes running a Nomad cluster and having the remote SaaS agent just be a Nomad client.
...
GKE SRE taught me that even the foremost Kubernetes experts cannot safely operate Kubernetes at scale.
This is rough to hear coming from a former engineer at a top cloud company which has often lead the way on “marketing brochures”.
It has been in productions for a long time without issue. Only downside is that we have to do blue green upgrades for major versions instead of in-place upgrades.
I keep pushing for our customer to adopt eks or gke, anything but my custom code base.
This is a pretty damning indictment, honestly.
But has k8s gotten so influential that it'll be impossible to dislodge? I guess k8s has no problem breaking backwards-compatibility with new versions, so maybe somebody can propose something better even if it breaks compatibility.
As for frabric8, the project is dead.
Then they go on to to suggest:
>"If you have more elaborate connectivity needs, you bolt those on as additional network interfaces and boring, predictable IPv6 routes. Need to secure node-to-node comms? Bring up wireguard tunnels, add routes to push node IPs through the wireguard tunnel, and you’re done.
and
>'We could also have some fun with NAT64 and CLAT: make the entire network IPv6-only, but use CLAT to trick pods into thinking they have v4 connectivity. Within the pod, do 4-to-6 translation and send the traffic onwards to a NAT64 gateway."
So changing one type of NAT translation for another type of NAT translation and throw in some additional tunneling in there? How is that any simpler, more elegant or even more manageable than the current state of K8S networking? There is no requirement that you have to have an overlay network at all. If you bring up EKS cluster today in AWS the default is the aws-vpc CNI which is a flat address space the same as your VPC there is no overlay.
Then further:
>"Sticking at the pod layer for a bit longer: now that they’re mutable, the next obvious thing I want is rollbacks. For that, let’s keep old versions of pod definitions around, and make it trivial to “go back to version N”.
>"Now, a pod update looks like: write an updated definition of the pod, and it updates to match. Update broken? Write back version N-1, and you’re done."
This is exactly what using a GitOps operator does. Then they go on in the next sentence to call Gitops "nonsense"?
Not much of this is convincing or even well-throught out. This is definitely not "from the ground up." It's more like "throwing some shit against the wall and seeing if something sticks."
It is easy as pie to manage and is easy to understand and secure.
I build the clustering into relevant services as needed, because what it means to be in a cluster together is highly service specific, so you are deluding yourself by thinking that a generic external clustering framework is anywhere near the answer.
If all you ever seen is Google prescribed order of the world (Google SRE), then of course you would contemplate rewriting K8s instead of throwing it out.
A project that picks one of [firecracker, gvisor, runc] and aggressively supports it, and combines it with a barebones network overlay that assumes nodes are running on a single subnet. Perhaps, also assumes that the control plane runs on dependable hardware and not distribute everything. May be it exists already, and I just don’t know it...
Fewer moving parts the better for people like me who are not looking at huge scale but still scale enough to span several servers in a few racks.
The networking bit reminds me a lot of Mesos, which was utterly hammered out of existence with corporations blind bandwagon riding of kubernetes. Mesos networking ran along with docker’s and required you to do borg-style service port mapping (albeit in an atomic number way). What I don’t like about 90% of this (and why ECS is my jam) is I want an orchestration of containers, I don’t want zookeeper, etcd, dns-proxy, the plethora of other services to make my orchestration - orchestrate.
I spent a good few years running cloud architecture and infrastructure and managed a few SRE’s. The thing that makes our lives easier was guaranteed red/blue deployments (mentioned a bit in the article with PinnedDeployments), auto-sizing clusters to resource totals (and out again for deployments). Terraform/CloudFormation or really anything IaC. Slackbot for deployments with rapid feedback of status through the CI chain. A kin to “bot deploy <repo> to <env>“ and let the bot figure out the config from yaml in the repo’s.
I had the most joy using DC/OS. I had the most commercial success with ECS. I’ve had the most requests for kubernetes. I’ve had the worst headaches with abstractions of kubernetes (packaged installers, canonical...)
For context I’m no stranger to what the shape of production ready systems should be and can fill in the gaps given enough time to research and educate myself, but I don’t do operational work day-to-day.
I’m bringing a project to life right now and can’t but feel like while tough to learn and manage, k8s would be a good investment to make. I don’t have anyone using it yet, so I just did the bare minimum to get my docker-compose stack up and running on a Linode box. It works great for making sure what I have now works in a remote environment too, and I had to do a decent amount of configuration rework to get it ready, which should be transferable.
Now I’m wondering, how will things like rolling deployments work? I want to decouple the monitoring stack from the application stack, how will I handle adding another physical machine to my setup? I’m sure more questions will come up like this as I run into them, but would be curious to hear initial thoughts from anyone here to help me make a decision :)
In regards to your specific questions... adding another physical machine to your cluster would be really simple if you are using Terraform, you would just increment your node count number and then re-apply the Terraform config. Here is an example Terraform config(main.tf) from a simple project I have:
variable "do_token" {}
provider "digitalocean" {
token = var.do_token
}
resource "digitalocean_kubernetes_cluster" "your-cluster" {
name = "your-cluster"
region = "sfo2"
# Grab the latest version slug from `doctl kubernetes options versions`
version = "1.16.6-do.2"
node_pool {
name = "your-pool-1"
size = "s-1vcpu-2gb"
node_count = 3
}
}
If you are running this on another test/development cluster (e.g. minikube) prior to deployment, then you should have great confidence that this will succeed. In the event that you did run into an issue, just roll back the docker image version number and reapply the yaml using kubectl again.
I've been following this method for a while and have never had any downtime with deployments. Eventually if you get sophisticated you'll want to add these steps into an automated CI/CD pipeline, but kubectl apply can carry you pretty far in solo operations.
The other principles we developed are “controllers” only control one resource and all resource updates for a resource kind must be serialized through a coordinator. Our paper has much more detail: A Cloud Native Platform for Stateful Streaming, https://arxiv.org/abs/2006.00064
I really liked the part about simplifying networking, I really feel there should be a more general push towards both ipv6 and srv records.
This GitOps 'nonsense' gives me a well defined and automatically backuped infrastructure setup with audit build in. It doesn't allow someone to snowflake around which is brilliant and forces you and your colleges to manifest stuff and not forgetting it and degrading your system over time (nix and nixos are also great examples of such systems)
This reminds me of the time i learned html and wanted to set new lines all over a text instead of using proper paragraphs and letting html take care of the proper formatting.
I would like to see a better/stronger statefulset though. As long as this pod is alive, make sure its state is not interrupted. Like allow a pod to be migrated to another node.
Nonetheless, i'm in the middle of setting up kubernetes with kubeadm and cilium network. Its already really easy to do so. It will just get more easy and more stable over time and its already great.
When you look at the storage example: yes its more difficult then just using a hard drive. But you ignore the issue with one hard drive: Backup, checksum / bit rot and recovery. With a storage layer, you can actually increase replica count, you can backup ALL storage volumnes automatically.
the same with networking: With cilium you can now have a lightweight firewall with dns support.
It is much more critical for the whole industry to start rebuilding software to be more cloud/container native. This will reduce the pain points we have right now and will make it more resilient to operate. For example Jenkins: Instead of one big master, have a ha setup for your working queue, a pod for a dashboard and schedule workers on demand.
My personal conclusion: Don't use it, if you don't need it. If you need it, embrace the advantages.
Which core benefit is that? I’m not following.
> This GitOps 'nonsense' gives me a well defined and automatically backuped infrastructure setup with audit build in. It doesn't allow someone to snowflake around which is brilliant and forces you and your colleges to manifest stuff and not forgetting it and degrading your system over time (nix and nixos are also great examples of such systems)
TFA says you can still use the GitOps “nonsense” if you want under his proposal.
But imagine a database as a pod with 60gig of ram and a ha setup. Now you need to update your node, what does k8s? It will throw it out and creates a new one which needs to recover or read all the logs to fill up 60gig of ram again from nothing. Instead it could migrate this pod to another node and keep the downtime to a minimum.
Or a jenkins master, it has to shutdown on node 1, recreate to node 2 which takes time and then your agents need to be able to recover from it.
You have to be able to roll through your whole k8s infrastructure to update every node on a regular basis; Alone for security reasons.
> A modest expansion of the previous section: make each field of an object owned explicitly by a particular control loop. That loop is the only one allowed to write to that field. If no owner is defined, the field is writable by the cluster operator, and nothing else
This is already a thing starting 1.17 i think with server side apply https://kubernetes.io/docs/reference/using-api/server-side-a... (except it’s opt-in)
Kubernetes is obviously overly-complicated. I’m just using it in a CI/CD environment and, in a month, have hit most (all?) of the issues mentioned in the article.
I don’t think it’s fixable. It’s the interesting to see an expert in this area come to the same conclusion for the same reasons.
That probably argues for "Deployment shouldn't have been a core object type", and I think k8s folks generally agree on that now, in hindsight. But the idea of generalizing CRDs to the extreme is relatively recent.
I'm running this in almost IPv6-only setup (https://blog.42.be/2020/11/using-nomad-to-deploymanage-conta...). I'm so glad I don't have to mess with overlays and NAT.
Doesn't kubectl rollout rollback already do this for deployments?
> Pinned deployments
Service meshes like Istio let you run mutiple versions of services that you can selectively route traffic to. You can sign up for it if you need it. What value do pinned deployments add over that?
I kind of get the problem parts of k8s networking. But other than that this seems like it makes already complicated kubernetes some more complicated for not so convincing reasons.
Not tacking on a service mesh middleware that I now have to manage and debug.
I tried to write a cluster manager from 2010-2015, and essentially what it did was write a big shell script for starting a service. The node could then reboot on its own, and the init system would invoke all the shell scripts.
The "host" part of the command line was on the left, and specified the container, and the user parts go on the right, after --
my-container --fs /images/mine --bind ... -- my-server --port $PORT
I still think that is a good paradigm ... The whole thing could just be a "shell script compiler", where the shell script uses a very small number of tools.
The problem with cluster managers is that you get into the "inner platform" problem. Once you have a lot of software to run the cluster on each node, now you have to figure out how to update and monitor THAT software. But you don't necessarily have the update and monitoring mechanisms it implements!
This would mean better integration/autocomplete in editors and no time wasted on silly things like number of indents being wrong.
Me too, hence why I look forward to projects like Project Tye and Quarkus to take us out of the current trend.
He had some ideas for a sizable re-architecting, but was rapidly burning out, oh I dunno, like 1 year ago, 18 months ago. Maintaining such a vital yet support-needy project seems like it was taking quite a toll.
I love this "A better Kubernetes" theorcrafting but the post I really really want to see from Dave is "A better MetalLB". The indie operators everywhere would love to have some sights on what a better MetalLB might look like.
MetalLB is in a healthier place now than with me at the helm. It's being maintained and developed by people who still believe in and use k8s, which is how it should be.
I'm glad there's good people at the helm. I don't know how to avoid uncomfortable personalization, but this really was a very notable incident for me, where on Twitter you talked a bit about how you didn't intend to blog or talk or converse with people about what sounded like some fairly large changes that you had in mind.
This is such an absolutely critical quasi-mystical level of the stack, in an area that's totally essential for people to get themselves online in a meaningful capacity, yet one which so few folk have much experience with. I'm glad we have new maintainers, but I still feel the loss of some greater brighter alternate reality where more people are able to better build their own stable online presences via the alternate-history version enhanced/remade version of MetalLB that, it feels like, no one will ever know. I can't imagine how stupid & terrible trying to support MetalLB must have been, I can't imagine what it feels like to have giant powers that be take & take without investing back, but I also just think of that brighter future, that unknown better model. Surely we'll make more goes in the future, more tries, to make better ways of bringing traffic in, & it seems sure MetalLB will continue to be one of those chief ways, but those paths will not be touched with the intimate grace of experience you, the creator's long trekking would have brought.
I've thought about trying to beg access, beg for some time to try to document the learnings, lessons, would do-overs over MetalLB. But I don't think I can bring enough critical review & questioning to bear, I don't think I'd ask the right questions at the right times. I'd hope time might soften some of the hard feeling, but I also knew I'd be trampling on a year+ old declaration, a declaration essentially that the world did not deserve your insights.
I'm sorry for punishing your good faith good reply with this post! I write it sort of as a confession, but the crime is the confession, not the thought. This has been a big event for me; apologies for being un-easy with it. Mostly, thank you thank you thank you, for having carried us all so so so very far.
So, what should most of us be using instead? Suppose I have a 12-factor app that I want to deploy on AWS (directly, not via a third-party PaaS like Heroku). Does anyone know if Amazon ECS is significantly simpler than Kubernetes? Is there a better option than either of those?
I mean, we all know that EC2 is still there, right? An EC2 instance is like a computer. If you have a program, you can run it there. You don't even need a container!
I just don't know a non-sarcastic way to say this. Containers, and especially container runtimes, just aren't necessary. Containers can be _useful_, certainly. But for decades before they were invented, we ran programs on computers and it was pretty good. We can still do that.
The biggest downside was that it took quite a while to spin up new tasks (but that might be Fargate specifically) which hurt us when we were trying to do async things (and we wouldn’t be able to use lambda for many async things because the run time would be too long or the enormous bundle size for a small-ish Python task would exceed Lambda’s limits). It would also make deployments take longer than we wanted (we were really pushing the envelope for frequent, tiny deployments) especially in rollback scenarios.
Dream bigger: why is Kubernetes so complex? Re-inventing what we already have. Why is it so complex? Lack of standards, lack of flexibility, too much focus on features.
A Linux system has a lot of complexity, but it very rarely gets in the way of the other bits, and it can [mostly] all be replaced as needed. Why? ABI compatibility, kernel-userland split, a big collection of independent composeable tools, a framework that provides everything you need but doesn't force you to use it in the most difficult way, and definitely doesn't force "patterns" on you (like "deployment").
On top of Linux, we've built the world's most advanced and wide-ranging systems, particularly because it's not opinionated. It gives you just enough rope and bamboo to build a hovel or a skyscraper. It is not modern, it isn't the best design, it doesn't force you into the minutia of understanding the system.
A new Kubernetes should either be stupidly simple, or incorporate itself into the OS, since the OS already has most if not all of the components of Kubernetes. They're just not being used properly.
Picking some parts to comment on at semi-random:
> For that, let’s keep old versions of pod definitions
> around, and make it trivial to “go back to version N”.
>
> [...]
>
> Bonus things you get from this: a diffable history of
> what happened to your cluster, without needing GitOps
> nonsense. By all means keep the GitOps nonsense if you
> want, it has benefits, but you can answer a basic “what
> changed?” question using only data in the cluster.
If you want an audit log, the obvious place is Git. I don't know why the OP derisively calls this "GitOps nonsense", because Google does the same thing and you should too. Figuring out what changed two months ago is much easier when each change has a commit message and a reviewer.
> The latter is the bane of MetalLB’s existence, wherein it
> gets into fights with other load-balancer implementations.
> That should never happen. The orchestrator should have
> rejected MetalLB’s addition to the cluster, because
> LB-related fields would have two owners.
> So, for starters, let’s rip out all k8s networking.
> Overlay networks, gone. Services, gone. CNI, gone.
> kube-proxy, gone. Network addons, gone.
Kubernetes already suffers from insufficient customization in some areas, and networking is one of the few bright spots where it gives in and lets the operator do whatever we want to. IPv4? IPv6? Dual stack heterogenous routing? CNI lets you mix-n-match anything you can put into a binary as long as it can output JSON, and if it were even slightly more opinionated then it wouldn't be fit for purpose.
> Let’s give every pod an IPv6 address. Yes, only an IPv6
> address for now. Where do they come from? Your LAN has a
> /64 already (if it doesn’t, get with the program, I’m
> not designing for the past here), so pluck IPs from there.
>
> [... lots of description of a thing that could be a
> CNI plugin ...]
>
> That leaves bare metal clusters out in the cold, sort-of.
> I argue this is a good thing, because there is no
> one-size-fits-all load balancing.
> We’re going to focus on doing one thing really well: if
> you send me a packet for a pod, I’ll get the packet to
> the pod. You can take that and build LBs based on LVS,
> nginx, maglev-style things, cloud LBs, F5 boxes, the
> world’s your oyster. And maybe I’ll even provide a couple
> “default” implementations, as a treat. I do have lots of
> opinions about load-balancers, so maybe I can make you a
> good one. But the key is that the orchestration system
> knows nothing about any of this, all it does is deliver
> packets to pods.
> I think this mostly translates to syncing more data down
> to nodes in persistent storage, so that nodes have
> everything they need to come back up into the programmed
> state, even from a cold boot. Conceptually, I want the
> cluster orchestrator to populate a set of systemd units
> on each machine, and then switch to a very passive role
> in the node’s life.
>
> [...]
>
> One way to view this is that in my “distributed” cluster
> my pods are more likely to be unreplicated pets.
I don't know why they would want that, since if they're using Kubernetes at all then presumably they've got at least a few hundred machines being managed by a couple different product teams.
There is a place for a central service that writes stuff to `/etc/` and assumes that individual machines are meaningful, but that service's target audience is completely separate from that of Kubernetes. There's no point in trying to design a replacement for Kubernetes to fit that market, any more than trying to design a 16-wheeler that competes with minivans.
I think this, unfortunately, does not hold up in practice. I am seeing way too many small orgs/teams use Kubernetes to orchestrate dozens and even sub-dozens of machines. From the perspective of solving for sub-dozens to thousands, I think there is some merit to TFA's perspective, even if it doesn't match the Kubernetes use-case you describe.
If I were advising a startup on technical architecture, I would recommend they write their software as if it runs in Kubernetes (avoid hardcoded configuration, bundle dependencies into the build artifact, use mTLS instead of firewalls) but otherwise behave like someone running a LAMP stack in 2003.
... but, using k3s is a lot simpler than fighting Ansible, it gives a standard. Yes, it takes a week to get used to it, but then it works, and it gives a lot of benefits.
Service with ClusterIP creates an internal IP that redirects traffic to that service's pods. Service with NodePort creates a CluserIP and creates a port on every node that lets external traffic also reach those pods.
Service with LoadBalancer creates a ClusterIP, NodePort, and a whatever load balancer implementation exists in that cloud to route external traffic to those K8S nodes and nodeport port numbers. This is the part that's missing in your bare metal setup, but you can create a nodeport service using port 80/443 and have DNS pointing to your nodes' public addresses.
That's all it takes to get cluster access and you can point this Service to another downstream Service/Ingress if you want the routing rules to stay in YAML too.
That's all the clouds do anyway, they just run a bunch of load balancers that move external traffic to the nodes of your K8S cluster with the port specified in the Ingress/Service and K8S takes it from there. There's no shortcut, the traffic has to be routed somehow so if you don't use a managed service then you have to do it yourself. It's got nothing to do with "developers keep it enterprisey".
Here's an answer I wrote about skipping GKE's cloud LB to accomplish the same thing: https://stackoverflow.com/a/54297777/173322
Also try using Ambassador as a reverse proxy/ingress/LB. It uses Envoy and is much faster and more configurable. You can set it to use the host network on your nodes and skip the K8S cluster mapping: https://www.getambassador.io/
What you're looking for is called an Ingress, which is like a Service that has more advanced routing specifically designed to support multiple hosts and backends from one "ingress" point for the cluster. The Ingress (again a set of routing rules) can be implemented by many different proxies like Nginx, HAProxy, Traefix, Caddy, etc. Or you can use Ambassador like I mentioned which bypasses the K8S Ingress and uses its own streamlined config with better performance, observability and automated HTTPS too.
If you want any better load balancing and/or failover, you'll have to either have something provided by your cloud provider (an LBaaS, and then tie that into Kubernetes via some controller), or have a network topology that can be exploited for this (shared L2 with a VIP or L3 with BGP, so that you can use something like MetalLB).
This is not a Kubernetes problem, but a networking problem.
1. be run in a specific range
2. you can expand that range, but if you do, any port in that range might be randomly assigned, so your service might start up on port 80.
Changing alone would make using Kubernetes 10x easier for small operators. But one bomb-throwing curmudgeon drops by with entirely content-free unsubstantiated anger-posting:
> Unless I'm misunderstanding the proposal, it involves unpredictable and difficult to diagnose failure of services, which seems like a complete non-starter to me.
And the small-operators who just want some ability to open http or dns ports get nothing. Maddening beyond belief.
On a different note, I wish we could use postgresql instead of etcd. That change alone would allow for amazing things with k8s.