The Kubernetes Kustomize KEP Kerfuffle
gravitational.com
gravitational.com
kustomize vs Helm is a bit if an improper comparison. Kustomize and the ways to use kubectl are often about configuration management and Helm is about package management. This is an important difference.
For example, take Ansible doing configuration management and using Helm to install a package in much the same way you might use an apt package elsewhere.
The key question that has to do with configuration management is should kubectl provide configuration management of things outside of the Kubernetes API, such as when you manage the config in git? This is both about scope (is this a Kubernetes concern) and in how to build things (a monolithic CLI?).
If kubectl has a configuration management opinion outside of the Kubernetes API what does that mean for configuration management projects like ansible, armada, chef, and the oh so many others? And, who working on Kubernetes gets to choose the opinion on configuration management it has outside the API?
At this point imperative vs declarative as a conversation point comes up. In all cases the configuration is declared to the Kubernetes API and stored by Kubernetes in a declarative manner. It becomes a question of how far outside the Kubernetes API do things need to remain declarative. After all, me typing into yaml into a keyboard is imperative.
As for the KEP process as a whole and the graduation criteria that was mentioned here, we know it needs work. There are efforts going to get it into better shape with better example guidance. There is actually a PR being worked on for this right now.
Projects with a healthy ecosystem and people of strong opinions have to deal with issues of scope. What do you hold onto and where should it go? How do you curate a healthy ecosystem? It's not always easy.
Having a place to do some experiments outside of a project like Kubernetes is part of the reason the CNCF added sandbox projects. Without being forceful people can work together on something and see how it fairs in the market. Experiments in the ecosystem can happen.
In any case, if people have questions please feel free to ask.
However, when it comes to cluster operators, or "runtime configuration", the application packager cannot possibly know all possible configuration changes that a cluster operator might want to make. Some are obvious, and Helm ecourages putting them into values.yaml. For example, most charts include a way to change resource limits or imagepullpolicy. But what if a cluster operator wants to change a config map to be deployed as a bitnami SealedSecret, because it's more secure and more compatible with the infrastructure? Or what if a cluster operator wants to deploy the Cloudflare Argo Tunnel Ingress controller which requires more: not only an annotation, but also additional fields. This cluster operator often ends up forking the Helm chart or deploying these parts out-of-band. Differentiating between runtime config and application config is why both Kustomize and Helm are both useful, together.
I like Helm as a package manager. But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included?
I like the sandbox projects and how the CNCF is managing them. I'm actively working on a different project (Open Policy Agent), and it's much younger than Helm or Kubectl. I agree with your concerns about the KEP process, and how it should be more well defined and followed at all times.
You can move from deploying to AWS to Kubernetes all while your CI pipeline and config management stay the same. Maybe at another time you change your CI system.
In all of this we can even have tool fights. Jenkins vs CircleCI. vim vs emacs. And new players can come along like vscode.
Different departments in the same companies, sometimes billion dollar companies, can even do this.
This is one of the reasons I personally like a separation of concerns with projects.
There is one thing about Helm...
> But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included?
I don't think so. Without being long winded, this is about user experience and encapsulations. Kubernetes is hard to learn. Then do learn the business logic for installing all the things you use. To use the old wordpress example, should someone installing wordpress into a cluster need to know the business needs of MySQL to create the k8s config for it? Most app devs don't want to learn all the k8s object configs... it's a lot to learn.
As for CRDs, that's about dependency management.
But, if we make Kubernetes really hard, with lots of CRDs we have to start to deal with what CRDs are in what clusters? And, what experience does that provide for end users? How do we not make the UX barrier to entry to high for the people who need to use it?
Are users interested in Kubernetes or are they interested in their apps and core business logic?
That's one of my favorite things about that composability of the Kubernetes tools. I don't believe there's a reason for a "tool fight" in the Kubernetes ecosystem. There might be disagreements about best practices, for example how to manage code in a cluster (gitops vs kubectl apply vs helm install vs ...), but that's not a tool fight as much as a methodology difference. And there's room for more than one pattern to emerge.
I think that any tool that is able to deploy an open source project to my cluster, while minimizing the amount of operational overhead I need to assume, is the tool that I want. I don't care if that is Kustomize, Helm, Ksonnet, or anything else, as long as it mets the requirements of a) it has to work with my environment and b) it shouldn't introduce unecessary operational overhead.
You also mention that Kubernetes is hard to learn, which is absolutely a problem. Adoption is growing, but it's getting harder to learn as more features get merged in. And you are right, nobody should have to learn Kubernetes YAML to deploy a standard, off-the-shelf Wordpress installation. But what about more complicated software that needs "last mile" customization done to work in a specific environment? This blog post (https://testingclouds.wordpress.com/2018/07/20/844/) shows a great way to combine the power of the Helm community and chart format with the last-mile tooling that Kustomize can provide to help keep charts simple but still flexible. That feels much better than forking the chart and maintaining a separate copy of it just to make a few changes that are specific to a single use case.
I have mixed feelings about Kustomize getting merged into kubectl. I don't like the idea of Google "crowning" a winner, and I hope the sig-architecture group and Google teams remain diligent to prevent that from happening. Kustomize is not a replacement for Helm, it's a very good tool to handle specific use cases that often involve Helm charts at the source.
From afar it looks like `kubectl -k` is due to land despite the expressed concerns. What was your response to sig-arch and sig-cli leads who mentioned that the kustomize features were always intended to be there? I may have missed that, if it was public.
Do you expect that the march toward kustomize inclusion will change after the kubectl usage data has been collected and analyzed?
And what are you setting your sights on next? May I write about it? :)
The document describing what kustomize should do and that looked at some other tools was started in mid-2017. Do you have tracability to the idea that overlays were meant to be there from the beginning? I ask because I'm honestly interested in reading that.
As far as people go, 1 of the 3 SIG Arch chairs has been around a long time. In addition to him I would want to poke at the co-founders on this one. Joe Beda and Craig McLuckie (2/3 of the K8s cofounders) started Heptio and they produced ksonnet (kubernetes + jsonnet which both came out of Google). This is different than the direction for kustomize.
Kubernetes has been declarative from the beginning. It's an important concept in all of this. Isn't declaring it to the API and then storing/creating objects outside the API separate concerns?
Back in 2017 Helm was a core part of Kubernetes as were projects like Komponse, that lets you convert Docker Compose files into Kubernetes manifests. Helm v2, which was part of Kubernetes, was the product of merging deployment manager, a Kubernetes project out of Google, and Helm v1, by Deis. Helm v2 began work before the CNCF existed. It was a part of Kubernetes back when it was governed by Google.
A history of tooling outside of the API had long been there.
To not have opinions outside the API to allow more diversity there, in 2017 we started treating Helm and Kompose like other ecosystem projects. We talked about not adding opinionated projects like this within Kubernetes anymore to keep the door open for the ecosystem and diversity there.
> Do you expect that the march toward kustomize inclusion will change after the kubectl usage data has been collected and analyzed?
I think it depends where the usage data comes from. Can we get real end user data? Or, will the feedback come from the Kubernetes insiders as has happened too often in the surveys? (I ran past surveys so I feel this pain) What will the data look like from large companies with bespoke processes and strong opinions on their toolchains?
I hope real useful data is collected? I would love data on more of the problem their facing.
To pick on some almost unfairly cherry-picked examples:
- the KEP called out "don't want to run scripts from github" as a design goal _for_ kustomize, when kustomize itself clearly ran arbitrary scripts in well advertised situations. Nobody during the (lengthy and open) KEP review stage raised this. (Since fixed by removing this kustomize feature)
- the various "to plugin or not to plugin" discussions are highly subjective (and always will be) and dance around the elephant in the room: "We think the plugin mechanism is great" obviously doesn't marry with "nobody wants to use it".
- the scope and approach of kustomize vs other "less stylistically opinionated" alternatives was never (openly?) considered afaics. For example, perhaps we could _just_ have a stack of json/s-m-p patches that get applied, or a generic mechanism for applying arbitrary "post-process" transformations. The kustomize KEP didn't really consider "not using kustomize" as an alternative, which is understandable from a kustomize pov (we must presume the authors like their proposal!), but not from a sig-cli and overall process pov.
Afaics, the kustomize folks did everything right through this, as did the rest of sig-sli, and the above is only apparent in hindsight. The reality is that the sig contributor group is small, and necessarily can only consider proposals that actually exist. The KEP process is entirely human-based, with all the biases and fallibility that comes from that. I don't have any useful suggestions for how to institutionalise scepticism that won't also bring far worse chilling effects. I note the KEP template already includes "drawbacks" and "alternatives" sections, but it relies on cultural expectations to ensure these are used effectively.
TL;DR: I actually think this "kerfuffle" is an example of the larger process working _well_. A decision graduated to a larger audience, was questioned, and rolled back to be reexamined in greater detail. We must _expect_ to have to iterate on some decisions sometimes, and that's ok, normal and healthy.
It's easy to see it as politics. I do, pretty much all the time, but that's because I have a background in politics. I'm primed to assume the worst.
I think the explanation is simpler. If you are a Googler, you mostly talk to Googlers, you mostly come to quick consensus that way, and it all makes sense to you and the Googlers you had casual whiteboarding sessions with. So you just go ahead with the to-you-obviously-best decision and get surprised when non-Googlers are mad at being presented with a fait accompli.
There is no ill-will and no conspiracy required. Some of the personnel allocations are driven by corporate strategy, but folks engineering on the line are all reasonable human beings. It's just simple personal dynamics. You talk mostly to the people closest to you unless sustained effort is made to keep stuff in the open.
This is why it was open sourced, as a balance between control leverage and adoption.
I feel like I read this kind of statement all the time and I don't understand the reasoning behind it.
Google came up with k8s. They could have kept it for themselves, or, open source it. They open-sourced it. It definitely benefits them, but, it also benefits us a lot.
Saying they "open sourced, as a balance between control leverage and adoption" is so cynical. So I'd like to ask, what's the alternative? Not open-sourcing? Who would that benefit?
What would be the non-ill-intentioned alternative that Google supposedly could have chosen?
After all, the created omega but it never did replace borg. Kubernetes wasn't meant to replace borg, was it?
Google didn't create and release Kubernetes out of the goodness of their hearts. Have you ever pondered the strategy behind it all? Maybe they couldn't have kept it to themselves because that would have defeated the underlying purpose in the first place?
Umm, we didn't?
> created omega but it never did replace borg
It did have material impact on Borg.
Sometimes the strategy is simply "we know how to do this, and we'd prefer not to see you go through all the same pain to figure it out".
'tis fascinating to dig in to open source serverless on Kubernetes options only to realize that there's probably 100+ engineers working on istio+knative, ~10x more than any prior alternative.
When is KnativeCon coming?
Since folks get so worked up about allocations unofficially shaping decisions, I suggest[^] that Knative be homed with the Cloud Foundry Foundation, where the rules are written to give votes based on how many fulltime contributors you assign. That way it's all out in the open.
Disclosure: I work for Pivotal, which is in both the CFF and CNCF. I was aware of Knative relatively early.
[^] in a ha-ha-only-serious way
Brendan Burns went to Microsoft to work on Kubernetes on Azure. He'd been working at Google for 8 years already. Microsoft probably paid a lot to acquire him and it wouldn't be surprising if he saw it as an exciting opportunity to keep working on his creation while leading an exciting new project.
Joe Beda and Craig McLuckie left together to found a company whose product _is_ Kubernetes (Heptio).
So I could guess they just realized that their resume opened them to exciting new opportunities.
> Have you ever pondered the strategy behind it all?
Yes, and I can't see what it could be, so I'd love to be enlightened.
So far all I see is that I'm using an amazing software that has considerably simplified my life when dealing with my infrastructure. I so happen to be using GKE (and maybe _that_ was the grand evil plan all along!), but I know that I could use AWS or Azure or even go back to bare metal if I wanted to.
I think it's misleading to conflate questions whether Google's k8s engineers can use shortcuts through the organizational fabric around k8s with the project's source licensing.
k8s being open source merely means that the code is open and for everybody to use under clear and rather generous (even if potentially reciprocal) terms that are OSD compatible.
In particular "open source" doesn't mean that everybody gets to have a say on anything or everything: those with commit rights always reign supreme.
As for the part about procedural rules and their (ab)use like the ability to take shortcuts because everybody outside the cabal happens to be absent for whatever reason: that could have happened just as well in any body under pay-to-play rules that covers "software standards" (or anything, really), eg. OSF, X/OPEN, UEFI Forum, ... You don't don't get to hear about the drama.
But that's still different from the day-to-day goings-on being discussed here. The fate of Kustomize is not the stuff grand strategy. My argument is addressed to that level.
It is somewhat hard to follow the actual information though, and non-native speakers will probably struggle quite a bit.
Thanks for speaking out for us!
During the Kubernetes Contributor Summit at KubeCon Seattle last year, I made lots of noise about using KEPs, and about needing non-technical contributors to help those of us who are "organizationally challenged". In what I hope was a response, people showed up at SIG PM.
See https://youtu.be/_7IIzH_4yUk?t=998, https://youtu.be/mwG2CzdCg_8?t=1389, and https://youtu.be/mwG2CzdCg_8?t=1893 if you want to see me rambling about it on camera
I think this conversation was really kicked back up in earnest as a result of Windows node support not landing in 1.13 due to a lack of clarity on what the bar was for release. There were other enhancements that could have had a smoother landings in this and previous releases, but this was definitely when we realized we needed to overhaul and document this part of the project.
As in, my chart contains 3 services , I remove one of them including some related things like secrets, configmaps, etc and deploy, helm will clean up everything redundant.
Does Kustomize handle this problem?
This is very difficult to write though, needs to go through a huge pile of history. Just my selfindulged request.
From technical point of view, I really like the composition-based/declarative approach of Kustomize.
I am testing different type of bigdata clusters in k8s (Hadoop, Hdfs, Spark, HBase, Hive, Ozone, etc.). I need a method to easily create/install different type of clusters with a set of predefined components. Sometimes I need Spark, sometimes not. Sometimes I need HDFS HA sometimes not. With or without kerberos.
With Helm chart it's almost impossible: all the charts would be full with different conditional branches and which hard to be maintained (and hard to handle dependencies: for example if both hdfs/yarn cluster would like to add something to the core-site.xml configuration).
The composition based approach works better for my use-case, but I found Kustomize very difficult to use. It has strict opinion/limitation how the thing should be defined/worked [1], which are very smart, but sometimes not for my use-case. Also it requires very verbose configuration.
Finally I just created my tool which is more flexible (shameless plug, it's here: [2]) which can do anything with the help of composition and more easy to use.
I respect the Kustomize developers and development and I am very happy that it exists. (And I know that it's way more mature/smart than my tiny tool)
I am just not sure if it it's possible to cover all the use-cases (or majority of the use-cases) with one existing tool (neither with Helm nor with Kustomize). Therefore I would prefer to keep all these tools in separated repo. Personally I would prefer to have more similar tools with new ideas and prefer to have better and better (and more flexible) Kustomize.
But this is just my personal preference.
[1]: https://github.com/kubernetes-sigs/kustomize/blob/master/doc...
And I would answer: Why are you surprised? I worked a bit with Kubernetes and it's mostly dominated by Google or by a top-level set of contributors that are all friends and think alike. This is also probably the reason why Google released Kubernetes as open-source. It allows them to indirectly control this project.