Kubernetes 1.14 released
kubernetes.io
kubernetes.io
I'm excited to see that the kubectl docs[2] are actually recommending -k as the default solution (vs -f).
Though Apply can be run directly against Resource Config
files or directories using -f, it is recommended to run
Apply against a kustomization.yaml using -k. The
kustomization.yaml allows users to define configuration
that cuts across many Resources (e.g. namespace).
[2]https://kubectl.docs.kubernetes.io/pages/app_management/appl...
I’m yet to deploy anything to a production k8s cluster, so I’m looking for opinions of people that know what they’re talking about.
It also lies about the actual status of the installation. A deployment that applies successfully but has issues with the new pods coming up can still show successful.
The larger charts are a nightmare which isn’t helm’s fault but it doesn’t make things appreciably easier either.
Kustomize has a lot of warts too. If the type definition in k8s source doesn’t have a merge key defined, it’s going to overwrite the entire section of the resource. You can use JsonPatches to get around that problem but that is super gross too.
The entire ecosystem is trying to solve the same problem and no one has come up with the definitive way of doing it (because it’s a HARD problem) It seems like Helm templating and then passing to kustomize to add in the one-off changes is the direction a lot of people are heading but it’s more of a least-bad solution than a good one.
I will say, if you’re using helm templates for your own stuff, you very likely going to be happy with it.
You can define a component which you install into many clusters and then slightly differentiate them based on cluster parameters, kind of like Puppet or Chef (without the application stage)
Alongside this, you can actually patch helm charts. An example component can be found here https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master...
The patches.jsonnet allows you to add a commandline flag that wasn't in the helm chart at one time.
Seriously though, gonna look closer at this tomorrow at the office. Thanks for the heads up!
I use helm fetch and pull the charts with dependencies into my repo. Running direct from the public repo is a little bit too much like piping the contents of a url to bash without checking first.
You can then customize however you like.
When I want to update the charts, helm fetch it again and merge the changes.
(Title = Customizing Upstream Helm Charts with Kustomize)
If you want to: have PITA with RBAC, forget about manual changes because there are no 3-way-merge in Helm2 (that's what kubectl apply does, only hope it will be in Helm 3) and after this change helm cannot apply desired state, some of the biggest pains lasting for years had been resolved only in 2.13 with --atomic parameter.
Aside from dependency resolution, everything that Helm does could be made by any template engine + API caller.
Unfortunately, many of those stable charts for core infrastructure are barely POC-worthy and take a great deal of effort to get into production state. For example, there is no helm chart for Kafka with TLS support. Maybe not necessary for every production implementation, but dammit, this is 2019.
That said, helm is not safe. A helm "release" is not an artifact. It's a deployment of a helm chart of a certain version. That helm chart version may or may not correlate to the underlying version of the software you're running as that artifact is presumably defined with an "image" in values.
For example, I've run the same aforementioned MSSQL helm chart with 2017-CU7-ubuntu, 2017-CU8-ubuntu, and 2017-CU9-ubuntu tags. A quick "helm list" might show you the same chart version across three different environments but they're still very different.
Helm is not idempotent. Helm upgrade is a nightmare. One has to perform a helm upgrade to set values. Helm upgrades often fail. Even though they report failure, a look under the hood reveals that it actually succeeded.
Occasionally a helm upgrade blows away the persistent volume attached to a stateful thing. That's always the best.
tl;dr helm is the most popular, but has some massive shortcomings so we're all looking for something to replace it and kustomize is exciting because it's now for better or worse part of the core.
I am, however, not sure what kubecfg's future is: ksonnet (its way more complicated cousin) is being deprecated.
Heptio (the owners of ksonnet) were bought by VMWare and will no longer maintain it. But it's possible the CoreOS guys will take ownership since they use it heavily in some of their tools.
Kustomize is purely hierarchical templates.
- kosko
- pulumi
Big kudos to SIG Storage!
There was talk earlier about using LVM to be able to really provide a Local PV that is like the network attached PV's, but it doesn't look like that landed yet
As a sole developer, I couldn't run what I do WITHOUT such an orchestration platform.
And yes, I administer my own cluster on a bunch of Vultr VMs. I've had fewer problems with this over the last 3 years than it seems people have had on GCS (recent news of outage fresh in memory).
Eventually I might want to add more machines to the cluster. That would take me a few minutes. I use sup scripts to do the setup. I like it much better than Ansible.
Also, shameless plug?
Keights also relies on kubeadm, which is released along with Kubernetes, so it can be up to date much quicker than Kops, which is several releases behind. Kops is a surprisingly large amount of code and probably needs significant changes between each release, whereas changing Keights to support a new release usually just means modifying the Kubeadm config files and changing dependency versions.
At the last two companies where I worked, they couldn't use Kops because it checked for an internet gateway on the VPC and failed if one was not found. A lot of companies have centrally controlled Amazon accounts with locked down VPCs and no ability for most teams to modify them. Keights was created to work even in such environments and assumes that your network is already set up how you want it.
The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts!
We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.
It sounds like Keights is the same as Kubespray then, isn't it ? Kubespray also uses kubeadm underneath, as far as I'm aware.
They have had a few big tech changes in the 1.9/1.10/1.11 timeline so hopefully they will be able to catchup a little.
1.12 has been a particularly tricky release getting everyone from etcd2 -> etcd3, but we've finally turned the corner on that one, so that should now let us catch up a bit.
Finally, we've also heard that users want more of a choice, so we're going to start doing e.g. 1.13-alpha and 1.14 alphas much sooner. We'll still wait to do kops 1.14.0 until everything is ready, but for users that want to run k8s 1.14 sooner, they will have an option that isn't building from source. And hopefully this also gets more people using pre-release versions of kops (in non-production environments) and also helps stabilize the releases more rapidly.