Hacker News Reader

Comment by conradk | Hacker News Reader

How does Keights compare to Kops ? https://github.com/kubernetes/kops

Where Kops is a command line tool with subcommands like create-cluster and update-cluster, Keights uses Ansible roles with CloudFormation underneath. This works well to manage the cluster from CI/CD, where updating the cluster is just updating the version in requirements.yml and retriggering the build with Ansible. Creating or updating the cluster is the same process.

Keights also relies on kubeadm, which is released along with Kubernetes, so it can be up to date much quicker than Kops, which is several releases behind. Kops is a surprisingly large amount of code and probably needs significant changes between each release, whereas changing Keights to support a new release usually just means modifying the Kubeadm config files and changing dependency versions.

At the last two companies where I worked, they couldn't use Kops because it checked for an internet gateway on the VPC and failed if one was not found. A lot of companies have centrally controlled Amazon accounts with locked down VPCs and no ability for most teams to modify them. Keights was created to work even in such environments and assumes that your network is already set up how you want it.

Thanks for explaining.

It sounds like Keights is the same as Kubespray then, isn't it ? Kubespray also uses kubeadm underneath, as far as I'm aware.

They both use kubeadm, but the design is very different. Kubespray runs Ansible against the hosts in the cluster, whereas Keights uses Ansible to build CloudFormation stacks, and all the instances then bootstrap themselves with kubeadm.

Thanks for the input on kops. There's a lot of functionality in kops - e.g. we can manage CNI providers, etcd etc. In some places this manifests as more code, but my gut is that if you add up all the code you're relying on it turns out to be a wash. And you can definitely use kops from CI, driving it from yaml files that are checked in to git - that seems to be a great configuration particularly for people managing large numbers of clusters.

The long-term strategy is to get most of the kops functionality upstreamed into standalone community projects - and we're making progress with etcdadm, addon-operators, cluster-api etc. Then it will be easy to write your own tooling if you don't like some of the kops decisions, but still benefit from the community investment in e.g. etcd management etc. kops itself becomes a thinner shim around those shared common pieces. A lot of the decisions that are now generally agreed (e.g. dynamically attaching etcd volumes) weren't as well accepted when we started off, so it was harder to get them going as community efforts!

We do have support for "phases" in kops which should allow you to use a provided VPC, but to be honest it's still not as easy as the rest of kops is. We also have a few PRs in-flight that to allow you to specify an alternative to an IGW e.g. a VPN, but it's hard to reach consensus (but I guess we should based on your input!). The big trade-off here is that once you start allowing arbitrary configuration, you lose the ability to validate things, and so for some fraction of people there are going to be mistakes. That works great for small community projects, it is really great if your business model is paid support, but for a large community project it really can be problematic. I don't think we've got the balance totally correct in kops, but that's the trade-off we wrestle with.

Hi Justin, thanks for the feedback. I'll take another look at kops as it's been a while. I do think it could benefit from having a knob to turn off some of the pre-validation, or for it to do functional checks instead of checking for the existence of specific AWS resources. Many companies treat their AWS VPCs as an extension of the private corporate network, and they come up with lots of creative ways to route traffic to the internet (or select portions of it) without the VPC necessarily having an IGW or VPN. AWS is also making a growing number of services available from private endpoints within the VPC, so there is often no need for internet access. I understand the value of validating the environment, but an "expert mode" toggle would also be nice.

Kops is (intentionally) very out-of-date. This is a blog post about K8s 1.14 being released, but the latest stable Kops is still based on K8s 1.11.x.

Ah yes, OK. What about Kubespray ? https://github.com/kubernetes-sigs/kubespray

Kubespray SSH:s in and does what you'd do manually, while Kops deploys a prebuilt image. That means Kubespray is pretty much cloud-agnostic, and much easier to customize, but it's also not really compatible with auto-scaling groups and the like.

I don't think they intend to be as out-of-date as they are. They are supposed to be around a version behind. So they should really be releasing 1.13 soon rather than 1.12.

They have had a few big tech changes in the 1.9/1.10/1.11 timeline so hopefully they will be able to catchup a little.

Yes, the reality is probably somewhere in between. We've heard that users like the idea that kops releases are more ready, where the .0 releases of k8s might have more issues particularly with ecosystem components. So we do lag a little behind - ideally one release.

1.12 has been a particularly tricky release getting everyone from etcd2 -> etcd3, but we've finally turned the corner on that one, so that should now let us catch up a bit.

Finally, we've also heard that users want more of a choice, so we're going to start doing e.g. 1.13-alpha and 1.14 alphas much sooner. We'll still wait to do kops 1.14.0 until everything is ready, but for users that want to run k8s 1.14 sooner, they will have an option that isn't building from source. And hopefully this also gets more people using pre-release versions of kops (in non-production environments) and also helps stabilize the releases more rapidly.

Reply on news.ycombinator.com