Is anyone here running k8s in production with kops? Are there any missing pieces that require "manual" work, like rotating certificates? How feasible is it to run, say, 30 clusters out of the box with a 2-person team?
But: we don't currently support rotating certificates. There used to be a bug in kubernetes which made "live" certificate rotation impossible, but that bug has now been fixed so it's probably time to revisit it. We create 10 year CA certificates, so it isn't something that you have to do other than just good security practice though.
If you file an issue (https://github.com/kubernetes/kops/issues) for certificate rotation and any other gaps / questions we'll get to them!
kubeadm is intended to be a building block that any installation tool can leverage, rather than each building the same low-level functionality. It isn't primarily meant for end-users, unless you want to build your own installation tool.
We want to accommodate everyone in kops, but there is a trade-off between making things easy vs. being entirely flexible, so there will always be people who can't use kops. You should absolutely use kubeadm if you're building your own installation tool - whether you're sharing it with the world or just within your company. luxas (the primary kubeadm author) does an amazing job.
How do you handle that DNS in a corp net can get weird and for instance in Ubuntu 16.04 the NetworkManager setting for dnsmasq needs to be deactivated?
How do you report dying nodes due to kernel version and docker version not being similar?
Do you report why pods are pending?
Does kops wait until a sucessful health check before it reports a successful deployment (in contrast to helm which reports success when the docker image isn't even finished pulling)?
Do you run any metrics on the cluster to see if everything is working fine?
Edit: Sorry to disturb the kops marketing effort, but some people still hope for a real, enterprise ready solution for k8s instead of just another fluff added on a shaky foundation.
* no_proxy - kops is getting support for servers that use http_proxy, but I think your issue is a client issue with kubectl proxy and it looks like it is being investigated in #45956. I retagged (what I think are) the right folks.
* DNS, docker version/kernel version: if you let kops it'll configure the AMI / kernel, docker, DNS, sysctls, everything. So in that scenario everything should just work, because kops controls everything. Obviously things can still go wrong, but I'm much more able to support or diagnose problems with a kops configuration where most things are set correctly, than a general scenario.
* why pods are pending: `kubectl describe pod` shows you why. Your "preferred alerting system" could be more proactive though.
* metrics are probably best handled by a monitoring system, and you should install your preferred system after kops installs the cluster. We try to only install things in kops that are required to get to the kubectl "boot prompt". Lots of options here: prometheus, sysdig, datadog, weave scope, newrelic etc.
* does kops wait for readiness: actually not by default - and this does cause problems. For example, if you hit your AWS instance quota, your kops cluster will silenty never come up. Similarly if your chosen instance type isn't available in your AZ. We have a fix for the latter and are working on the former. We have `kops validate` which will wait, but it's still too hard when something goes wrong - definitely room for improvement here.
In general though - where there are things you think we could do better, do open an issue on kops (or kubernetes if it's more of a kubernetes issue)!
For instance: "Yes, we can. We considered most of that and also have some enterprise customers with similar setups. Check out "googleterm A", "googleterm B", "googleterm C". If you don't find all of that join our slack chat to get more details."
And a more likely answer, also single line: "WTF are these questions? We thought docker+k8s already solves that." (I would've also expected solutions from there but don't hope for it anymore.)
PS (actually an edit to the previous post, but it's already too old): For instance Openshift, as I just found, considers the docker-version kernel-version problem via "xxx-excluder" meta packages: https://docs.openshift.com/container-platform/3.4/install_co...
A step in the right direction!
We have run into some issues with kops. Customizing the Kubernetes executables, e.g. using a particular Docker version or storage driver, has been buggy pre-1.5. Upgrading clusters to later Kubernetes versions has left some of the kube-system services, like kube-dns, in a weird state. Occasionally we encounter issues with pods failing to schedule/volumes failing to mount - these are fixed by either restarting the Kubernetes (systemd) services on the problem nodes or by reprovisioning nodes entirely. On one occasion, a bad kops cluster update left our networking in an unrecoverable state (and our cluster inaccessible).
I don't think there are any missing pieces, the initial configuration is what usually takes the most time to set up. You'll have to become familiar with the kops source as not everything is documented. As far as running 30 clusters with a 2-person team, it's definitely feasible, just complicated when you're constantly switching between clusters.
And it is way too hard to switch clusters with kubectl, I agree. I tend to use separate kubeconfig files, and use `export KUBECONFIG=<path>`, but I do hope we can find something better!
We're at three live and two dead (decommissioned) clusters with a two man team, and while we regret some decisions, most of the time it just works.
I don't know if AWS has the disclaimer up anymore, but the default limit is 50 with limit increases available to 100 with "no guarantee that performance will remain unaffected"... or something like that.
What network type are you using, out of curiosity?
I agree the basic networking has a lot of limitations. Compared with added more layers with networking, I'd rather have a simpler setup with fewer nodes, even if they are larger.
We are slowly open sourcing some of that and more here:
http://blog.kubernetes.io/2017/06/kubernetes-1.7-security-ha...
We open sourced our kube generator code, called pentagon, which uses kops, terraform https://github.com/reactiveops/pentagon
It seems like a lot of the work like this just "isn't there yet" when it comes to orchestrators like Tectonic or Stackpoint, or Kops, in making this easy for you. (So there's surely a market for people who know how to do this stuff, but it seems like this would be the first feature that every tool supporting AWS would want to have. Unless there are hidden gotchas, and it seems like there would be a lot of blog posts about it if that were the case.)
Maybe kops should use ALBs for the apiserver, and maybe k8s should support ALB for Services of Type=LoadBalancer. Neither of those are supported at the moment, if they should be then open an issue to discuss. (Even if you're going to contribute support yourself, which is always super-appreciated, it's best to start with an issue to discuss!)
Thought it worth mentioning though, as the older ELB+k8s isn't great, and because the ALB support hasn't shaken out yet, a cluster created with kops could be suboptimal unless you address it afterwards.
I assume once it all shakes out, kops would support whatever the direction is.