Has anyone already successfully created a HA-cluster on a bare metal infrastructure?
It seems overly complicated and not that well documented from what I can see.
The biggest complication is around Ingress and load balancers. https://github.com/kubernetes/ingress/issues/23
https://github.com/kubernetes/ingress/issues/17
The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset? How do you set up ingresses to pass through source ip.
a lot of this is taken care for the cloud...but NOT for bare metal.
[1] https://groups.google.com/forum/m/?utm_medium=email&utm_sour...
https://github.com/kubernetes/kubernetes/issues/27343
I was writing a blog about installing it on bare metal, but this issue got me blocked.
Though honestly i think it is a fundamental issue and not specifically related to BM.
Disclaimer: I'm on the team that works on this.
It is a lot easier on the public cloud, or easier still on a managed service.
You're absolutely right that it takes some effort.
on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same.
And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo.
For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months.
[1] https://github.com/kubernetes-incubator/bootkube
[2] http://kubernetes.io/docs/getting-started-guides/kubeadm/
In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.
Please join up at the SIG to help out! https://github.com/orgs/kubernetes/teams/sig-cluster-lifecyc...
Disclosure: I work at Google on Kubernetes.
Achieving HA isn't necessarily complicated. Etcd supports clustering by default and the master components have master election builtin.
We also use haproxy locally on every machine to loadbalance between the different api servers. So we don't need a central LB.
Setting up HA was only a small part of our overall effort. Making things robust and figuring out the small details are a lot harder.
FWIW, HA is complicated in general, K8s HA is actually reasonable by comparison.
Other tasks, like upgrade and security are harder. We're working on docs in the SIG Cluster Ops and you're welcome to join in discuss with us there.
There will be a 1.5 compatible release coming soon.
Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.
They support bare metal providers, like Packet, but they also support inexpensive platforms, like Digital Ocean.
Really handy if your goal to experiment with an up and running cluster and don't want the hassle of installation.