Kubernetes clusters for the hobbyist
github.com
github.com
etcd stores a load of sensitive cluster information, so unauthorised access to it is a bad thing.
There's an assumption in the guide that you have a "secure network" and therefore don't have to worry about etcd authentication/encryption. The thing is if you have a compromised container (say) and that container, which has an in-cluster IP address can see your etcd server, then it can easily dump the etcd database and get access to the information held in it...
Personally I'd recommend setting up a small CA for etcd and using it's authentication features, there's a good guide to this on the CoreOS site https://coreos.com/etcd/docs/latest/op-guide/security.html
Related issue on GitHub: https://github.com/hobby-kube/guide/issues/6
One of the challenges of K8s security is that there are a wide variety of distributions, each with their own defaults and configuration, so it's hard to make blanket statements about their security.
I've definitely seen some with this issue out of the box.
> Choosing a cloud provider
This really annoys me about Kubernetes. Essentially all the official documentation is about how to select a cloud and let a cloud-specific tool magically do everything for you. There's no procedure for setting up a single host for development purposes or to have a Dokku-like personal PaaS.
This guide is super useful because it avoids all the magic and lets you set things up properly (despite assuming you're doing it on a cloud) and potentially even do it on a single host.
It's not worth bothering to make a single machine setup.
The same can be said of Heroku, yet Dokku exists. The same can be said of Docker, yet plenty of hobbyists use it for their projects.
> It's not worth bothering to make a single machine setup.
Sure it is. Say you're a web developer and you use MySQL, PHP and Apache on Kubernetes in prod. Why would you go to the effort of setting those up individually on your machine when you could just put your prod manifest on Kubernetes on your workstation?
There's nothing about Kubernetes (aside from poor documentation) that prevents it from being used on a single host. Actually, according to its own documentation (https://kubernetes.io/docs/getting-started-guides/scratch/), it consists of:
- Docker/Rkt
- Kubelet
- Kube-proxy
- Etcd
- Apiserver
- Controller
- Scheduler
The bottom 6 are actually written in Go and can be run from a single binary.
There's no reason Kubernetes should be hard to set up. There's no reason it can't run on a single node. There's no reason you need a cloud. There's no reason you need a "supported" operating system.
A basic LAMP stack doesn't need Kubernetes to operate. It's up to you to justify the effort of setting up 7 super complicated and useless software.
I've grown quite a thick skin since I exposed the first project of mine to a wider audience. But still, it's feedback like yours that keeps me going.
I felt exactly like this, but then found one.
https://kubernetes.io/docs/getting-started-guides/kubeadm/
It requires manual kubelet setup (usually there are OS packages for that - which is the proper way to install software on the host system), and then it generates a CA, node certificate, spins up etcd and flanneld containers and sets up k8s on top of those. It also lets one join that cluster in a semi-automatic manner, making the whole setup no harder than Docker Swarm.
I think it's a reasonably nice middle point between unexplainable cloud magic and do-it-all-yourself-the-hard-way setups.
Can you clarify what you mean by this? Since the beginning (pre-1.0) it's been possible to stand up a cluster on bare metal/VMs using a single command. This used to be done with the `kube-up.sh`[1] tool, which has been replaced with kubeadm in 1.6 [1].
If anything, I thought the fair complaint would be that there are too many ways to set up a cluster, and it's confusing to figure out which is the right one [2].
[1]: https://kubernetes.io/docs/getting-started-guides/kubeadm/ [2]: https://kubernetes.io/docs/setup/pick-right-solution/#table-...
I think I eventually got something set up using the manual documentation but at the time said documentation resulted in a completely insecure installation, entirely unauthenticated. Since I don't have (or want) a "private" network and the docs didn't tell me how to get security any other way, I just gave up on Kubernetes.
And really, it's incredibly frustrating how inflexible the docs and setup tools are. As detailed in another comment (https://news.ycombinator.com/item?id=14296176), Kubernetes really just consists of Docker and 6 Go binaries. There's no reason setup should be so difficult and certainly no reason why it should be OS dependent.
There are actually a lot of tedious configuration parameters that need to be tuned per-distro, which is why there's only now a candidate for a unified solution; it's harder than you might think to create a truly generic tool for this job. (Take a look at the kube-up saltstack files if you want to be horrified by some surprising complexity).
Nowadays, it looks like there are a couple options on Fedora, specifically
https://kubernetes.io/docs/getting-started-guides/fedora/fed... https://kubernetes.io/docs/getting-started-guides/fedora/fla...
If you're looking for something that "Just Works (tm)", the ansible config (above) would be my bet for where to start these days.
I'm just asking because I was born in the 90's, I'm not very aware of the state of email server management then.
Usually in my day job I have to work with these tools, and currently have a stack of 5, that mostly have incomplete documentation, zero explanation of how they actually solve the problem and nearly zero ability to debug (e.g. what value have kubernetes logs and events. Usually when you have a problem is when you have no kubernetes logs yet/anymore and the events only tell you what you already know). Now I'll probably need to learn another one, considering these three options for storage.
At the weekend, where I'm mostly trying to relax and have physically only 2/7th of the time to alot I learn the basics behind containerization, e.g. namespaces, cgroups, virtual network adapters, iptables. And I feel in this small time slot I make a lot more progress at getting to that end-to-end solution that people actually need.
The example I'm using is wordpress+mysql. It's a simple thing that covers >70% of what anybody wants to deploy. And on-premise with kubernetes+docker it's still not possible without hacks (e.g. for volumeclaims and logging), after, I don't know, 4 years of Kubernetes? I bet in spending 2 years worth of weekends any normal person can come up with something better.
---
Re missing features, examples for Kubernetes:
A) Why does Kubernetes not solve the networking part? If I have a cluster and containers that may run in different places, of course the tooling I use to maintain that cluster needs to ensure that containers can talk to each other. There can be an abstract API and the option for other people to write plugins, but the core needs to come with one solution, that mostly works and is debuggable when not.
B) CrashLoopBackoff. Why did no Kubernetes developer get the idea that this state may require any kind of log/debugging?
C) Why does kubernetes assign random ports to services and not provide a simple way to retrieve them? Of course I can get them after I've learned the JSON api. But usually that is not considered a solution but hacking. I really don't care what the service is running on, I just want to use it.
D) Why do I need to manually say how many containers should run for each service? There are very distinct options a user may need. E.g., most services should run 1 instance and replace it if it dies. If it needs reliability I want to define how reliable it should be and accordingly the cluster should decide if it needs 2, 3 or 5 instances. And lastly I want to run stuff on all my nodes. Each node with a container. That is not even possible afaik.
E) Most kubernetes tools are not working well with environment proxies. For instance kube-proxy shell tool will completely bug out. But surprise, clusters make a lot of sense in enterprise environments, and enterprise environments have proxies. It's also not a hard setup for testing. A raspberry pie can be your home networks proxy.
F) on premise storage solutions, considering that a restarted container may not run on the same host.
G) Since not much is really running right now we didn't run into that problem yet. But it's totally possible that the whole cluster runs out of resources. I haven't seen any piece of info about the general cluster status and when I need to increase or replace infrastructure.
Honestly if I don't have all these things solved, am I really better of using Kubernetes or writing my own scripts? I think it currently is about equal in effort. And if that's the case my own solution has the huge advantage of being under my control and allowing me to learn a lot of new things.
How are you setting up your cluster?
A) Kubernetes abstracts the networks solution so people can choose between multiple implementations. Some people already use openstack and want to use their openvswitch network for their containers others want to use a pure container network like flannel. Kubernetes distributions usually come with a integrated network solution.
B) You can still retrieve the logs from a container is CrashLoopBackOffState. You can even retrieve the logs from the previously failed container by using the `kubectl logs <container> --previous`. Applications can write information about their failure to /dev/termination-log which can be used to debug the failure.
C) You can define the port yourself otherwise kubernetes defaults to a random port to avoid port conflicts. The recommended way to expose HTTP services would be by using ingress.
D) You can run an instance on every node by using a daemonset.
E) Are you talking about outbound traffic from your containers to an external system? You would need to configure this in the container engine and the container itself. I had little problem doing this in an enterprise environment that required an http proxy for all external communication.
F) You can attach you existing SAN solution over iSCSI, fibre channel or even NFS. Another solution would be to run a distribution storage like ceph or glusterfs in kubernetes for kubernetes. You then provision persistent volumes that are attached to the node your pod is running on. If your pod is rescheduled the volume will be moved too.
G) If you have resource requests/limits set kubernetes will not schedule your pods if you have no resources available.
Re proxy: Not only that, but also. Let's say you have single instance deployment, and run kube-dashboard. You want to access the dashboard on localhost:8080 so you start the kube-proxy shell command. What actually happens is that the go code underneath kube-proxy will redirect the request through your $http_proxy, even if localhost, the dashboard's internal address, and your hosts external address are in the $no_proxy environment. And if your network proxy doesn't allow the dashboard's port you get a HTTP 403 instead of the dashboard.
Re storage: We have a cluster with 10+ nodes, each with 5+ disks a 10 TB. How would you make sure that your software talks to the correct disk, after it gets restarted and may end up on another node?
Re limits: The highest goal is not avoiding ressource exhaustion, the highest goal is service continuity, though. It can all run on 100% and die. no problem. I just need to know that I should exchange the first few disks/processors/machines before the customer facing service slows down.
@storage If you want to move your disks with your containers you need an additional storage system(SAN, cloud provider or distributed storage in kubernetes). Then you create persistence volumes in kubernetes that references a disk from the storage provider. This allows you to assign this disk to a pod with a persistence volume claim. The disk that is linked to the container through persistent volume and persistent volume claim will be moved on the same node the container is scheduled. If you want to run stateful workloads on kubernetes I would advice you to use a storage system. You can use local disks but you loose some of the flexibility that you gain from kubernetes by tying your containers to certain nodes with the data. There is also work being done on improving the handling of local storage to treat it more like a resource and introduce separation by using local volume per kubernetes local disk volumes.
But it'll be alpha in 1.7: https://github.com/kubernetes/features/issues/121
Deis aimed (aims?) to be the lightweight version, but then got acquired by MS.
And the installation was just as confusing it seems.
Or even better just use GKE for development / learning purposes. Just stop the cluster when you're not using it, and it'll be a lot cheaper than something you won't want to take down because you spent days installing it.
Personally I've used Stackpoint.io to provision some small clusters but I was very excited to see this project because deploying my own cluster from scratch is next on my todo list. Kelsey Hightower's "Kubernetes the Hard Way"[1] is the canonical go-to reference here but it's also very daunting so this looks like a great middle ground.
Let's face it, even today the k8s docs can be quite sparse sometimes or gloss over the details, so knowing how all of the pieces work from the ground up can be a big help. Plus, you prevent vendor lock-in when whatever automated tool you're using doesn't solve your use-case or decides to start charging a lot of money.
[1] https://github.com/kelseyhightower/kubernetes-the-hard-way
With all that being said, kops is an incredible tool (as are others) and we used it to learn about the system and test some of the functionality for ourselves. Can't recommend it enough.
In case you missed it, there's a repository[1] in the same org which offers fully automated provisioning using Terraform.
Please add a disclaimer on the top pointing beginners at some of the better resources for them, and then tell them to come back once they've learned what stuff like "ingress" means.
Then it does the installation and gives you urls for the control panel.
I don't think I'd use it for production, as everything is behind stackpoint.io urls, but for experimenting, it saves a ton of time.
https://stackpoint.io/#/pricing
They haven't announced a date for when they'll start charging, and I'm not sure how that will affect existing clusters.
What's wrong with paying for something as a hobby?
Many people don't have $50/mo to spend on unprofitable hobby projects. There's nothing wrong with paying for hobbies, but most people's hobbies fall under (a) infrequent large purchases or (b) really cheap occasional costs.
Upgrading my computer falls under (a). It's expensive but I only do it once every 3-4 years. Buying new guitar strings for my guitar every so often falls under (b). It's cheap and while I should do it every-other-month I do it maybe twice a year.
If the cost of new guitar strings was equivalent to the cost of upgrading my computer - I'd give up playing guitar as the hobby would become too expensive.
We open source our Kube bootstrap toolkit (which uses kops). No one should build this by hand. It would be like, I dunno, setting up a MariaDB Galera Cluster by hand when you intend to use RDS (not a perfect example, please no pedantry!). It's a fun learning experience, I guess - if you're into that sorta thing - but is not what you want to use in any production context.
https://www.reactiveops.com/blog/kops-102-an-inside-look-at-...
https://www.reactiveops.com/blog/using-k8s-scripts/
Use the automation to bootup clusters, please!
Like learning shortcuts in math, you only "earn" automation once you've done things the hard way often enough to know how it works at a low level. That knowledge is critical. Otherwise you're just cargo-culting something you don't understand.
I'm curious, why no mention of AWS as a provider for roll-your-own? Is this a cost thing?
Also, I get the feeling that Ubuntu is _not_ a first class citizen of the k8s ecosystem, but perhaps my newness to the ecosystem is to blame here. The Saltbase installer, for example, only supports Debian and RHEL distros, `kops` prefers Debian, and the documentation for cluster deployments on kubernetes.io and elsewhere also seems to be somewhat suggestive of Debian and Core OS. Perhaps thats just a mistaken interpretation on my part. I'm curious what other peoples thoughts on this topic are!
The front page of https://kubernetes.io/docs/ has a bullet that links to a super simple way to deploy Kubernetes to Ubuntu on any of [localhost, baremetal cluster, public cloud, private cloud]!
See:
* Installing Kubernetes on Ubuntu: Deploy a Kubernetes cluster on-premise, baremetal, cloud providers, or localhost with Charms and conjure-up.
> There are plenty of ways to set up a Kubernetes cluster from scratch. At this point however, we settle on kubeadm. This dramatically simplifies the setup process by automating the creation of certificates, services and configuration files.
1) Cost. In VPSs like Digital Ocean/Scaleway, there's usually a large network out transfer quota included in the price, which just isn't there with GCP, where you pay for network usage on a metered basis.
2) Learning. Although you can defer most of the heavy work to GKE, it's still good to understand the moving parts so you can make better choices as you grow.