Kubernetes the Hard Way
github.com
github.com
There is a great need of good source of production setups. In open source software this seems to be the secret that no one is willing to reveal. I tried to setup a kubernetes cluster from scratch a while ago and soon I was browsing the source code for answers. Openstack is the same, you need to understand a lot about the inner workings before you can even attempt to setup something for production.
There is always a simple "this shell script starts your own <name your tech here> cluster in vagrant" but it is still not a production setup.
And even if this article is the "hard way" it describes:
> This is being done for ease of use. In production you should strongly consider generating individual TLS certificates for each component."
And it does not mention that the crucial part is the common name field in the certificate maps to the user name that is the magic information that I once needed.
I sincerely appreciate this article but production setup is still a long learning experience away.
Kubeup is designed to run once, unlike systems such as Puppet and Terraform that declaratively set up the world to fit your specification.
Kubeup also does a lot of mysterious stuff. By using it, you don't have a clear idea of which pieces have been set up and how they slot into each other. It is, in short, opaque and magical.
For comparison, I set up Kubernetes with Salt on AWS. It was, by all means, "the hard way", and took me a few days to get running and a couple of weeks to run completely correctly (a lot of stuff, like kubeconfig and TLS behaviour, is still undocumented), but as a byproduct I now have the entire setup in a reproducible, self-documenting, version-controlled config.
However, now that I have a cluster up and running, I can take the time to build a parallel cluster with more understanding, and migrate the services to it.
I found a terraform example, but it declares itself out of date, and looked more complicated than I thought it should be... that was just a gut feel though.
I have not used Salt, but I always like to learn new things, especially if they make my life easier.
I'd be very interested in checkout out your setup, and any lessons learned you have to share.
Thanks in advance
I just need to generalize it a little bit. Email me and I will send you a link once it's done, sometime next week (I'm on vacation).
I've been playing with https://github.com/kz8s/tack lately, but your implemention sounds like even more comprehensive.
Terraform + Ansible for K8S on CoreOS
I don't know what the future of Kubernetes setup is, exactly, but right now it's quite safe to settle on Salt, Puppet, Ansible or Terraform. I haven't used Terraform, so I don't know how suitable it is to OS-level setup (things that the aforementioned tools are good at), as opposed to orchestration.
There's no getting away from configuration management and software installation at SOME level of your stack, and setting up a substrate for Kubernetes is no exception.
Basically what's arguably the best publicly available Kubernetes setup in the world is hiding in that Salt codebase, and EVERY would-be Kubernetes admin should look at it before venturing on their own.
Official upcoming replacement for kube-up
Just look how google runs stuff! https://cloud.google.com/compute/docs/tutorials/setup-joomla
Also I was setting it up on coreos and baremetal servers. It should be possible to run pods in google container engine or similar very easily, but would there be any fun in that?
https://coreos.com/kubernetes/docs/latest/kubernetes-on-bare...
And the installation flow that builds on top of that for Tectonic:
[1] https://github.com/kubernetes/kops [2] https://github.com/coreos/coreos-kubernetes/tree/master/mult...
Lol
* Actually pretty much works for what's in scope..
* It's got some nice configuration options that are discoverable and not hidden away in envars...
* Some good prelim docs explaining how kubernetes is bootstrapped
* Cluster management seems to function properly
* Updating/upgrading
What's missing IMHO(from an AWS user's standpoint including kops and k8s):
* SUPER unapproachable codebase ATM for KOPS and friends
* More flexible cluster dns naming so we can leverage real wildcard certs accross dev environments
* Running kubernetes in private networks
* Passing in existing networks created through other tools(terraform, cloudformation, custom etc)
* Responsibility for stuff seems spread out across projects and is unclear which lies where(also leading to an unapproachable-ness for contributions)
* AWS controllers that don't seem to fully leverage the AWS API's (traffic balanced to all nodes and then proxy'd via kube proxy; no autoscale life cycle event hooks)
* Unclear situation on the status of ingress controllers; are they even in use now or is it all the old way?!
* No audit trails
* IAM roles for pods
* Stuff I'm probably missing
It's very frustrating TBH. On one hand AWS ECS has IAM roles for containers now, for the new Application Loadbalancer, and private subnet support. On the other hand they DON't have pet sets, automatic EBS volume mounting(WTF), a secrets store, configuration API, etc. Also frustrating is I feel the barrier to contribute is a too high ATM even though I have the skills necessary..
It's SO close though. If I can get private, existing subnet support I can probably start running auto provisioned clusters that are of use for some of our ancillary services in production. From there I might be able to help contribute to KOPS and AWS controllers. Right now it looks like there is just this one guy doing most of the work on AWS and KOPS; probably quite overloaded.
Running kubernetes in private networks: You could probably get private subnet support by - Deploying manually or deploying with a script, then changing things in AWS (route tables, public IP, etc) to be private, manually afterwards (both cumbersome but possible) - Using NodePort instead of LoadBalancer on any services
Its amazing how much actually goes on here. Firewall rules, certificates, Docker storage configuration, etc. Its definitely not something that you can just thrown in a VM and assume everything will work.
We've worked really hard to make it simple to stand up hadoop on clouds, containers, and metal (https://jujucharms.com/hadoop-processing/). Juju brings the modeling, Bigtop brings the core apps. Scaling, observing, and integrating are old news; HA is landing now; your post and others like it have put security on our -next radar.
Having read a few kdc/hdfs stories, i think i'm going to miss the days when dfs.permissions.enabled was good enough ;)
1. Documentation is not satisfying work, maybe because it is as absolute as code?
2. Contributing documentation doesn't get you as much recognition as code
3. If you set it up differently yourself there is no need to maintain a fork (unlike code changes)
4. For open source projects that are company backed there is a perverse incentive to keep the documentation vague if they only make money though support
I have no idea how effective they are, but it's something to consider when setting up a deployment from scratch.
- If it's too easy to set up, nobody will hire us to make it work.
- Implement a kickass setup dirt cheap for some big-name company, so we can claim they use it in production. Yeah we tweaked it so it bears little resemblance to the original product, and only fits an incredibly narrow use case, but nobody stands to benefit from blogging about that.
- Better ship with a configuration file that isn't production ready.
- Did I say one? Better have three configuration files, each duplicated in distribution-dependent directories (in some cases), needing manual sync between servers to prevent catastrophic data loss.
- Remember not to publish the program that checks for errors in configuration; half of our income would disappear.
- Benchmark with a configuration file that nobody would use in production, but looks really impressive when taken out of context.
- People want transactions, remember to claim support (and if you must, explain somewhere in the fine print that a transaction can only span a single operation on exactly one document, and btw. is precisely none of A, C, I or D).
- Somewhere on the front page, it should say how we can support petabytes of data (and it performs very well, as long as you write all your data in one batch, never modify it, keep it all in memory, and turn off persistence).
- Never give away answers online. Answer every question about configuration with "it depends".
- Don't release a new version without renaming a few configuration options. Be "backwards compatible" by ignoring unknown, obsolete and misspelled options.
* Extremely difficult to set up.
* Claims that half of Fortune 100 uses it (read: many are required to support it; the rest have one guy with a toy installation in some branch office).
* Consists of dozens of components, each with several-thousand lines config files (actually Python code) that must be kept in sync between all nodes (yet have node-specific data).
* Claims to be "modular", but have complex interdependencies between each of the components.
* Upgrading is not officially supported, but some companies will help you.
* Will break in mysterious ways, and require you to backport bugfixes since you're stuck on an unsupported version after a year.
* Have unhelpful error messages (e.g. throw Connection Refused exception when you're actually receiving an unexpected HTTP return code).
* Write documentation in a way that appears OK to new users, but vague enough to be useless for those who are looking for specific information.
I think if 15 years ago during the opensource vs close source wars you had mentioned that opensource projects would eventually decide that they wouldn't make any attempt at documenting the product, no one would create actual installers, and continuous development methodologies would leave 99% of opensource projects in a state of 0 testing before releases, it would have been a lot harder sell.
Part of this goes back to the early linux RPM/DEB decisions to refuse to follow in the footsteps of more traditional software installers and provide an interactive UI for configuration (see bottom about HPUX). Resulting in chef/puppet/etc. This has removed the onus on projects to dedicate resources to that portion of the project. I've worked at a few companies shipping commercial software and there were always either a team responsible for building an installer, or a part of the development cycle dedicated to it. There were also frequently technical writers (a job position that seems to be mostly gone). Now, with opensource/git its done when the code gets pushed to github. Forget any documentation more complex than a few docbook/etc comments scattered around the code base and a wiki so fragmented that going from one page to another switches the version of the code in question.
Its a pretty sad/depressing state, and sometimes I wonder how anything works anymore. Thank god for RH, which actually has testing labs, people to fix bugs, and strong opensource ethics about making sure the upstream/other distro's benefit from their work. But, then they go and behave like poo flinging CADT https://www.jwz.org/doc/cadt.html in other projects.
* See HPUX for a good example of how to do it on unix, without some of the problems windows installers frequently have. The packages have a standardized form description language that are picked up in a preinstall process, so the user can be prompted for install locations/configuration/etc before any of the packages are installed. The user runs though the entire dependent set of package forms before the install actually begins. Alternatively an ignite script (like kickstart/autoyast/etc only with most of the functionality of chef/puppet/etc) can be generated for automated deployments.
kasey_junk hints at this bit I'll spell it out:
This is a problem with enterprise software as well as some Open Source software. IMO Open Source might sometimes even be simpler than some commercial stuff.
Jokes about SAP consultants moving in isn't funny anymore.
Former boss of mine told me every one of his customers that ever had decided to go with SAP had burned themselves. (IIRC the latest one I heard of started with an estimate of 5 illion monies and when I was there to install our stack they were at 25 and counting.)
What's missing is this:
1. Architecture for your specific needs. This is a design process and not easily condensed into a tutorial. It still requires critical thinking on the part of the designer.
2. How the different components fit together and why they matter.
For (1) to be commoditized, there needs to be sufficient number of installs where people try different things and come up with a best practices that the community discovers. There are not enough of that for that to take place.
For example, I put brought up a production cluster on AWS. I also had to decide how this was all going to interact with AWS VPS and availability zones. How do I get AWS ELB to talk to the cluster? The automated scripts are only the starting point because they assume a certain setup, and I wanted to know what the consequences of those are. This is where the consultants and systems design comes in.
On the other hand, Kesley Hightower probably has a lot of that knowledge in his head. By getting it out there, I think more people will try this, understand the principles, and collectively, we'll start seeing these best practices emerge. Maybe Hightower will eventually write a book.
In the meantime, if you want to know how to design and deploy a custom setup, you do need to know the building blocks, how they are put together, and how you can compose them in a way for specific use-cases. It's no different than choosing a framework, like Rails or Phoenix, and then learning how to compose things that the framework offers you in order to do what you need to do. You get that knowledge from playing with it and experimenting.
Having said all of that, while I'm glad I do have a good foundation for Kubernetes, if I want to use it in production, I'm probably just going to use GKE.
A big chunk of that work is what is being called "Self-Hosted Kubernetes". The idea is that once you bring up a single machine running a Kubelet you can bootstrap the other services that make up a Kubernetes cluster from there. You can learn more about that here: https://coreos.com/blog/self-hosted-kubernetes.html
As far as TLS there is ongoing work upstream to add a CSR system for the "agents" called Kubelets. This will allow people to automate the TLS setup and simplify the management. Details are tracked here: https://github.com/kubernetes/features/issues/43
Also, there are more discussions happening to improve the first install experience. https://github.com/kubernetes/kubernetes/pull/30361#issuecom... https://github.com/kubernetes/kubernetes/pull/30360
Kubernetes is really focused on not just making it easy to install. Which is the trivial scripting part, as you point out. But, to make Kubernetes easy to manage over the lifecycle of the cluster. Which is where work like self-hosted, TLS bootstrap, etc start to come in.
It makes me cry because, running k8s is hard, ops is hard, and so is telling people they might be wrong. K8s consists of half a dozen components, they have dozens of config flags, and much functionality is buggy, in beta, or flux. To top this off k8s is based on etcd. Etcd is barely production ready by their own admissions (remember /production.md in github?) but if you have run it you will understand the bugs, and vague docs coupled with reading the source constantly when problems arise. K8s consists of many components, kubelet, proxy, controller, scheduler and more. These you have to install and configure, and many scripts do this badly in a one size fits all approach, and many CM methods do this barely in an ok manner currently. I cry, because of overlay networking too, its a nightmare, and the alternative cloud permissions are scary.
Also do people really think k8 is a drop-in/trivial solution? I just got done evaluating it and the overriding sentiment seemed to be "it's super flexible but super complex and badly documented and you'd better hope you're using the happiest of happy paths."
We're always looking to improve on both complexity and documentation - but, as Kelsey pointed out in his tutorial, it's definitely got a few steps. The biggest question is would you really want any fewer? That is to say, you could use a hosted solution (like Google Container Engine) or set up with a one-line command, but most people who want to run at production scale definitely want individual steps so they can customize.
Why? With VMs there is no reason, ever, to be modifying the OS with the VM still running, rather then cutting over to a new, tested instance.
The biggest reason people use Kubernetes is because it is the best solution to solve the need "how do I bring my containers to production." You're absolutely right, you can solve this in many other ways, but, no matter what, you're going to have to address the problem of scaling, updating, monitoring, logging, securing, maintaining persistence, implementing 12-factor applications and many many other things before you do - Kubernetes solves all this for you.
Just a quick correction - Kubernetes is not based on etcd, nor do you have to set up half a dozen components. There are three components to Kubernetes - an API server, a scheduler and a controller manager - there are flags, true, but you can easily use a configuration file (just as you would with any other production ready server).
If you saw anything that's badly documented or buggy, please let me know!
Highly recommend watching the video when it's released if you weren't able to attend.
We had a conversation later in the hotel lobby where he made a great analogy: running this stuff yourself is going to be like running your own mail server. Its really nice to know how to do it, but at the end of the day unless you are a very large organization, you're most likely going to use a hosted service.
Personally, I'm going to go through with setting up a test kubernetes cluster just so I know what it's made of. Then if I think it's great, maybe .. just maybe I'll give Google's hosted solution a try with a small project to start.
For personal stuff GKE looks really nice.
While one-liners are typically pretty limited, the charms come with quite a few knobs to help tweak for deployments.
http://www.jorgecastro.org/2016/07/29/ubuntu-kubernetes-v1-d...
There's still room to improve, but we've been happy with the cluster so far. Considering Juju and charms are open source. Eitherway, great guide for those getting started.
The irony.
I've spun up a few clusters using this, and I absolutely prefer learning this way. I love to get something running before I dive in and look at all the pieces and individual options. I also really love "convention over configuration". I want to study a production-ready reference implementation that simply works, with sane defaults.
I've spent a lot of time banging my head against the wall while I try to follow some complicated tutorial that doesn't work with the latest versions. Maybe this approach works for others, but it's not for me. I like to stand up a cluster, deploy a database and a real application, try to scale it, set up some test DNS records, do some load testing. Figure out the pain points and learn as I go. If something just works perfectly fine behind the scenes, then I probably don't need to learn that much about it (or I have enough experience that I already know what it does and how it works.)
This might not be a suitable learning style for a beginner, but I think I have enough experience working and experimenting with AWS, Saltstack, Chef, Puppet, Ansible, OpenStack, Deis, Flynn, Terraform, Docker, CoreOS, etc. etc. So at this point, I just prefer to evaluate new technologies by spinning them up and diving straight in.
And it's lacking wayyy to much documentation deploy in production on own cluster. It's probably gonna take years to improve.
I'm wondering. Did anyone tried the kubernetes on GCE?
If Google can handle all the annoying setup and makes a mostly managed service, that would be extremely attractive. Actually, that's probably the only way k8s would be achievable in production, i.e. have someone else do it.
for your CI woes there is a system that isn't hard to setup and is actually really easy to use: Mesos and Marathon with weave (without the plugin, for now) and docker.
Your biggest challenge is learning zookeeper, but really if you're dealing with large scale deployments, you're probably already using it or something like it.
there are puppet/chef/ansible modules for installing and configuring mesos and zookeeper.
toss in gluster as a storage driver in docker and you're pretty much ready to go for most types of application deployments using docker.
heck, there's even kubernetes integration if you're really hung up on it ;)