Official Kubernetes on CoreOS Guides and Tools | Hacker News Reader

Official Kubernetes on CoreOS Guides and Tools

coreos.com

125 pointsphilips38 comments

If you are looking for the fastest possible guide for playing around with Kubernetes check out the vagrant single machine instructions: https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagr...

This will get you to a "single machine" kubernetes cluster and a working `kubectl` tool in a few minutes.

This is awesome.

The existing single node solutions have been painful to get up and running, and always seem to be missing some feature or other (DNS, for example).

I just spent 30 hours figuring that out (DNS, flanneld), and that's with using hyperkube. (I'm building a new framework that helps you build a custom dev-environment for your workflow using Kubernetes, called Matsuri).

I don't regret it, since I know a lot more about the details of how this works together, and this will help a lot when it comes time to putting this on AWS for production. But I don't want to force this on my team either, so I'll end up putting something together with Vagrant (or finding something that works with it).

One thing I realized though: this stuff should be baked in as part of a standard Linux distro. A working, single-node Kubernetes setup works far better than docker-compose / fig / crane / any number of homebrewed orchestration for dev work. No wonder people are using CoreOS and RHEL 7/Atomic, and not Ubuntu. It's seriously making me consider switching to one of those, off of Ubuntu.

You can always still use ubuntu from within a container on CoreOS if you like ;)

I know. I'm just not as familiar with CoreOS.

You can get a one-click managed Kubernetes cluster on Google Cloud Platform (we call it Container Engine.) Would that be interesting to you vs. running on AWS? [Googler asking.]

Yeah, I saw there was a of great instructions on Kubernetes and GCE.

AWS, for all it's warts and cruft, is familiar to me, and we have our operations on it right now.

What I'm focused on right now is building a dev environment framework. There's already excessive focus on getting this stuff into production. Neither AWS or GCE will help me with that.

We've been using CoreOS in production with etcd and fleet for over 1 year now on 500+ machines. They're have been some growth pains, specifically with etcd - but now they are mostly gone (with the new raft implementation in 2.x). I really appreciate the CoreOS team and all related contributors.

First persistent storage tackled, then networking and now resource-aware orchestration.

How has etcd been running for you more recently? One of the things that kept me away was my test run of etcd failed gloriously (twice), and just wasn't stable/resiliant enough for me.

We published etcd 2.x earlier this year, and it is super stable in current stage. We run harsh failure injection tests on it all the time, and the cluster could survive well. You could check https://coreos.com/blog/new-functional-testing-in-etcd/ for more details.

We're about to start running with it but I am definitely a little concerned at the failover table. I'm a lot more comfortable when I can achieve 2/3 failure and still be okay although the cost of adding a few more instances isn't too bad.

To avoid split-brain a consensus system like etcd cannot make progress with less than 50% + 1 members operating. This is just a hard constraint of this type of distributed system.

You can find resources that explain all of the constraints in depth at the Raft homepage: https://raft.github.io/

Thanks I'll check it out. Again, the cost of spinning up even a 9-node cluster these days is negligible but my initial plan was to separate etcd clusters by application rather than sharing between but we'll obviously be doing some serious experimentation as we move forward.

You'll be glad to know that etcd has added basic ACL support to keys. If you were worried about isolation/access to the etcd cluster across apps, this might help you avoid running a cluster per-app.

Yep, that's exactly what I have some staff internally looking into. Very happy to see it.

It's important to note, because it's often missed, that etcd can operate either as a participant in the high reliability / consensus portion of the cluster, but also as a proxy that does not participate. So you don't have to think in terms of 50%+1 of an entire cluster needing to stay up for consistency, only 50%+1 of the etcd cluster. This is nicely illustrated in the the cluster architecture docs at CoreOS.com.

https://coreos.com/os/docs/latest/cluster-architectures.html

Everything changed in 2.x - it's extremely stable and consistent. Never lost any data in the last 6 months and we beat the hell out of it. 2.x is really a completely different product

I went down a similar road with etcd and fleet but abandoned it earlier this summer after testing failure scenarios with etcd. With a cluster of 5 etcd nodes in EC2, I started hard-killing etcd EC2 instances and noticed fleet inconsistency (e.g., nodes being restarted, not able to see the entire fleet).

Can you expand on the etcd growth pains you've been through?

I would definitely recommend that you reevaluate it with a newer version of etcd, as it has had some significant stability improvements post-2.0. I've been doing some fuzz testing of it lately and found that it has gotten much more reliable.

The basis of this, was being pointed in the right direction by the community.

etcd had a HUGE issue with the implementation of the raft consensus algorithm they were using. This was in version 0.x

The tough part was that, even though etcd 2.0 was released in January [1], it was not put into CoreOS alpha until April [2]

After moving to 2.x - all my problems went away. It had a small learning curve of setting up lots of nodes in the cluster vs proxies [3]. 2.x had a lot of functionality added, but the main one for us was it's reliability. Being able to query status of members, add/remove members from the cluster and monitoring.

Before etcd 2.x, the whole etcd infrastructure would die (and consequently, fleet) if just ONE node restarted. Needless to say, it's come a long way.

We've been running etcd 2.x since January in a container [4], then just doing export FLEETCTL_ENDPOINT=http://127.0.0.1:2379

[1] - https://coreos.com/blog/etcd-2.0-release-first-major-stable-...

[2] - https://coreos.com/blog/coreos-alpha-with-etcd-2/

[3] - https://coreos.com/etcd/docs/latest/admin_guide.html

[4] - https://coreos.com/blog/Running-etcd-in-Containers/

Nice! Wikimedia Labs is going to start offering Volunteers who want to do any Wikimedia related projects Kubernetes access for free soon (https://lists.wikimedia.org/pipermail/labs-l/2015-September/...) although that isn't going to be on CoreOS.

Anybody know of any projects of a similar concept but applying to unikernels instead of containers? I've had way more luck working with OSv than I have had with Docker and I'm kinda burnt out with futzing with it...but I do like the ideas behind kubernetes.

There is no fundamental reason why you shouldnt use the same tools. There is a bit of work required to get it going, around standards and so on that I am looking at.

Has Kubernetes grown past its limit of ~100 underlying servers? What's the largest production Kubernetes cluster running today?

Yes, the community is working hard to make k8s scale to 1000s of machines and beyond. There is a great blog post about the engineering effort from the various communities (k8s, etcd) to make this happen: http://blog.kubernetes.io/2015/09/kubernetes-performance-mea...

In fact someone recently made a video showing a 1000 machine cluster; let me find that.

UPDATE: here is the video https://www.youtube.com/watch?v=sgWv9sVTIYQ

why don't they add something like master slaves to make that happen? i mean 1000s machines for a single master is way more than most software will handle anyway, still kubernetes makes it easy to manage it, however even 50 machines are working really fine on kubernetes and that even with a slower master server. currently we run with 20s at production with a master at 4 gigs which works really really well.

There is an open PR to this project to document an HA control plane. Basically, you add the kube-podmaster onto each master machine, which does a leader election for the API, controller manager, etc so that only one instance is mutating the cluster. If that does down, another leader election happens.

The "master" in kubernetes is a collection of components that can be distributed. The kubernetes component that is the most loaded in a 1000 node system is the api server. The 1000 node cluster config in that video is running 10 api servers.

I'm the author of that video, BTW, so I know that for sure. :-)

The last time that I saw somebody from Google give a presentation about k8 it didn't seem like "larger cluster sizes" was a real priority at this point, and they were more focused on features. Good to hear this is getting some attention, since it was the major issue that made our team dismiss it as useful in production.

May I ask at which scale do you operate? Keep in mind that is not necessary (and I believe not advised[1]) to keep all your infrastructure in one kubernetes cluster.

[1] https://youtu.be/-8aUxpVrD40?t=3105

I've heard googlers mention that same thing, but now that I think about it I believe they were meaning to say that "for 1.0" they were targeting ~100 nodes, and once they got out there door they would start tackling the large cluster case, which appears to be progressing nicely.

Here's an updated video with a voiceover...

https://www.youtube.com/watch?v=1bHUsBhPL20

Support for 250 nodes is scheduled for October.

https://github.com/kubernetes/kubernetes/blob/master/docs/ro...

How does having the kubelet binary shipped with CoreOS itself work with installing and upgrading Kubernetes? Won't the versions get out of sync and cause problems?

We rely on the API boundaries between the kubelet and the kubernetes API server remaining well versioned without backwards compatible changes. The k8s project is very conscious of the needs for that and work hard to ensure that there are no breaking changes between releases. This is similar to all pieces of software in a CoreOS release: systemd, docker, cloudinit, Linux, rkt, etc.

I see. I suppose the only down side of using the CoreOS version is having to wait for it to be updated when new minor releases come out with new features/improvements.

Here's another way to jumpstart and learn Kubernetes. Copy and paste this single command into any Ubuntu 64bit VM and it'll give you a single node Kubernetes cluster with all the tools and services(incl. DNS)

    wget -qO- http://git.io/veKlu | sudo sh

Piping a random shortened link to sudo sh? What could go wrong?

Reply on news.ycombinator.com