brendandburns

Kubernetes handles most of this seemlessly for the cluster infrastructure.

The central master handles node failures by removing nodes that aren't heartbeating.

On the node, we require a process monitor for the kubelet (by default we use supervisord) but then the kubelet monitors Docker [and also does garbage collection and resource limiting ], and then all of the other node daemons (e.g. the kubernetes proxy) are run/monitored/restarted by the kubelet.

this was originally sent out on twitter by a bunch of people (including myself) who are working on containers (and have a container-oriented set of followers) so I'm pretty certain there is a forward looking bias in the data.

I tried to describe this bias at the top of the blog post.

Disclaimer: I work on Kubernetes for Google

Its great to see this launch. We've had flexible resources for container workloads for a while, its great to see them be available for more traditional VM workloads too!

NB: I'm one of the founders of the Kubernetes project

I'd recommend anyone trying to install Kubernetes check out: https://get.k8s.io/

Or the single node Docker based instructions:

https://github.com/kubernetes/kubernetes/blob/master/docs/ge...

Or boot2k8s: https://github.com/skippbox/boot2k8s

Or kubernetes on OS X: https://github.com/rimusz/coreos-osx-gui-kubernetes-cluster

Or of course the "as a service" versions like Google Container Engine, CoreOS Tectonic, etc.

We're definitely also working on simplifying the install instructions.

I'd also like to expand a little on some of the features that differentiate Swarm from Kuberentes, namely:

   * Secrets
   * Replicated sets of containers
   * Rolling update from one version of code to the next
   * AutoScaling (Kubernetes 1.1, in Release Candidate now)
   * HTTP load balancing for autoscaling  
   * Load balancing for sets of objects (e.g. Frontend)
   * Service discovery of those replicated sets (Swarm has discovery for individual containers, but no concept of Service)

In general, our goal is to build a system that makes distributed system construction easier.

I would also refer people to the discussion in https://github.com/docker/compose/issues/1899#issuecomment-1... as well as https://github.com/docker/docker/pull/8859 on why we feel that the Node-based Docker API is problematic for a cluster level API.

(which is a long-winded way of saying: PRs welcome, please help us make it better ;)

(acknowledgement, I'm a contributor to Kubernetes)

It is well supported on AWS as well, and a variety of bare-metal solutions (e.g. Red Hat Atomic, CoreOS)

However, concretely, it is a challenge to maintain good support for N different platforms without an owner who is willing to stand up and ensure that it works, and continues to work for that platform.

We have gotten a number of drive-by contributions of "how to" guides that (sadly) bit-rot over time. As always, we're working on improving the situation, but it is complicated and requires a great deal of time and access to infrastructure (e.g. Rackspace) that the core team simply doesn't have.

the problem with port mangling is that your application starts running on random ports, so in addition to requiring discovery for IP addresses, you now also have to do discovery for ports, which pretty much requires custom code and infrastructure linked into your binaries (how do you convince nginx/redis/... to use your lookup service for ports?)

And ports are different between different replicas of your service, since they're chosen at random during scheduling.

It also makes ACLs and QoS harder to define for the network, since you don't have a clean network identity (e.g IP Address) for each application.

Please check out the Secrets object in Kubernetes:

https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

which is designed to address some of this.

Users of the Google Cloud run Docker in VMs, since VMs are what the Google Cloud Platform sells.

(as does every public cloud provider [e.g. AWS])

For now, VMs are required to ensure a security barrier between different user's containers on the same physical machine. See some of Dan Walsh's posts on the subject (e.g. https://opensource.com/business/14/9/security-for-docker) for more context.

(kubernetes contributor here)

SDN isn't required for k8s, what is required is that each Pod (group of containers) get it's own IP address, and that the IP address is routeable in the cluster. In many cases, the easiest way to achieve this is via an SDN, but it is also achievable by programming traditional routers.

The reason for wanting an IP address per pod is that it eliminates the need for port mangling, which dramatically simplifies wiring applications together.

I'd encourage you to check out:

https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

It's a fairly straightforward getting started experience.

Also, if you want to turn up a cluster in a cloud provider, it's as simple as https://get.k8s.io

Omega is a separate system than both Borg and Kubernetes.

Kubernetes is heavily inspired by both Borg and Omega, and incorporates many of the ideas from both, as well as lessons learned along the way. And many of the engineers who work on Kubernetes at Google, also worked on Omega and Borg.

Please check out: https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

for turn up instructions on AWS, it's as easy as:

export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | bash

Yeah, I think that sadly, there is going to be a little bit of an inevitable equivalent to the unix wars of the early 80s. The sooner we can reach a standard place, the better it's going to be for the container community and developers more generally.

One of the reasons that I pushed hard to get Kubernetes open sourced, is the hope that we could get out in front of this, and allow the developer community to rally around Kubernetes as an open standard, independent of any provider or corporate agenda.

Clarification:

Each pod has it's own IP address that is routeable anywhere in the cluster. This makes life much easier because you don't have to do port-forwarding onto the host node.

In all current k8s set-ups, each Minion/Worker node has a subnet that it allocates these Pod IP addresses out of. This isn't a hard requirement necessarily, but it tends to be much easier to make this work, since you only have O(Workers) routes to configure instead of O(Pods), but long term, I think we would rather do away with subnets per node, and simply allocate IP addresses for each Pod individually.

This is also very true. The purpose of systems like Kubernetes/Mesos is to be set up once by a company, and for most developers to just interact with the CLI tool to run containers. Tools like Google Container Engine, make this into a software as a service product, where the cloud provider provides the API. In that world, as a developer, you never really think about the OS, just your app. But unlike in traditional PaaS, there is no framework that restricts what you can code (language, libraries, etc).

Disclaimer: lead engineer on kubernetes here...

It really depends on how you do deployment. Containers provide deployment (and more important rollback) that is better than other deployment tools like Puppet/Chef/... because they are atomic (they either work, or they fail, they don't get stuck in the middle) and they package up all of their dependencies within them, so that they don't have the "well it worked on my machine" problems.

Systems like Mesos and Kuberenetes, decouple applications from the individual machines (and the operating system on those machines), and are online systems with self-healing properties (so that they will fix themselves rather than waking you up in the middle of the night)

k8s and mesos turn containers into an API that spans an entire fleet of machines, and enables you to dynamically use (and re-use) the fleet of machines for multiple different applications. No more dedicated boxes for mysql, mongo, etc. This in turn enables you to have an easier ops experience, because every single machine in your fleet is homogenous (same OS, same patches, etc) OS management is abstracted away from Application management, so that they don't interfere with each other. Since things in the API are expressed in terms of applications, it's easy to add health checks and automatic restart to the system, and provide self-healing properties as well. Both kubernetes and Mesos also make replication a first-order primitive so that it is easy to scale in response to load.

Ansible is sort of orthogonal to systems like Kubernetes and Mesos. Kubernetes and Mesos are designed to be online, self-repairing systems. Ansible is a way to easily execute commands on a bunch of machines. I can see collaborative use cases, where you generally use Kubernetes for deployments, but use Ansible for querying some data while debugging, or somesuch.

Anyway, sorry for the extended response. There actually is way more that I could say about the topic ;)

I updated it over the weekend, it's down to 3 steps now:

https://github.com/brendandburns/kubernetes/blob/hyperkube/d...

And I think I can get it down to a one-liner.

(also, can you update your blog to point to the hyperkube:v0.14.1 image instead of :dev, :dev is a random binary from my client, where as v0.14.1 is an official release... Thanks!)