Kubernetes: The Future of Deployment
bashton.com
bashton.com
They're also at least two build versions behind Docker.[2]
[1] https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
[2] https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
Your implication was, at the time, there was only a GCE Persistent Disk provider.
Today, there's that, plus AWS EBS volume, git repo, GlusterFS, NFS, Ceph block device, iSCSI, as well as host path and empty directory.
Sounds like the product has evolved with a broad spectrum of support to me!
Side note: I should be a better citizen and link to specific commits next time.
[0] https://github.com/docker/docker/blob/master/CHANGELOG.md
I understand you might want to play with it in a lab environment, but in production at scale (especially Google-scale) sounds very strange.
(as does every public cloud provider [e.g. AWS])
For now, VMs are required to ensure a security barrier between different user's containers on the same physical machine. See some of Dan Walsh's posts on the subject (e.g. https://opensource.com/business/14/9/security-for-docker) for more context.
https://cloud.google.com/container-engine/
There's also Amazon EC2 Container Service
http://aws.amazon.com/ecs/details/
So Google and Amazon don't just sell VMs. They sell "CMs" as well (Container Machines).
Also Amazon bills you for that EC2 instance as any other instance.
Personally I have hard time understanding the benefits of running docker in public cloud, you still run a VM you still pay for that VM. It just one extra abstraction layer which increases complexity of your infrastructure and also reduces performance.
I do understand the benefits of using containers in own data center, when you run it on bare hosts. There's simplicity and and lower costs (because you don't have VM) you have more resources which lets you run more containers than VMs on that host.
Simpler deployment and basically forcing "12-factor", as well as easier development environment setups. Nothing you can't achieve with other tooling, but it's nice to be able to guarantee that your dev environment is identical to your prod.
Given this, it actually makes sense to combine VM with Docker, check out www.hyper.sh
[1] http://mojo.codehaus.org/rpm-maven-plugin/ (the website does not seem to be available at this moment due to recent CodeHaus shutdown)
However, it isn't too difficult to setup a Layer 2 unification overlay network. I've had success with flannel[1], it's pretty easy to set up and supports VxLAN.
The k8s upstream was forward thinking enough to make the scheduler parts of it pluggable, which allow the (imo) holy grail of something like this https://github.com/mesosphere/kubernetes-mesos. This gives you the nice logical "pod" description for services, running on the battle tested mesos scheduler.
There are many 10k+ node bare metal mesos deployments (apple, twitter, etc). There aren't yet many kubernetes deployments of that scale. They truly are mutually beneficial. Mesos makes ops happy, and k8s makes devs happy. Together you have a relatively easy to setup internal PaaS (your own heroku with not a ton of work) more or less.
Disclaimer: I'm a heavy mesos and apache aurora user.
The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max.
Source: One of the core k8s developers I met at a CoreOS meetup in SF earlier this year. They said if I needed to go beyond 500 nodes that I should probably look at something else.
[0] http://research.google.com/pubs/pub43438.html
Disclaimer: I work for Kismatic.
It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?
I did this on my Macbook, using Virtualbox, Vagrant, and this:
https://github.com/mesosphere/playa-mesos
^ I started with that, and then installed Chronos with apt-get in addition.
Specifically, for launching Docker containers, this was useful:
https://docs.mesosphere.com/tutorials/launch-docker-containe...
I didn't try Aurora but it seems it'd be an alternative to Marathon + Chronos (Mesos calls all of these "frameworks").
The whole thing is pluggable and can use various distributed state backends (etcd, zookeeper etc) or IP connectivity backends (veth, macvlan, vxlan, openvpn etc) without changing your application. Service discovery uses DNS so you don't need to modify your application to take advantage of it. It's probably the most significant change to Docker in the last year.
This will make the integration with Kubernetes smoother. Currently Google is forced to rip out Docker's native networking stack because it is not flexible enough for their opinionated networking model. This causes many Docker applications to break in Kubernetes today. That problem should go away with Docker 1.7+ because Google-style networking can be expressed as a Docker plugin, which Kubernetes can load programmatically as part of its orchestration. An added benefit is that you can augment Docker with Google-style networking even if you use Kubernetes competitors like Mesos, Swarm, Cloudfoundry etc.
(EDIT added details more relevant to Kubernetes)
Sorry about the barrage of questions. As you can probably guess, I'm very interested in trying this out.
On a past project I did service discovery with Docker and CoreOS using SkyDNS with etcd. Services would register their network location in etcd and SkyDNS would translate those entries into DNS records. SkyDNS ran on every host and the app-level containers linked to the SkyDNS container.
If a container was moved across hosts, the etcd entry would be updated automatically and eventually the updated DNS entry for that service would propagate across the cluster.
Well, let's not get ahead of ourselves.
ohh i bet we're in the same building right now
I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks?
My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
That being said, you k8s is sexy stuff, it just ties you to docker, and I believe soon to be rocket. When I first started evaluating both (around docker 1.2.x), docker was not super viable and was pretty buggy. With 1.6.x and newer, most of my original concerns cease to matter. They are both excellent technologies, use whatever works for your environment.
I've worked on Cloud Foundry, which is ostensibly a Heroku competitor.
The idea that you can replicate Heroku's full functionality "easily" is just silly.
Full-feature PaaSes do a lot of things, including a whole bunch of tedious nitty-gritty details.
We're well into the days of early maturity on PaaS products. You can install Cloud Foundry or OpenShift, or host on Heroku. Writing your own PaaS at this point is a bit like writing a custom operating system circa 1995. Unless you have a compelling reason to do so, you'd be utterly crazy to.
when we were looking at building k8s our mission was to help the world move forwards to a more cloud native approach to development. by cloud native i mean container packaged, dynamically scheduled, micro-services oriented. we figured that in the end our data centers are going to be well suited to run cloud native apps, since they were designed from the ground up for this approach to management, and will offer performance and efficiency advantages over the alternatives. we also however recognized that no matter how cheap, fast and reliable the hosting offering is, most folks don't want to be locked into a single provider and Google in particular. we needed to do what we were doing in the open, and the thing that we built needed to be pattern compatible with our approach to management and quite frankly address some of the mistakes we had in previous frameworks (Borg mostly as a first system).
we looked really closely at Apache Mesos and liked a lot of what we saw, but there were a couple of things that stopped us just jumping on it. (1) it was written in C++ and the containers world was moving to Go -- we knew we planned to make a sustained and considerable investment in this and knew first hand that Go was more productive (2) we wanted something incredibly simple to showcase the critical constructs (pods, labels, label selectors, replication controllers, etc) and to build it directly with the communities support and mesos was pretty large and somewhat monolithic (3) we needed what Joe Beda dubbed 'over-modularity' because we wanted a whole ecosystem to emerge, (4) we wanted 'cluster environment' to be lightweight and something you could easily turn up or turn down, kinda like a VM; the systems integrators i knew who worked with mesos felt that it was powerful but heavy and hard to setup (though i will note our friends at Mesosphere are helping to change this).
so we figured doing something simple to create a first class cluster environment for native app management, 'but this time done right' as Tim Hockin likes to say everyday.
now we really like the guys at Mesosphere and we respect the fact that Mesos runs the vast majority of existing data processing frameworks. by adding k8s on mesos you get the next-generation cloud native scheduler and the ability to run existing workloads. by running k8s by itself you get a lightweight cluster environment for running next gen cloud native apps.
-- craig
More references:
[1] https://mesosphere.com/blog/2015/04/22/making-kubernetes-a-f...
[2] http://blog.kubernetes.io/2015/04/kubernetes-and-mesosphere-...
[3] http://thenewstack.io/mesosphere-now-includes-kubernetes-for...
For instance in a non cloud-native app that runs in VM's, you might have one app per VM and have firewalls between different VM's that don't need to talk to each other. Then if a non-critical app got compromised and an attacker got remote execution or SQL injection or something they can't get to your other app servers or databases.
If all your apps are in a cluster, the non-critical compromised app might be running on the same host as a critical app, in which case the only thing keeping the attacker from your database credentials or other secrets is the docker container isolation which if I understand correctly is not assumed to be secure the way VM isolation is.
What are people doing to address this? Or are my assumptions wrong and it's not actually a problem to worry about? My initial impression with mesos was that you'd only use it if you're at big enough scale that you're running a huge number of instances of the same app or you're running a lot of different data processing tasks that all access the same data so no isolation is needed between them. Now I feel like I see Kubernetes being discussed frequently as a great way to run all your different microservices at any scale (e.g. "The Future of Deployment"), but I've never seen this aspect of security discussed.
In particular, Cloud Foundry has more advanced security groups features, because it's mostly being marketed to enterprise customers.
Disclaimer: I have worked on CF and I work for a company which is a major contributor to CF.
https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
which is designed to address some of this.
(The natural advice is to "hit the books", actually read the papers related to Kubernetes and find out what is both theoretical and useful to this area. I intend to do that soon, but sifting through "practical papers" and looking for something interesting in theory is a lot of work, and I just hoped there might be somebody who could provide a shortcut.)
https://engineering.opendns.com/2015/05/06/docker-container-...
With a large, fairly homogenous environment it didn't outperform random assignment that well, though. It worked best with small, inhomogeneous loads.
> With a large, fairly homogenous environment it didn't outperform random assignment that well, though. It worked best with small, inhomogeneous loads.
Yes, that's probably a piece of the puzzle that I don't have yet -- to know a theoretical model that is both useful in practice and at the same time greedy/randomized assignment is not "good enough" for practical uses.
This is because while you can try to produce an optimal distribution, it's a fool's errand in a distributed system. A perfect solver, given imperfect data, produces nonsense. GIGO, as they used to say.
Amit Gupta wrote some notes last year about it[2].
Disclaimer: I work for Pivotal.
[2] http://blog.pivotal.io/pivotal-cloud-foundry/products/app-pl...
[0] http://info.meteor.com/blog/meteor-and-a-galaxy-of-container...
(Why the pun? Kubernetes was heavily inspired / guided by Google's internal scheduling tool, which was named Borg (http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubern...).)
e.g: "kubernan" in ancient greek means to steer "kubernetes" is helmsman
"gubernare" means to steer or to govern in Latin "gubernator" is "governor" in Latin
Which then leads into the modern word "Gubernatorial", et al.
comments above are right -- we wanted to stick to the nautical theme that was emerging in containers and 'kubernetes' (or helmsmen is greek) seemed about right. the fact that the word has strong roots in modern control theory was nice also.
fun fact: we actually wanted to call it 'seven' after seven-of-nine (a more attractive borg) but for obvious reasons that didn't work out. :)
Iaas provider is not going away,paying for the cost of SDN now for features that doesn't even exists yet, is insane.
SDN isn't required for k8s, what is required is that each Pod (group of containers) get it's own IP address, and that the IP address is routeable in the cluster. In many cases, the easiest way to achieve this is via an SDN, but it is also achievable by programming traditional routers.
The reason for wanting an IP address per pod is that it eliminates the need for port mangling, which dramatically simplifies wiring applications together.
And ports are different between different replicas of your service, since they're chosen at random during scheduling.
It also makes ACLs and QoS harder to define for the network, since you don't have a clean network identity (e.g IP Address) for each application.
Seriously didn't mean to be pushing Juju, I was just surprised at how similar it was to Juju. I had always sort of assumed it was Google cloud only, and/or containers only, etc.