Adding Kubernetes support to the Docker platform
docker.com
docker.com
When people first started using Docker containers, we were promised things would run identically in dev and production - no more "but it worked on my laptop" issues. Then the rise of orchestrators meant that there again became a significant difference between running an app locally (in compose) and in production (on Kubernetes). Docker for Mac/Windows will now bridge that gap, giving me a k8s node to run against in dev.
Whilst Kubernetes has provided a great production orchestration solution, it never provided a great solution for development, meaning most users kept developing with Docker and Compose. It's great to see these worlds now coming together and hopefully leading to a first-class solution all the way from dev to prod.
/me is one of the main contributors of Minishift
https://kubernetes.io/docs/getting-started-guides/minikube/#...
another approach would be to run docker inside a k8s-pod (docker-in-docker), that way you can run images without having to push them to a registry but still test it in k8s-environment (at least to some extent).
There a few minor UX flaws that make it frustrating to use, e.g. having to set Docker host, shared filesystem performance is poor, networking in enterprise desktop environment is broken (just to name a few top most issues).
Also, a lot of folks end up running Docker for Mac and minikube VMs, why should they have to run two VMs?
Additionally, minikube is completely different from production-grade deployments (single binary, which means a rewrite of main function for etcd and all control plane components, as well as hard to debug basic performance issues in control plane, there is one large process and you don't know what is wrong, also there is no way to use your favourite network add-on).
Additionally, minikube is based on legacy Docker libmachine, it is not really maintained anymore.
Shared folders, especially using 9p and/or cross-platform have been an issue, and I personally also experience this in the fork Minishift, and this likely the performance issue you meant.
But back to an earlier question I posted, have you filed the issues you had in the issue tracker? https://github.com/kubernetes/minikube/issues
Yes, the docker/machine code is an issue. For this, the libmachine is mostly moved inrepo and we are working on abstracting and even replacing this.
FWIW, we've found minikube a bit wonky. It's resource intensive, so if you want to run more than a couple services, your laptop starts to melt. One of our open source projects is Telepresence which relies heavily on Kubernetes networking, and we definitely see more weird/networking issues with Telepresence/minikube than with regular K8S clusters.
Note: resource intensive might be because of the hypervisor but generally shouldn't be that bad.
i am excited about this move from docker but i don't think it will solve all the problems. i think once you have a bigger team it is worthwhile to run a second k8s-cluster besides prod where people can just test things on it. otherwise it is actually not that hard to run a local k8s-cluster with vagrant, not sure how docker wants to top that - i think there is no need to top vagrant.
It's been working quite well, no need for multi-node-setup so far that I'm aware of.
This is not like ops vs dev. When you use a library or framework (say, spring) - you don't test whether HTTP MIME Types are working correctly in spring. You assume the library already has all that tested and covered, and as a consumer, you write tests for what you code. The library's code (and tests) are abstracted from you. This is similar, except for operational stuff. In fact, there is no major difference between them. Its just layering and separation of concerns.
There is no such thing as 100% prod except prod. Even for rocket launches. 90+% is good enough for majority of cases, and is already on the higher side.
Kubectl was already providing docker cli commands like "exec" "logs" etc. So you can execute some of these commands on a k8 cluster with the docker binary too? And why would you do that?
All I see is struggling for relevance, duplication of functionality and a very unnecessary vendor lock-in vector.
The API is still k8s and the YAML files are identical, so migration at a technical level off that platform should be easy enough.
I think this move is about Docker maintaining the trajectory in enterprise where people want the management GUIs and extra features but where Kubernetes and particularly Openshift is making progress at the expense of Docker EE.
I'd like to see a price comparison between Docker EE and openshift, because Openshift aint cheap.
Your fallacy is comparing a product that is open core, with a product that is commercial open source.
https://www.quora.com/What-is-the-difference-between-OpenShi...
It's a pretty common model amongst these companies, and hey they've got to make money somehow :)
Edit: I forgot, the logo is not open source. So the logo is withheld :)
For me, the benefit of Openshift over vanilla Kubernetes is the additional management tooling and the strong default settings for production use.
Openshift has a lot of focus on things like manageability and security which make it well suited to production workloads in enterprises and anecodotally it seems to be taking off quite well in enterprise customers.
There is also an OSS version of DC/OS minus some security features. Also, historically Mesos has had better support for running stateful workloads but K8s is catching up too.
Recently, DC/OS announced support for K8s too which is similar to the K8s announcement from Docker.
Disclosure: Apache Mesos committer.
Hopefully this is K8s, but with Swarm simplicity to get started.
There are a lot more exciting work around Helm (packaging for K8S), third party extensions (plugins for K8S), and Operators (embedded managed services). The K8S community already has more contributios going towards the stack, and the three technology I just mentioned reduces enough friction for contributing innovations that k8s will likely phase-shift.
I just don't see Docker catching up. Sure, developers know them and think it has a good developer story. It doesn't. Docker for Mac and Window is practically useless when you have to struggle with file mounting for dev work. Docker Swarm is just not as robust as K8S, and the source of innovation going into Docker Swarm is coming from ideas from K8S.
Someone said it. ... K8S, not Docker, is the Linux of the cloud native platform.
When you create a new Swarm cluster with `docker swarm init`, Swarmkit creates PKI that is used for everything onward: gRPC cluster state communication, encrypted overlay networks, secrets, and so on. Keys are automatically rolled every 12 hours by default. This is done automatically and transparently to the operator, without any effort on their part. Adding workers or managers after that is as done with `docker swarm join --token <token>` on the new node, that's all.
I believe operation ease for starting a cluster and maintaining it for k8s is something that's going to get a lot easier soon from Moby and k8s joining forces.
Secrets. Swarm got encrypted secrets in January, and you create one with `docker secret create <secret-name> <file/stdin>` or the API. It looks like k8s got it as an alpha feature at the end of June and it doesn't look easy: https://www.twistlock.com/2017/08/02/kubernetes-secrets-encr... They are making secrets functionality plug-able, to match the flexibiliy in networks, volumes, and logging.
As far as development power goes, 9k contributors and 9k pull requests in the last year is nothing to scoff at. That's also a misrepresentation because of the effort put forth for CNI and container standards that make a lot of the work interchangeable.
k8s has a lot of nice stuff and a great, evolving ecosystem, but I think you are being a bit harsh on Swarm. I think Moby (and derivatives) and k8s will benefit mutually from this, as they have over the past year already.
As far as secrets go, there are a couple approaches the K8S community is trying for with encrypted secrets. The one I am rooting for is the integration with Hashicorp Vault. Chances are, that area will be fairly extensible.
I agree, 9k contributors and 9k pull requests are not to be scoff at. I wasn't scoffing at them. I'm simply saying that (1) center of gravity and where the influence is now has already shifted off of Docker, and (2) Just like Github.com phase-shifted open-source development (in way that Sourceforge never did), there are three technology in K8S that will likely phase-shift K8S development. This isn't about development power, but about a phase-shift.
I don't think I am being harsh on Swarm (and since when was this about being harsh? At the core of this is about whether the technology is fit-for-purpose and helps enable people achieve their mission and Docker is a commercial enterprise) The tipping point has come and gone. I saw Docker squander a lot of developer good-will over the past two years. I remember when CoreOS announced they were creating rkt, there were a lot of backlash to CoreOS. I liked CoreOS but I thought at the time it was a weird move. If that had happened this year, there would not be as much of a backlash.
Hopefully both sides _now_ come together and sing kumbaya, and we don't see a continuing KDE versus Gnome war a embrace and extend attitude by Docker or a continuing push to marginalize docker by the Kubernetes folks.
And given enterprise support Openshift is still ongoing as the best solution. They are afaik the only ones that offer a complete set of answers to most questions you can have in the PaaS space. Everybody else is like "here's an API, choose one of 3 billion plugins" (just thinking CNI here). In the end for the customer it doesn't matter though. Customers just want things to run smoothly and if possible reduce their maintenance work force. They don't want choices, they want solutions.
Being low on the stack is a power move. It's like a NFL lineman, the lower player has considerably higher leverage then the higher player. Docker can go in, run Kubeadm legitimately, but use Docker based CNI and volume plugins and displace Openshift.
Plus, Openshift is _12k_ a application node on AWS: https://www.openshift.com/dedicated/index.html#pricing
That's for OpenShift Dedicated, in which you're literally buying dedicated engineer & support time along with IaaS, PaaS plus our AWS costs.
OpenShift in on-premise or cloud environments comes in a different pricing structure based on (v)Cores or CPU Sockets.
It's not like kubernetes folks very pushing to marginalize docker for bad reasons. docker runtime was one of the bottlenecks of kubernetes in production and docker inc didn't feel like improving it. Hopefully, it is changing now.
There are some technical decisions in docker that could provide similar functionality with better production support. I am not expert and just saying what I heard from dev working on it: for example, btrfs would work better than overlayfs for COW file system or docker would be better of using parts of systemd rather than implementing everything from scratch.
I've said before and I'll say again: Red Hat, Microsoft and Google (and Pivotal and VMWare) are going to wind up making more money from Docker than Docker Inc does. Kubernetes has swept the field at the container-orchestrator level, the rest is a fight for the upper part of the stack.
Disclosure: I work for Pivotal, though not on PKS.
Now reality has intruded and I am glad, though I predict they'll continue to maintain that Swarm is a first class platform for a while, then quietly let it wither on the vine until one day it's forgotten about.
Also of note is Rancher 2.0's dropping support for Swarm and Mesos and focusing solely on K8S going forward.
The same incomplete debugging is also in kubeadm. It often hangs in the "waiting for control plane" level without any additional info. Helm also has such problems, reporting networking erros when there's no networking problem (if it checks ipv6 first but hten switches to ipv4 for instance). It's also possible that a helm deployment fails, gives no real reason why, then you can't uninstall it at all without restarting the k8s master.
It's maybe even more general, and a problem in the whole Go programming language world. Everytime I see a tool written in Go I immediately cringe and already know that there will be debugging problems. No ideas why nobody inside this community realises it or how they debug. I suspect they don't really debug and live in the illusion that actually others know something better, but actually the others also don't know it.
I'm not really sure where your original comment is going, but I don't really feel the problem is endemic to Go. Using/debugging most software is an exercise in frustration. Just look at Linux on the desktop (which I use btw, I'm not criticizing). Fixing things is usually reduced to tribal knowledge, IRC, and Googling.
It is sometimes hard to read the logging messages and understand how they came to be. But just having a different status report for a different problem is already so helpful. For instance if kubeadm fails with "I need cheeseburgers" when you actually forgot to configure your proxy correctly, and with "I need more minerals" when you forgot something else, then the first time debuggin is quite frustrating. But after that you know "cheeseburger means proxy" and you can continue. But if you hit "waiting for control plane" for ALL the problems, then your brain can't even remember for what to check right now. I'm the best example, I already forgot the other five things that can go wrong and I would need to check my work internal wiki for that.
I think that's the main reason why logging exist, to increase the speed of hitting a symptom to discovering what's actually going wrong. And Go in general, k8s specifically, simply goes in the other direction the whole time. They don't report any errors, and sometimes even report errors when there is no error. This is systematic in some way but I would need to study the community to tell you more specifically what's wrong.
"Docker: powered by Kubernetes" seems to be more of a marketing thing to move down the value chain, and not be seen as a basic piece of infrastructure.
The thing I'm unsure about - and it would be really interesting to get your perspective on - is what this means for Swarm longer-term. Is there still going to be a reason why people will want hybrid? Is it a migration play?
In a hybrid, over time I'd want to two to behave the same, and getting k8s up and running is a one-time cost and likely decreasing maintenance. It seems like a point solution.
Longer term, I think all orchestrators will converge to look more and more the same. Orchestration will become a commodity, and it will matter less and less which orchestrator you use, especially to developers. But this process will take a long time, and in the meantime enterprises (our primary customers) need to deal with the situation on the ground, which is a lot of Swarm and Kubernetes living side by side because of historical decisions made in 2015-17.
I think that's an overstatement. I talk to a lot of professionals in this area and never seen Swarm deployed to production (from small to big corps). Anecdata warnings apply.
Could you share some numbers?
Or check out the customers section on docker.com.
The development of those bleeding edge features is well hidden. The contributions graphs seem to indicate that Swarm is at best a ghost town. Perhaps the action is happening somewhere else and/or Docker will start investing in Swarm once more.
For example, containerd has been moved to the CNCF and made a broader project. Fine, but Docker runs a separate fork of containerd anyways. A neat marketing sleight-of-hand, but to what end?
That separation is already in place for Windows and Mac, so we're starting there.
Don't get me wrong, the work you guys do is cool and all, but that isn't a valid explanation from my point of view. Any company should have some sort of staging to test updated before rolling them out - it isn't up to the developers of the software to take care of this.
And not only that - the switch will come at some point or another either way, so it doesn't make sense to hold that back from CE on linux so that someone doesn't 'find an unexpected and unwanted kubernetes distribution wedged in'. Those who would find that now would also be surprised by that later on.
To add to that - containers are tested using CI/CD tools anyhow, which are predominantly powered by linux machines, which again makes this decision less convincing. The build may be fine on the developers machine and in production, but the CI/CD environment wouldn't reflect both of these environments.
This looks more like a facade for selling more Docker EE licenses rather than wanting to protect users. Which is fine, of course - but then please say that.
Tbh, I wouldn't even have said anything if they were making that an EE-only feature. Thing is - they want to make money with that move and they're not honest about it. And in the process they're throwing the larger demograph using docker on the bleeding edge side of things in the mud. The people who are trying the new features on their own servers in their own time.
> There has been plenty of hue and cry in the past about the rapid rate of change of Docker
Yes, well - that is what happens when a company decides to use bleeding-edge hipster software. With puppet one minor version may not work with the server whos a few minors behind, with ELK in pre-5 versions the cluster may have gone keel over if the migration of the version hadn't been planned meticularly, with consul you may get better performance (dc-local speaking) than with etcd on one release and way worse the next.
Crying to the devs not to produce good software so quickly shouldn't be the solution.
In general enterprises don't like to be told "throw away your existing system to adopt my platform".
(edit: or maybe I'm thinking of CRI-O)
Looking at moving to k8s but that was my reason for swarm at the time. And it was the right one as it just worked with minimal effort.
- Setting up Swarm is trivial, setting up Kubernetes is not so much (kubeadm is still not recommended for production, and there are valid reasons for this). I guess the only thing that's probably easier on K8s is (totally unsupported) multiarch cluster - it's somewhat messy with Swarm[1]. Although my experiments with multiarch K8s had failed (got issues with CNI stuff and postponed research for later).
- Compose file format is significantly simpler and concise. Less code to write is good.
- Debugging failing Swarm is significantly easier than debugging failing Kubernetes. Well, that's probably subjective and I haven't truly deeply debugged either, but at least I believe so - on occasions I was able to find my way through moby, swarmkit & libnetwork source, and K8s feels a very different beast.
____
Update[1]: I re-checked multiarch status for Swarm and found out that now `docker service create --name test --placement-pref 'spread=node.id' --replicas 2 --no-resolve-image ubuntu sh -c 'while true; do uname -a; sleep 1; done'` just works on a freshy set up mixed x86_64+armhf Swarm cluster (no multiarch alpine images yet, though - promised to come next week). Guess, Swarm had beaten K8s here.
A big part of that is because I have had experience running Docker (not Swarm), ECS, K8S, and building developer tooling (Vagrant), in addition to being a regular developer.
The flip side: I had to opportunity to try out these different tech in production and saw where the pain points are. The overhead of K8S exists to solve those pain points, though that is probably not that obvious to a small team without prior knowledge.
For example, I had set up a prod k8s by hand. I will never do that again. On the other hand, I know roughly what is going on when something breaks in our Google GKE cluster.
https://rocketeer.be/blog/2015/11/kubernetes-from-the-ground...
And although I never ran through Hightower's Kubernetes the Hard Way, it is like that. https://github.com/kelseyhightower/kubernetes-the-hard-way
After running through that as a kind of kata, it was easier to infer and troubleshoot things when things go wrong. The transfer-of-learning happens only if you run yourself through these exercises.
I can share some things at a higher level though:
Label selectors are your friend. Master them. They are used everywhere.
Stateless is still easier than stateful. Start with putting stateless workloads in production before ever trying stateful.
If you have the expertise to mix your stateful pods with your stateless pods, make sure you master StatefulSet and things like persistant volume claims.
If you fake stateful pods like I did in production, then Kubernetes does not know how to cleanly shut them down. Automated maintenance involving kubectl cordon and drain no longer function well. You end up having to hand migrate stateful pods from node to node.
I work with a team of four total developers and realistically two of us handle the vast majority of "operations." As a result, what was important for us in orchestration was ease of setup, speed of initial implementation, and the lowest immediate and ongoing difficulty associated with whichever orchestration tooling we chose.
My company is currently fairly locked into another DO for misc. reasons, and as a result would be deploying to DO, where you do not have the benefit of all of the automated tooling/management provided by GKE.
What people usually don't realize is that once you're in the cloud, all your services talk TCP. A service in GKE and another one in AWS are just a network hop away. Two considerations are:
* Network egress costs, which is currently outrageously high, and serves as a lock-in device of sorts. Depending on actual workloads, it may or maynot make sense for you.
* Security. Though there are numerous VPC solutions out there, some of them supported by the cloud providers themselves.
Anecdotically, we also run a small RDS database in AWS, though we only need ~100 small queries a day.
Unfortunately our egress costs would likely be significant.
I am honestly anxious to get my employer started on the path to k8s, but until the tooling reduces the hours required to maintain it successfully, Swarm seems like the superior solution if you are locked in on a non-GKE/Azure Container Service provider.
I have heard good things about Kops helping setup/enable production-stable orchestration, but they are not ready yet for Digital Ocean either, although I think it is on the list™.
Disclosure: I work for Pivotal.
With GKE, most of your focus will be around (1) tooling for generating manifests, such as Helm or (forgot the name of it). I had written something called Matsuri, but that is only useful if your team is a Ruby shop. (2) what to put into the containers and how they link up.
So yeah, I agree with your assessment.
Historically, most IT orgs requiring supported k8s have either gone cloud with something like Google Container Engine or gone with OpenShift and get support from RedHat. OpenShift is a fork if kubernetes though and lags a year or so behind. It also adds opinionated features such as Image Streams.
Docker's announcement said they were using "real" kubernetes, not a fork or a wrapper. I've setup kubernetes by hand before and it is no easy feat. I'm looking forward to evaluating Docker's solution and maintenance upgrade process.
UPDATE: My goal with this post is not to sell people one way or another, but moreso to explain where some of Docker's reasoning for this integration is coming from.
Disclosure: I work for a Docker partner
Disclosure: I work for a Docker Partner
Disclosure: my company is (was?) a big Openshift consumer...
I have skin in this game as well, as my other comments on this post demonstrate. We (Pivotal & Google) kinda skipped the hard bits of keeping up with k8s by packaging it as a BOSH release, so we'll always be up to date. That's actually one of the project goals: parity with vanilla k8s, because it is vanilla k8s, operated by a lower-level system.
What are the details here?
Docker Swarm is beyond awesome and a great path for someone to scale up at the lower end of the scale spectrum (two containers). I really hope that this brings more people into Swarm.
I'm also keen to see what it means for the Kompose project.
> In 2014, Google approached a startup called Docker proposing the two collaborate on software each was developing to help companies manage lots of complex applications, according to people with knowledge of the proposal. But Solomon Hykes, Docker’s founder and CTO, said no. He wanted to go it alone.
> Three years later, the cost of Mr. Hykes’ previously unreported decision is becoming apparent. The software that Google was developing was Kubernetes, an open-source product that now dominates its segment of the cloud software market. Docker’s rival software, Swarm, is also open-source but isn’t anywhere near as popular, two former Docker employees say.
https://www.theinformation.com/when-docker-said-no-to-google (Sorry...it's paywalled :/)
It seems more likely to me that the CRI-O 1.0 announcement was a tactical move from Red Hat to hijack the conversation during Docker's own conference. CoreOS did the same thing 3 years ago when they announced rkt, trying to capitalize on Dockercon as a time to make a bunch of noise for themselves. Docker themselves have been no perfect angels in this regard, for instance with their infamous "accept no imitations" shirt at Red Hat's conference, I'm just calling it as I see it.
Disclaimer: I worked for Docker, Inc. for 3 years.
The only problem I didn't solve yet is debugging python code running in kubernetes using pycharm. If I run a container in pycharm using "Debug..." dialog it launches inside "docker context" rather than "kubernetes context". For example, I can't connect to kubernetes services via their ClusterIP - the container launched via pycharm does not see it. The only solution I found is using docker compose to set up an environment similar to kubernetes and using docker compose from pycharm. Hopefully, this announcement from docker will simplify this story.
For ECS (EC2 Container Service), they use Docker on top of their VMs.
So they have a Docker offering but not a Kubernetes one, at the moment.
I think technically and in some regards also politically the Docker people are smarter though. In the most pro-docker comment I answered with "don't start to ignore k8s yet" and here I feel like saying "don't start to ignore docker(-swarm) yet".
The battle is ongoing and these two are both possible leaders.
https://github.com/antoineco/kOVHernetes
Go with OVH, unlimited resource usage (including traffic) and they allow you to create/own your own private network (vRack) of dark fiber. Look into using multiple points of presence that they offer. If you don't need it nownow, wait for OVH to offer local US machines rather than just geolocated IPs.
You can do this well with bare metal servers or with a number of whatever dedicated and/or shared cloud stuff they offer.