There are parts of docker that are relatively stable, have many other companies involved, and have been around for a while. There are also "got VC money, gotta monetise" parts that damage the reputation of stable parts.
Let me ask: WTF are people are doing that <100 machines is a "small cluster"? I ran a Top 100 (as measured by Quantcast) website with 23 machines and that included kit and caboodle -- Dev, Staging and Production environments. And quite some of that were just for HA purposes not because we needed that much... Stackexchange also runs about two dozen servers. Yes, yes, Google, Facebook runs datacenters but there's a power law kind of distribution here and the iron needs are falling very, very fast as you move from say Top 1 website to Top 30.
Usually my cluster is very small, unless the pipeline is in progress. At this very moment I am not doing any processing, so I only have 3 machines up.
But if you need to store a lot of data, or need to look up data with very low latency, or do CPU-intensive work for every request, you will end up with a lot more servers. (The other thing to consider is that SaSS companies can easily deal with more traffic than even the largest web sites, because they tend to aggregate traffic from many websites; Quantcast, for example, where I used to work, got hundreds of thousands of requests per second to its measurement endpoint.)
Also, some sites are simply larger than StackExchange, and you never heard of them. There's a huge spectrum between StackExchange and Google.
- If you don't use google container engine (hosted kubernetes, also known as GKE), kubernetes have a reputation of being hard to set up. I am running on GKE, so I can't comment too much.
- It's hard to see how all of the orchestration/docker/etc. things play together when entering the area. I expect many people hear "docker" and want to just try "docker", not being aware that there exist alternatives for some parts, some parts are more reliable that others, etc. E.g. the article we are discussing seem to be doing this.
You still have to deal with the other drawbacks of those platforms (slow networks, disks etc.) but that's not really a k8s issue.
Yeah, but there's no shortage of things wrapping Kuber (e.g. OpenShift).
Let's give it a try and see if I can find a good tutorial for kubernetes to do what I want. (I haven't tried this experiment in a couple months, since before nomad and swarm got my interest.)
-----
Ok, I'm back. I went to kubernetes.io. there' "give it a try" has me creating a google account and getting set up on google container engine. Due to standard issue google account hassles, I quickly got mired in quicksand having nothing to do with kubernetes.
I have no interest in google container engine. Let me set it up using vagrant, or my own VPSes, or as a demo locally, whatever, I'll set up virtual boxes.
They lost me there.
K8s might seem more unwieldy than swarm, but from that feature set you can expect things to work the way they are explained.
Swarm on the other hand has made my entire team question whether 1.12 is even worth upgrading to.
http://kubernetes.io/docs/getting-started-guides/
Ok, let's see, not only do I immediately get diverted to another page, but I feel like every OS+Cloud combination isn't represented. I guess the CLOSEST thing to working is Ubuntu+AWS. Click.
http://kubernetes.io/docs/getting-started-guides/juju/
YAY! JUJU a new technology I need to learn. Hey guess what, this only works with Ubuntu. Closes browser. I spent weeks trying to map this out in my head. I can't understand why Kub doesn't just "install" like Docker does.
Ok, back to the deployment guide. Let's see there's a GIANT TABLE OF LINKS based on cloud+OS+whatever. So I think it's a massive understatement to say that Kub is more unwieldy than Swarm 1.12.
I would love for our docs to be better - good people are working on it though documentation is always hard. In the meantime, the community is a wonderful resource!
http://kubernetes.io/docs/hellonode/ Is the guide that I was taking about. From your post it was not clear that you weren't using gce.
It's a local setup to let you give Kubernetes a try. We haven't made it the default in the "give it a try" dialog yet, but we're considering it.
Disclosure: I'm an engineer at Google and I work on Minikube.
Kubernetes is definitely more to learn before getting started than swarm, but that's mostly because it has different and more powerful primitives and more features built in.
We hired a contractor to do this part. My team is so resource constrained that I don't have time for this, so we farmed it out. But now I'm thinking, the risk of project failure is much higher than I thought, made worse that the contractor is also showing signs of having poor communication skills. (I would rather be updated on things going wrong than to have someone try to be the hero or cowboy and figure it all out).
The only reason why we didnt go ahead is because of broken logging in k8s - https://github.com/kubernetes/kubernetes/issues/24677
This is the blocker for me. I cant switch to GKE already because I use AWS postgresql. But I want to use k8s :(
If you were not using AWS EBS .. what were you doing ?
I was on AWS. I sidestepped the issue by using AWS RDS (postgresql).
I had tried to get the nascent EBS stuff working, but when I realized that I'd have to get a script to check if an EBS volume was formatted with a filesystem before mounting it in K8S, I stopped. This might have been improved by now.
On the logging front, kube-up comes up with automatic logging via fluentd to an ElasticSearch cluster hosted in k8s itself. You can relatively easily replace that ES cluster with an AWS ES cluster (using a proxy to do the AWS authentication), or you can reconfigure fluentd to run to AWS ES. Or you can set pretty easily set up something yourself using daemonsets if you'd rather use something like splunk, but I don't know if anyone has shared a config for this!
A big shortcoming of the current fluentd/ES setup is that it also predates PetSets, and so it still doesn't use persistent storage in kube-up. I'm trying to fix this in time for 1.4 though!
If you don't know about it, the sig-aws channel on the kubernetes slack is where the AWS folk tend to hang out and work through these snafus together - come join us :-)
From what you wrote, it seems that lots of people consider logging in k8s to be a solved issue. I'm wondering why is there a detailed spec for all the journald stuff, etc.
From my perspective - it will be amazing if k8s can manage and aggregate logs on the host machine. It's also a way of reducing complexity to get started. People starting with 1-2 node setups start with local logs before tackling the complexity of fluent, etc
Is that the reason for this bug?
If you want logs to go into ElasticSearch, k8s does that today - you just write to stdout / stderr and it works. I don't love the way multi-line logs are not combined (the stack trace problem), but it works fine, and that's more an ElasticSearch/fluentd issue really. You'll likely want to replace the default ES configuration with either one backed by a PersistentVolume or an AWS ES cluster.
Could it be more efficient and more flexible? Very much so! Maybe in the future you'll be able to log to journald, or more probably be able to log to local files. I can't see a world in which you _won't_ be able to log to stdout/stderr. Maybe those streams are redirected to a local file in the "logs" area, but it should still just work.
If anything I'd say this issue has suffered from being too general, though some very specific plans are coming out of it. If writing to stdout/stderr and having it go to ElasticSearch via fluentd doesn't meet your requirements today, then you should open a more specific issue I think - it'll likely help the "big picture" issue along!
1: https://cloud.google.com/error-reporting/docs/formatting-err...