I'm very curious to hear field reports from people who switched to using Kubernetes in production in the last year or so. Why'd you do it? What got better and what got worse? And are you happy with the change?
When its done we expect to be able to scale the different parts of our application independently. This also makes it easier to detect problems (why did Gitaly suddenly autoscale to twice the number of containers).
So a lot of our effort in moving to kubernetes has required us to try and duplicate all of our previous installation work in ways that don't require root access while the containers are running.
To help, we've chosen to use Helm to manage versioning/upgrading our application in the cluster: https://helm.sh/
Journey to Cloud Native: Breaking the monolith and scaling towards tomorrow - https://docs.google.com/presentation/d/1fsgvSuGpn-MnMqKaTOoi...
Our biggest blockers have been how to separate file system dependency for our various parts. For Git content, we've implemented Gitaly (https://gitlab.com/gitlab-org/gitaly/). For various other parts, we've been implementing support for object storage across the board. Doing this while keeping up with the rapid growth of Kubernetes and Helm has been a challenge, but entirely worth the effort.
We are also bringing all of these changes to GitLab CE, as a part of our Stewardship directive (https://about.gitlab.com/stewardship/, https://about.gitlab.com/2016/01/11/being-a-good-open-source...). We don't feel that our efforts to allow for greater scalability, resiliency, and platform agnosticism belong only in the Enterprise offering, so we're actively ensuring that everyone can benefit, just as they can contribute.
We have a lot of improvements coming as well: https://about.gitlab.com/direction/#ci--cd
While it's not a single button, GitLab is getting closer to that goal with the addition of our cluster integration feature: https://docs.gitlab.com/ee/user/project/clusters/#adding-and...
If you create a trial account on GCP, you can then use GitLab to make a new GKE cluster and it will be automatically linked. You can then click a single button to deploy the Helm Tiller, a GitLab Runner, and then run your CI/CD jobs on your brand new cluster!
The GCP trial is pretty nice, you get $300 in credits and you won't be automatically billed at the end: https://cloud.google.com/free/
What made you eventually go ahead with K8s other than the main obvious reason (the community) itself?
One big reason we've chosen to implement with Kubernetes is community adoption. You can now run Kubernetes on AWS, GCP, Azure, IBM cloud, as well as a slew of other platforms both on-premise and as a PaaS. With the expanding ecosystem, our decision has only been further confirmed, as the list of available solutions and providers continues to grow rapidly (https://kubernetes.io/docs/setup/pick-right-solution/)
Another big reason is far simpler: ease of use and configuration. A permanent fixture in our work for Cloud Native GitLab has been that this method must be as easy, if not easier to provide a scalable, resilient, highly-available deployment of GitLab at scale.
We can’t say, “This is our new suggested method. By the way, it’s harder.”
What we have found is that many other solutions require a much larger initial investment of time to understand and configure GitLab as a whole solution, as compared to the combination of Kubernetes with Helm. Helm provides us templating, default value handling, component enable/disable logic, and many other extremely useful features. These allow us to provide our users with a practical, streamlined method of installation and configuration, without the need to spend countless hours reding documentation, deciding on the architecture, and making edits to YAML.
We've been looking at these technologies for two years now, with our focus being Kubernetes for the last one.
We also try to use the same deployment tools for GitLab.com that we provide to customers, and this lets us offer a scalable production-grade deployment method that can run nearly anywhere.
We also no longer have to manage servers.
All of this was possible without Kubernetes, but it is so much easier with it. Although admittedly much of the ease of use is due to not having to manage Kubernetes itself – we use Google Kubernetes Engine. I would not want to install or manage a Kuberenetes cluster/master myself.
I can recommend managed Kubernetes to anyone that runs many different apps/services.
We first deployed v1.2 nearly 2 years ago, and I can say Kubernetes has made some amazing improvements in that time – in terms of functionality, usability, scalability, and stability. This release continues that trend.
We've invested a lot in it too with things like making sure our etcd clusters backing Kubernetes are really solid [2], and we've even added some of our own features to Kubernetes like allowing CPU throttling to be more configurable to get more predictable >p99 latency from our applications.
We've been through our share of production issues with it (some of which we've posted publicly about in the hope that others can learn more about operating it too [3]), but I don't think there's any way in which we could run an infrastructure as large and complex as ours with so few people without Kubernetes. It's amazing.
[1] https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...
[2] https://monzo.com/blog/2017/11/29/very-robust-etcd/
[3] https://community.monzo.com/t/resolved-current-account-payme...
I spent a couple weeks not quite full time going through tutorials, reading the documentation, reading blog posts and searching for solutions to the problems I was having. My biggest problem was with exposing the services "to the outside world". I got a cluster up quickly and could deploy example services to it, but unless I SSH port forwarded to the cluster members I couldn't access the services. I spent a lot of time trying to get various ingress configurations working but really couldn't find anything beyond an introductory level documentation to the various options.
Kubespray and one blog post I stumbled across got me most of the way there, but at that point I had well run out of time for the proof of concept and had to get back to other work.
My impression was that Kubernetes is targeted to the large enterprise where you're going to go all in with containers and can dedicate a month or two to coming up to speed. Many of the discussions I saw talked about or gave the impression of dozens of nodes and months of setup.
Other options I'll probably look at when I have time to look at it again: Deis https://deis.com/ , Dokku http://dokku.viewdocs.io/dokku/ , Flynn https://flynn.io/ , LXC https://linuxcontainers.org/lxc/introduction/ and (though I'd been trying to avoid it) Docker Swarm https://docs.docker.com/engine/swarm/
You can use either their free tier in cloud or use the Open source OpenShift Origin for trials (there's also MiniShift, which is similar to MiniKube).
From my looking at it OpenShift comes with some of the parts that base Kubernetes leaves to plugins, so things like ingress, networking etc are installed as part of the base.
Anecdotally, I got an HA cluster running across 3 boxes in the space of about a month, with maybe 2-3 hours a day spent on it. The key for me was iterating, and probably that I have good experience with infrastructure in general. I started out with a single, insecure machine, added workers, then upgraded the workers to masters in an HA configuration.
I don't think it is really that hard to get a cluster going if you have some infrastructure and networking experience, especially if you start with low expectations and just tackle one thing at a time incrementally.
At Red Hat, we define an HA OpenShift/Kubernetes cluster as 3x3xN (3 masters, 3 infra nodes, 3 or more app nodes) [0] which means the API, etcd, the hosted local Container Registry, the Routers, and the App Nodes all provide (N-1)/2 fault tolerance.
Not to brag, since we're well practiced at this, but I can get a 3x3x3 cluster in a few hours, I've lead customer to a basic 3x3x3 install (no hands on keyboard) in less than 2 days, and our consultants are able to install a cluster in 3-5 working days about 90% of the time, even with impediments like corporate proxies, wonky DNS or AD/LDAP, not so Enterprise Load Balancers, and disconnected installs. Making a cluster read for production is about right-sizing and doing good testing.
[0] http://v1.uncontained.io/playbooks/installation/#cluster-des...
The great thing about automation is that once you have these basic tools (Prom/Graf monitoring/alerting, ELK, node pool autoscaling, CI/CD) implemented as declarative manifests, they're deployable anywhere in minutes.
Edit: especially load balacing the master servers. (that's actually the hard part of k8s, not even setting it up with/out openshift/ansible whatever) load balancing services on k8s itself is basically just running either calico network and use one or two haproxy deployments of size 1 with a ip annotation or just using https://github.com/kubernetes/contrib/tree/master/keepalived...
[0] https://github.com/openshift/openshift-ansible/blob/master/i...
I do have a lot of infrastructure and networking experience, it was mostly a matter of the ingress setup having many moving parts which were poorly documented. I could see that it had set up bridges and iptables rules and NAT and virtual interfaces, but I was never able to get a picture of how the setup was supposed to work to be able to see what parts of that picture were right or wrong.
There was no clear road-map of setting up a cluster. Most people talking about Kubernetes were doing "toy" deployments, which only had limited application to what I was doing. I only found kubespray because of a passing mention, for example.
I'd say your about right with a month. Had I given it another week or two, I probably would have gotten it going. I had really only expected it to take a couple days to have a proof of concept cluster, so at 2 weeks I was way beyond what I had slotted to spend on it.
Looking over the Getting Started Guide it looks very simple to get a test cluster set up. Which maybe set my expectations unreasonably high.
I guess that's what I'm trying to say: With the current state of documentation, it's probably a calendar month investment to get going.
Rancher 2.0 promises nice UI for K8s
(1) it's like 15 minutes to learn how it works and then maybe a morning or so playing around with it. Very low investment.
(2) if you're already using docker-compose, it's pretty much an in-place switch. You might need to deal with a few restrictions (mandatory overlay network, no custom container names, no parameterized docker-compose), in exchange you'll get zero-downtime upgrades, automated rollbacks, and of course the ability to add more machines to your swarm.
I know that people like to talk about cost savings and stuff like that, but I'd like to see if it actually lowered your app latency and increased sales/conversions/whatnot. Things that matter a lot to a growth business.
I ask because the various overlay/iptables/NAT/complicated networking setups in Kubernetes lend themselves to adding more overhead and being much slower than running on "bare metal" and talking directly to a "native" IP. I really, really wish that Kubernetes had full, built-in IPv6 support. It would remove a lot of this crud.
Our solution works around this by assigning IP's with Romana and advertising those into a full layer 3 network with bird. The pods register services into CoreDNS, and an "old fashioned" load balancer resolves service names into A records. Requests are then round robin'd across the various IP's directly. There's no overlay network. There's no central ingress controllers to deal with. There's no NAT. It's direct from LB to pod.
The nginx ingress controller is not a long term solution. It's a stop-gap measure. Someone really needs to build a proper, lightweight, programmable, and cheap, software-based load balancer that I can anycast to across several servers. That or Facebook just needs to open-source theirs.
For load balancing, have you looked at Traefik?
This tarball can be used to re-create the exact replicas of the original cluster, even on server farms not connected to the Internet, i.e. it includes container images, binaries for everything, etc.
But if the replica clusters do have internet access, they can dial back home and fetch updates from the "master". People use this to run complex SaaS applications inside their customers' data centers, on private clouds. These clusters run for months without any supervision, until the next update comes out.
Thanks to Kubernetes, you have a complete, 100% introspection into a complex cloud software, which allows for a use case like this. Basically if you're on Kubernetes you can no longer be tied to your one or two AWS regions and start selling your complex SaaS as downloadable run-it-yourself software. [1]
If the offline server farms hosted a private Docker registry (which is very simple to set up), couldn't you then simply push the container images to the registry, copy the relevant YAML files, and instantiate an identical cluster that way?
Efficiency is increased a great deal. Previously, I'd have idle instances costing money when they were unused. Now my resource consumption is amortized across my entire infrastructure.
It also drives an architecture that lends itself to better availability. A pod in k8s may be rescheduled at any moment. If it's a stateless service, one may just increase the number of replicas to prevent a service interruption. If it's a stateful service, one is forced to think about how to persist data and gracefully resume operations.
It took a while for everyone on my team to get familiar with it, but once we started to really grok it, the sense of safety went way up. I feel pretty good about the fact that if someone accidentally blew away a cluster with thousands of pods in it, that I could easily replicate the entire thing and get it back up and running in tens of minutes without a lot of hassle.
Also, my customers don't have strong requirements for network isolation. I'd be more concerned about compliance in areas like finance or healthcare.
I can't say that anything got worse. Once you (collectively) get past the initial learning curve and add automation in place, it's way better than most other deployment scenarios. Kubernetes worker nodes fail from time to time, AWS kills them and we barely notice. POds get rescheduled automatically and mostly work.
Frankly, the remaining headache-inducing things are mostly related to the software which is not running on Kubernetes (mostly stateful infrastructure, and mostly for non-technical reasons). Managing VMs is a pain.
There is one thing that one needs to be careful. If you are moving from a VM for each service, to Kubernetes, now suddenly your service is sharing a machine with other services, which wasn't the case before. So I'd suggest that proper resource limits be set so that the scheduler can work properly.
One thing that can get worse is network debugging. There are way more moving parts, so it is not as easy to just fire up wireshark.
So far zero problems.
The knowledge/documentation base just flat out sucks. It feels like documentation expects you to just know what you are doing and are really just reading it for the smaller feature switches to change how things work. That or expect you to be running on GKE.
That said, it's the best at doing what we need which is a scheduler and manager for running multi-container workloads.
I expect it to be more fulfilling/difficult when we move to more long-lasting pods but ultimately still use kubernetes.
P.S: if you’re interested in working on Kubernetes at reddit, send me a message at saurabh.sharma [at] reddit.com
We don't really have tremendous auto-scaling needs. But there are a few reasons why we did it.
We've since scaled to a small handful of services (frontend web app, backend node api, legacy stuff, custom deployments for high paying customers). We knew that standardizing on Docker was a good idea just to simplify the build process. What were previously service-specific, hard to maintain, hard to read, just weird shell scripts became a simple Dockerfile for each service, often only 5 or 6 lines long. Once you have that, setting up CI is ridiculously easy; AWS CodeBuild into ECS ECR took all of a day to implement. If we were on GCP it would have been even easier.
Comparatively, we'd spent longer than that actually maintaining the old scripts. The new ones require zero maintenance; they "just work", 100% of the time.
So we knew we wanted Docker. And that was a very good choice. No regrets. We start there. But once we had docker, our eyes turned to orchestration.
Its worth saying that there are a lot of nonobvious answers to questions in the "low/medium scale up devops" world. One big one we ran into early is "where do we store configuration?" Etcd or Consul are fine, but we're a pretty small shop; we didn't want to have to manage it ourselves. We could go with Compose or something. But, how do we get that configuration to the apps? We were in the process of writing new services and we wanted to follow 12-Factor and all that, so envvars make sense. But to get configuration from a remote source would break that, so we'd need some sort of hypervisor for the app to fetch that...
Additionally, how do you deploy? Let's say we go with a basic EC2 instance running Docker. We'd have to patch it together with shell scripts, ssh in, pull new images, GC old images (this was a huge problem with our "interim" step in the first paragraph, our EBS volumes filled up all the time, lots of manual work), restart, etc. Can you do that with zero downtime? Probably. More scripts...
Load balancing and HA? Yeah of course we can wire that up. More scripts. More CloudFormation templates or whatever...
Eventually you arrive at an inescapable conclusion: At a surprisingly low level of scale, you are reinventing Kube and Kube-like systems. So why not just use Kube? You get config management built in. You get a lot of deployment power in kubectl. You get auto-provisioning load balancers on AWS. You get everything.
And I could not be more serious when I say: It "just works". We've had two instances of random internal networking issues between our ingress controller and a service, which were resolved by restarting the ingress controller. That's... it, in a full year.
Like Docker before it, I think Kube is a foundational piece of technology that is only going to get more and more popular. Its incredible. But I do think that there is room in the market to make it easier to adopt. Best practices around internal configuration stuff is hard to come by; even something as simple as "I want a staging env, do i use two clusters or namespaces in one cluster" doesn't have a clear answer. Getting a local development environment cycle up and running is a pain in the ass. Monitoring and alerting is still pretty DIY; GCP solves some of this, but there's no turn-key alerting piece that I'm aware of. Logging is a nightmare if you are self-hosting, including on Kops; we ended up installing the Stackdriver agent and we use GCP for it, even though literally everything else in our stack is on AWS.
I literally don't believe there's a level of "production scale" your organization could be at where you wouldn't benefit from Kube. It is far far easier to set up than a bare deployment of anything if you do GKE. Connect Github to CI to Kube... good to go.