$15 Production Kubernetes Cluster on DigitalOcean | Hacker News Reader

$15 Production Kubernetes Cluster on DigitalOcean

5pi.de

219 pointsdiscordianfish71 comments

How many people are actually orchestrating a fleet of containers vs. pushing simple apps to a single box or two with Capistrano, Maven, or Fabric? I don't understand the popularity of Kubernetes and Docker relative to the aforementioned simpler technologies. Certainly some organizations have use for container orchestration, but are startups beginning with that vs. just simple deployment scripts in hopes of scaling up? What's going on? Why are these in the news so much?

Containers are also about development and testing, afaict.

I work for a 100K+ company and we are doing that, though far from google scale. I talk about this here:

https://www.youtube.com/watch?v=Yj5I-yOaDao

and blog on related things here:

https://medium.com/@zwischenzugs

but I do understand your skepticism - the hype is far from reality.

Something really simple like capistrano works usually only for a small subset of your services. With Kubernetes, you can run everything in there. And you can automate it to a very high degree: Imagine pushing a branch not only deploys it to some test system, but it actually bring up depending databases, a queue and a load balancing layer which then gets load tested.

This is partially possible in the cloud as well, yet it requires building machine images which is slow. And you end up with a provider specific solution especially if you try to integrate it very well.

But in short: It makes sense very soon if you run on bare metal (and arguably makes it more feasible option to run on bare metal) while it makes only sense much later if you're running in the cloud. I'd say you should have at least more than one team deploying more than every few days.

I'm not super familiar with Capistrano, but you can provision systems and databases with Fabric. I'm still very doubtful of the utility of these more heavyweight technologies relative to the simpler ones for all but a very small subset of organizations. To me, it reeks of people trying to make simple things much more complicated. I haven't used Kubernetes. I tried docker and never understood why people would use it over an ordinary VM and some reusable deployment scripts.

Kubernetes isn't just about deploying. It comes with central config store, secrets store, service discovery, load balancing, routing/ingress, health checks, rolling updates for your software, auto-scaling, a powerful storage volume system, and a bunch of other things that you don't have to do a crappy job reinventing.

It doesn't make deployment easier, per se. It gives you sane, scalable ways to solve deployment, credentials, logging, autoscaling, infrastructure provisioning, etc etc so you don't have to solve them yourself.

Overall I'd say it doesn't solve anything that was unsolved before, but it solves a common problem in a palatable way.

Part of it is that there's a feedback loop: as Kubernetes + Docker become more and more popular, the tooling around them gets better and better.

Also, I find Docker a lot easier to work with than trying to manually maintain parity between my dev environment and production hosts. Having previous "pushed simple apps to a single box or two" I'd never go back, even for the simplest of apps.

I work for a company that has one team which just has 20 nodes, all running a docker daemon and they push stuff to it using Capistrano .. mainly really old Perl containers.

Our team uses CoreOS and we're slowly moving over to Mesos/Marathon. Our team also has really streamlined unit testing, integration testing and deployment. When you need to deploy an app that may need 4 or 5 nodes, and the rest of the company also is deploying microservices that need anywhere from 2 ~ 20 notes, having a system like Mesos or Kubernetes is essential.

They are big systems. They can be difficult to maintain. I've been trying to setup my own mini Kubernetes in Vagrant and have struggled quite a bit. This post really goes to show a large amount of the complexities too.

It really depends on your scale. If you start growing, investing in people to maintain a container infrastructure is pretty key.

could you talk about how you are using capistrano to push to docker ? i thought you had to go the whole continuous-deployment complexity to do any such deployment.

I don't think many are using it now, but with the out-of-the-box Docker web app projects coming in VS2017 I bet that number will go up. It's a much better proposition than WebDeploy.

I agree that this is a lot of complexity and seriously doubt the popularity of it in real use, especially with all the quietly productive companies out there.

Putting a binary on a VM works just fine and is perfectly repeatable and scriptable already.

Why does Rancher[0] never get mentioned on here? I've been using it the last few months and it's been great. It handles everything for you without the huge learning curve.

I'm using Gitlab.com pipelines also and it means my CI/CD pipeline is now free, and once by build/tests are run it pings the Rancher API to upgrade the service.

[0] http://rancher.com/rancher/

We have been using Rancher as well... It allowed us to move away from DO and AWS. Now most of our infra is from OVH :).

It's been smooth sailing.

Because of massive costs savings we were able to just reinvest it in our own redundancy. Also 12-factor apps are pretty damn resilient.

That's cool. I've actually just moved from ESXI hosts to OVH too.

How are you handling persistent storage? I have opted to not "dockerize" Postgres for example after reading horror stories.

Because theres no perfect solution at the moment we have opted for a 4 disk setup, and mirror them all using ZFS. We also do 10 minute snapshots, and incremental snapshot backups (to S3).

Feels safe enough for our needs. Before this we didn't have 10 minute snapshots of all our data so this has actually allowed us to recover from dev errors as well (if they were to happen).

We do have our DB running in a container + mounted volumes that use the above configuration.

Which kind of setup do you have i OVH? do you rent servers or VMs? Do you deploy docker? via kubernets or what? quite interested in this, if you have material/links on OVH and docker/kube I would not mind if you can share.

We currently just rent a few big boxes on OVH, and manage them via Rancher.

You should checkout ranchers channel for some videos: https://www.youtube.com/channel/UCh5Xtp82q8wjijP8npkVTBA

It's mind-blowing that they offer this for free, considering the impact it has.

could you talk about whats your underlying platform? are you using k8s or mesos or something.

k8s currently has incomplete support for metal deployments. for example the integration between ingress and load balancers on metal vs cloud.

im kind of wondering how people are deploying them in real life.

secondly, how do you deploy new code?

As above, I have VMs on OVH public cloud[0]. Rancher can also offers an OpenStack driver which is what OVH Public Cloud runs on, but I have not yet tried that.

I've found certain things quite buggy with their cloud offering but the price/performance can't be argued with.

[0] https://www.ovh.co.uk/cloud/instances/

OVH don't expose OpenStack APIs so don't expect that to work.

I have used the nova command line tool to manage some instances, is that not using the OpenStack APIs?

Rancher is great. I am curious what was it about Rancher specifically that allowed you to move away from DO? I found that intriguing, can you elaborate?

It looks quite interesting.

Though the requirements for a multi-server setups are quite large IMO.

I was using rancher, but it is seriously memory hungry (mine seemed to need 2G) so you would need a $20 Digital Ocean box just to run rancher. Also, the agents / monitoring rancher uses was eating quite a lot of memory / CPU on each host.

So he basically followed it's the future story? (https://circleci.com/blog/its-the-future/)

Now he can finally deploy WordPress.

You did not read it lol, he deployed Ghost!

You caught me :( But close enough. Point being overkill.

While funny, there are uses for the types of orchestration that containers can provide... for me, it's kind of awesome that finding a script for an automated deployment of any subject is now as simple as searching: <subject> dockerfile

And voila, I have the steps to install <subject> ... ;-) I'm being a little snarky, but I do think that the dockerfile is pretty cool by itself, and wouldn't even mind seeing other automation frameworks use the dockerfile and the swarm configs for other purposes.

For sure! But please, for the love of Vint Cerf, don't go into production with boilerplate containerization configs.

I usually just start with the published dockerfiles and modify as needed... Really depends on my own usage.

Well, it is funny, because after all that he just installed Ghost on it.

We are using containers for pretty much same purpose - try it out different things and just throw it onto one of our dokku boxes - fast and easy way to play with something without investing heavily.

Lol. I was thinking similarly. If this is what you need to do to get Kubernets up in production, Kubernetes is far from production ready. Sadly, this type of UX is the norm in the ops world and everyone is expected to build out a custom in-house solution on top even though everyone is doing the exact same thing. AWS was built on this insane model, promising companies that they don't need operations when the need was shifted from hardware onto software, not eliminated. The devops movement has decided it is also fine with this as long as they can automate the dozens of moving parts using dozens of other moving parts. Surely nothing will fail then. Even products that manage a really great UX like Docker itself fail when it comes to providing a proper UX for higher level tools like docker swarm.

I'd like to think that the industry is moving forward towards better, higher-level abstractions (like Kubernets or Heroku) but it seems to me it's only moving sideways, just like it did with the adoption of VMs.

For what it's worth, CircleCI is going all-in on containers: https://circleci.com/blog/say-hello-to-circleci-2-0/

Pretty much! :)

No, for real: It looks like I failed to communicate the intent very well.

I do NOT recommend setting up such infrastructure for running Ghost. I do recommend such setup (not necessarily on DO though), if you have a sufficiently complex stack and you want to be able to deploy to it quickly. This is often the case if you're have multiple teams who want to deploy independently and you can't wait for building and spinning up VM images. Or if you're running on bare metal and can't get away with something like Docker + a few shell scripts.

In my case I just run this way-to-complex-stack because I've worked in the space (at Docker and Mesosphere to be specific) and want to stay in touch with the space.

Flynn offers a UI installer that works with Digital Ocean.

1. Login to DO and get your API keys

2. Select what images you want and how many instances

3. Click install and Flynn will setup your load-balanced environment.

4. Push apps like Heroku

I've been evaluating dokku, Flynn and Deis. Just rancherOS left to do.

I would love to hear you overall review of everything. I'm really into all the options. Dokku seems too simple for the stuff I want to do. Flynn is pretty awesome, but I've been loving the steady progress Deis makes. Rancher is also interesting.

I'd love to know what was the smoothest process for something like a phoenix/node/rails app out of these. Feel free to email me at jbhatab@gmail.com if you feel like breaking it down over chat too :D

Just curious, outside of multi-host support - I'm not in the business of writing a scheduler, so we'll most certainly use an existing one if we ever get to this feature - what is too simple about Dokku?

I'm going to write a three-part blog posts that discusses the strengths and weaknesses of each system. I'll email you when I'm done.

We use Cloud Foundry in production and we've used Dokku in development. CF is similar to Dokku and Flynn as it can provide services (databases etc) and wire them to your container.

Everyone running hobby projects should be using Dokku. It's very easy to get started and can replace Heroku in many cases. It is missing HA but you can always scale horizontally or load balance multiple dokku nodes (behind say nginx/haproxy) against an external db/redis although you are essentially running commands for each node.

Dokku should look into Docker Swarm to do orchestration but that is a mammoth task for a pure open source project (no support business to fund it) and as josegonzalez mentions he needs the cash to be motivated to do it. Dokku also lacks a stable way to ship logs to external services. dokku-logspout is available as a plugin but it frequently loses it's connection and needs to restart. But then excels past any of the other PaaS solutions when it comes to SSL integration via Let's Encrypt. It's a truly remarkable project.

Flynn provides services and sets up Postgres to balance between nodes so it too is HA. Flynn works with Google, Amazon and Digital Ocean via it's web installer (I had trouble getting it to run on a third-party provider, SSH was broken). Flynn provides Postgres, redis, MySQL and MongoDB out of the box and HA but with the caveat that you can't really customize these services. I also killed one of the three nodes via the DO console and it still carried on with no issues.

Deis was the hardest of the bunch. It essential is a manual process to get Kubernetes up and running unless you use the cozy Google Cloud Container which I did but isn't cheap ($100/month before bandwidth for 2 containers). Once up and running it is as easy as the other two. They don't have any services though - people suggested running either running Postgres in a container. I think a PaaS should include the DB but that's debatable.

Things I haven't tried:

- adding a node to Flynn or Deis once up and running

- killing a node on Deis and checking if it still runs (I assume Kubernetes takes care of this)

- incorporating these systems with Gitlab to provide a full continuous delivery system

Dokku + Swarm actually isn't too hard (at least old swarm). I wrote a prototype in ~10 lines of code changes this past summer.

As far as shipping logs, I feel as though thats the responsibility of the underlying container environment, and so I'd probably have something that is based on setting docker options for containers (or an interface to do that for whatever scheduler you are using).

This is a great guide! Very thorough. However there are so many moving pieces it makes me think I should just use a hosted Kubernetes solution like GKE instead of trying to roll my own...

Shameless plug: I also blogged about what you should roll on your own and what not: https://5pi.de/2015/04/22/scope-and-ownership-in-tech-compan...

Continuing some of the feelings here in the comments. If you want to run Ghost/Wordpress or any other container for that matter, there are much simpler orchestration tools than kubernetes.

All of my blogs are hosted on a DO machine, all of them are dockerized and I "orchestrate" them with simple chef scripts.

For this simple workload, I would go with swarm or something as simple as bash with upstart.

That is pretty much what dokku is, with some additional bells and whistles.

Yeah. Point is. You don't have to go all Googly on your infrastructure, start simple.

Deploying wordpress/ghost to DO takes 10 minutes.

Actually, probably a bit less time if you use the project I recently wrote: https://github.com/josegonzalez/dokku-wordpress

Note: I am one of the Dokku maintainers.

Author here: Yes, this is not a setup for running Ghost/Wordpress. It's for getting started with a production(ish) kubernetes environment which would be a complete overkill for a simple stack like the one I deployed on.

I've been working on setting up something similar up, but it's a pain because I'm trying out a fork of kubernetes with support for digitalocean volumes. The author has a PR upstream and it looks like it'll probably get merged.

I've played with kubernetes on DO in the past and had a lot of trouble with the controller node using more than 512mb of ram. Maybe I was running into a bug, but I'd be more comfortable with a 20 dollar cluster running a $10 controller node.

I really look forward to more provider volume support landing upstream. If we end up with support for digitalocean, vultr, and packet volumes, we're going to be in a great place. You can run a whole lot on a 200 dollar packet cluster. Maybe most people use more of AWS than I do, but blockstorage is really the final thing I need kubernetes to abstract over for the provider to not matter to me.

I've wrote a simple flex volume driver in bash which is included: https://github.com/5pi/infra/blob/master/packer/files/usr/li... but will probably rewrite it in golang once the new flex volume spec is final.

I actually hadn't heard of flex volumes until now, interesting.

If you're interested here's the digitalocean pr upstream: https://github.com/kubernetes/kubernetes/pull/36894

I've had a few questions about Kubernetes and production grade environments and container orchestration.

1. Why Kubernetes? (when mesosphere/dcos is available?)

2. Any available best practices when setting the entire environment from scratch? (right from server level security, to doing data backups for example)

---

As a critic, I thought that if you were running it in production, you'd need 3 masters? (to ensure that if the master goes down, another takes over?)

So with 3 masters, and 2 slaves...the production cost go up to about $25/m. I feel that's too expensive for what you do get (price vs performance) compared to others (I personally feel a production environment running 5 servers from packet.net @ ~180$/m is the way to go [0])

[0] - https://www.packet.net/bare-metal/servers/type-0/

Why DCOS if swarm is available? Not trolling, just want to put my head around it too.

From what I understand, Swarm is not yet in it's prime state and seems to have a lot of issues that are not around. DCOS is based of mesos which is supposedly battle tested (I know it's the same for Kubernetes)

Also, at least in mesos, you can set up time scheduled tasks which I don't think is possible with Swarm.

I commented about DCOS vs Kubernetes many times before here on HN.

I had a production cluster running on mesos/marathon (mesosphere). Also have an open source project heavily invested in it [1].

Recently, I refactored the entire environment to work on Kubernets and the open source project is following it as well (It is basically a miror of my production ENV).

After this long story I will say this.

They are both hard to set up correctly, especially if you follow best-practices of closed networks and publicly accessed resources (VPN, Internet Gateway etc...).

Kubernetes has a lot of "force" around it right now with the community booming. Posts like this (with it's many problems), is somewhat of a proof becuase it shows a vibrant community around a project.

Kube is more stable from my experience.

Like you mentioned, I would not run Kube production on anything less than 200-250$ which means multiple masters and multiple slaves for the pod runners.

It's nice to run something cheap to check the capabilities but not more than that.

Regarding DO, I'm a big fan and run a lot of projects (personal) with them, for production grade stacks I would use AWS/Google/Azure, those give you better capabilities around hardening a cluster (security, networking) for production workloads.

[1] http://docs.the-startup-stack.com

this is very interesting! can you talk about the problems you found. Im not sure what you mean by pod runners.. are you referring to kubelet here?

we are considering a k8s based stack.. but have seen very few on-metal deployments with k8s.

I'll bite:

> Why Kubernetes? (when mesosphere/dcos is available?)

Frankly, if you're familiar and comfortable with DCOS, the core financial benefit is the same: increased hardware utilization, assuming your workload can realize that benefit. And if your application is relatively monolithic and handles the same steady load, I'd advise to stay away from advanced orchestration tools because they're only add moving parts and more complexity, unless you have [1]

> Any available best practices when setting the entire environment from scratch?

For education, I would recommend Kelsey's excellent "Kubernetes the Hard Way": https://github.com/kelseyhightower/kubernetes-the-hard-way

For production, I would recommend Telekube: http://gravitational.com/managed-kubernetes/ It does a lot of extra heavy lifting to make sure your k8s feels like always-on black box (I work there).

[1] Kubernetes also has a "side" benefit of abstracting the underlying infrastructure away, so if you have a use case of running the same complex stack on AWS, colo, vmware, etc then running it on top of a "cloud abstraction layer" lowers your ops/dev costs significantly. We talk about this here: http://gravitational.com/telekube/

Thanks for the links, I had to setup a Mesos cluster on ~200 machines for my university (which is what got me interested in DCOS in the first place).

I was then trying to replicate the same in the cloud but with more constraints (security, networking etc.) and found both Kubernetes and DCOS complicated.

Recently Azure seems to have DCOS (and Kubernetes) as it's orchestration tools for Azure Container Service and I wanted to learn more about each.

If you're already on Google Cloud, and wanting to deploy a containerized app, then the side benefit alone is probably worthwhile because Kubernetes takes care of dealing with both Google Cloud and Docker, with little or no downside.

You could always get started with something simpler like dokku (don't run on the $5 DO plan, and make sure to setup a swap file). In the end, it is pretty cool. Though, I'd just assume Docker's (the company) administration tools for DO become even better, as I'd love to "self-host" with Docker mgt tooling, and deploy my apps/services.

Easier setup and flow with Let's Encrypt is also essential. In the end, I think it's interesting, but overkill for a lot of scenarios... In a prior role, we went with 3 instances of dokku, with everything (but DB) deployed to all three instances, and an LB in front... was easier than all the orchestration tooling at the time.

1) Simply put: Because I believe it's the best option both right now and from it's prospect. You might look into mesos(phere) if you want to run other workloads than containers for which support on mesos is more mature

2) This setup is done from scratch, so given all the constraints this is what I would do. Then again, I'd probably care little about saving a few $$ by using DigitalOcean and look into deploying it on AWS or GCP which is more mature you don't need to use terraform which could be more reliable..

The cluster I describe has 3 nodes, all running controller and worker services.

But again: Please don't use this setup 1:1 for production. If you really just want to run a blog and stuff, there are better options.

One thing I hear a lot when people talk about Kubernetes vs Mesos is the developer experience. Such that a developer can run a kubernetes cluster on their laptop to work with "easily" vs it being essentially impossible with Mesos.

Okay, so there's 1.5GB of RAM total in the cluster. How much of that is taken up by the OS and how much is taken up by the Kubernetes components? Can I reasonably expect to run more than just one application on this setup?

Not much but this is what I'm running: https://github.com/5pi/services Basically postgres + two ghost instances. I'm actually surprised it works that well given the constraints. With Cloudflare in front and agressive caching it even survived this HN posting without issues (at least none I'm aware of).

But sure, it's overkill if you're not mainly interested in Kubernetes itself.

I notice that credential/secret management is kinda lackluster in this otherwise pretty sweet looking setup. It would be great to have Vault thrown in too for good measure. Maybe a followup article?

Yes definitely an interesting topic. Next follow up article is probably around testing though.

Very interesting. I'm currently working on a nearly identical project, except I started what you're only considering: Using CoreOS and cloud-config (plus some small extras). It's not done yet, but if you're interested, have a look: https://github.com/m3adow/coreos-kubernetes-tinc-cluster

I think k8s is "cheating" the private cloud story quite a bit. There are very few on-metal production deployments using k8s. This is because some of the harder problems of setting up load balancers, multiple masters, etc has not been really solved.

in fact, just yesterday, there was a new post on the group to create a SIG-Metal ( https://docs.google.com/document/d/1oYtW7fgSJsQDl-ln6ETvAQrN...) because there's so less information.

load balancer integration is written for all cloud providers... but not for metal (https://github.com/kubernetes/kubernetes/tree/master/pkg/clo...). there's experimental integration with haproxy/traefik in contrib... but not in production.

the reason is evident - a lot of the difficult parts of k8s is being "outsourced" to the underlying cloud platform.

k8s needs to fix this fast... because the real disruption is being able to eliminate a cloud platform (or build your own).

that sounds bad to say the least. Isnt that the entire selling point, rolling your own cloud... I always wondered why Azure and AWS were seemingly more of a priority for a software solution that supposedly is made to replace the cloud or compete with exacxtly those services.

What people call production nowadays...

You can also use kubeadm to bring up a cluster really quickly. We use this in production.

Reply on news.ycombinator.com