Comment by stuntkite | Hacker News Reader

Comment by stuntkite | Hacker News Reader

I've since moved on to Kubernetes and SaltStack, but I got into production Docker containers with Deis[1] a year ago. With full Heroku buildpack support, Kubernetes backed, and great container tools. I feel like it's sad that this project doesn't get more love. It's a 20 minute deploy to GKE or AWS (I recommend GKE).

If you run OSX, TheNewNormal's solo and cluster Kube setup is a breeze and has the Deis Workflow as a one command install, then you have full local parity with deployment.

When I started with straight Docker figuring out the transition to the real world was be really daunting. Getting into a real production environment that I could compost quickly as I learned learn the paradigms was invaluable. Articles like this seem to buzzword their way into making the path muddier than it actually is.

Another big help for me was any time I needed to set something up, first thing, I'd look at how the bleeding edge was being done with Alpine. Practically you might not always run that light, but that community has all the answers.

[1]https://deis.com/ [2]https://github.com/TheNewNormal/kube-solo-osx [3]https://github.com/TheNewNormal/kube-cluster-osx

When I first heard about Deis it was heavily compared to Flynn [1]. It's a little hard to tell from the outside if the two are still competitors or if Deis is focusing on a lower level of abstraction.

https://flynn.io/

They both support Heroku-like buildpacks. Major difference IMO is that Flynn supports databases and other stateful services.

I evaluated Flynn, Deis and Dokku and wasn't entirely satisfied with any of them opting for Heroku instead.

I've promised people a blogpost on this and I will do it once I get time to evaluate RancherOS too.

Can you evaluate Kontena as well while you are at it?

Flynn was much less polished than Deis. At least, that was when i tried it last year.

I also started with Deis and move on to k8s later on.

>"I've since moved on to Kubernetes and SaltStack"

I curious how those relate to each other. Are you using SaltStack to bootstrap Kubernetes?

Not the OP, but we use Google Container Engine (hosted Kubernetes), with Salt for the non-GKE VMs.

This is needed because K8s is not mature enough to host all the things. In particular, stateful sets are still in beta. Not sure if I would trust K8s to run our production databases even with stateful sets. We've had K8s kill pods for unknown reasons, for example, and the volume handling has also been historically a bit flaky. Fine for completely redundant, stateless containers, less fine for stateless ones.

Sure that makes sense. Statefuf especially databases would make me nerovous as well. am curious, what are the advantages of GKE over deploying K8 on AWS? Or were you already GCE?

Our production setup is actually still on Digital Ocean; we're currently testing GCP (GKE + Salt-managed VMs) with a staging cluster to see how it works in practice.

Before GCP, we set up a staging cluster on AWS. It was okay. The biggest pain point is that AWS's VPC does not match Kubernetes' requirement for per-pod IPs, so anyone who installs on AWS ends up setting up their own overlay network (such as Flannel or Calico). That's a big downside, because VPC isn't fun to deal with.

You don't really notice how antiquated AWS is until you move to GCP. Everything — networking, UI, CLI tools, etc. — feels more modern, sleeker and less creaky.

One area where GCP is particularly superior is networking. Google gives you a Layer 3 SDN (virtual network) that's more flexible than the rigid subnet-carving you need to do with AWS VPC. The tag-based firewall rules are also a breath of fresh air after AWS's weird, aging "security group" model.

It's not all innovation, of course. Some services are carbon-copy clones of AWS counterparts: Pub/Sub is essentially SQS, Cloud Storage is S3, and so on, with only minor improvements along the way. Cloud Storage, for example, doesn't fix S3's lack of queryability. I've also been distinctly unimpressed with the "StackDriver"-branded suite of services, which do things like logging and metrics. I don't know why anyone in this day and age would bother making something that doesn't compare favourably to Prometheus.

I should add that the security situation on GKE could be better:

* GKE's Docker containers run in privileged mode.

* There's still no role-based authentication.

* Containers end up getting access to a lot of privileged APIs because they inherit the same IAM role as the machine.

* You can't disable the automatic K8s service account mount [1].

Another scary thing, unrelated to GKE, is that the VMs run a daemon that automatically creates users with SSH keys. As a team member, you can SSH into any box with any user name. Not sure if real security weakness, but I don't like it.

I love the CLI tools ("gcloud" and so on), much nicer than awscli etc, and hopefully much friendlier to junior devs.

[1] You can disable the mount by mounting an "emptyDir" on top, but that makes K8s think the pod has local data, which causes the autoscaler to refuse to tear down a node. Fortunately there's an option coming to disable the service account.

Is there a bug for the emptyDir autoscaler node tear down issue? That sounds really wrong by itself.

It's working as intended, as I understand it. A pod with a host directory is tied to the host. The problem isn't emptyDir, it's that GKE doesn't let you disable the service account mount.

Thanks for the detailed response. You mentioned "GKE's Docker containers run in privileged mode", do you why this is?

Not sure. Speculating here, but it's possible that it's because they're still working on building out GKE itself.

GKE is basically a wizard that just runs a pre-built image for the master and node VMs, and comes with some support for upgrades. There are very few settings [1] aside from the machine type. So it's pretty rudimentary. You'd think that GKE would come with a flashy dashboard for pods, deployments, user management, autoscaling and so on, but you're actually stuck with kubectl + running the Dashboard app [2] as a pod, which, while nice enough, is not integrated into the Google Cloud Platform web UI at all. Kubernetes runs fine, but GKE itself feels unfinished.

Anyway, a lot of people were asking for privileged mode back in 2015 [3], and it kind of looks like they turned it on by default rather than developing a setting for it.

[1] http://i.imgur.com/6pGRzl9.png

[2] https://github.com/kubernetes/dashboard

[3] https://github.com/kubernetes/kubernetes/issues/12048

>"GKE is basically a wizard that just runs a pre-built image for the master and node VMs..."

Interesting. So the master and the VMs are deployed in KVM or Docker containers running inside of KVMs then?

> Pub/Sub is essentially SQS

This is just wrong

> Cloud Storage, for example, doesn't fix S3's lack of queryability.

And this is now fixed in S3

Pub/Sub is feature-equivalent to SQS. There are improvements (e.g. gRPC, better performance), but it's still a basic topic/subscription system with a very simple model; for example, just like SQS, it doesn't support fanout.

Is S3's queryability fixed? Are you referring to the Inventory service or Athana? Because the former is just a big CSV file generated daily, and the second is for querying content. You still can't query the inventory of objects (e.g. find all objects created yesterday whose key contains the letter "x").

Okay, with S3 I was referring to Athena.

SQS is a "Simple Queue", Pub/Sub is not. There are no topics in SQS, just distinct queues. There is no 1-to-many. There is no push. And so on...

SNS is Amazon's push notification service and any number of queue's can subscribe to an SNS topic for 1-to-many.

I'm actually right in the middle of putting that together as well. I quickly arrived at analysis paralysis when choosing how to bootstrap Kubernetes and run it in production given the tremendous flexibility. Tons of options[1], but nearly all of them are incompletely documented. Ended up walking through Kelsey Hightower's K8s the hard way[2] repo to peel it all apart.

End result will probably look something like Terraform -> SaltStack -> Kubernetes.

[1] https://kubernetes.io/docs/getting-started-guides/

[2] https://github.com/kelseyhightower/kubernetes-the-hard-way

That's exactly how we started: manually bootstrapped a cluster through "Kubernetes the Hard Way", assembled the steps from there and what else we found during discovery out to Ansible playbooks and now we are contemplating kube-aws to setup the production cluster in AWS.

Reply on news.ycombinator.com