Setting up an HA Kubernetes cluster with private networking in AWS | Hacker News Reader

Setting up an HA Kubernetes cluster with private networking in AWS

nivenly.com

79 pointskris-nova18 comments

I wrote a similar article a while ago; except using Rancher - Which adds extra features for infrastructure management. https://blog.baasil.io/how-to-install-kubernetes-on-aws-d9fb...

Good lord this is so much easier than it was 6 months ago. Really glad to see the barrier of entry going down to get these clusters built. Kubernetes has been paramount to our engineering growth at FarmLogs.

Yah I agree. K8s install has been the bane of the projects existence.

Happy to know we are finally "here" with deploys/installs!

kops is pretty great for managing the lifecycle of a k8s cluster. It even supports outputting Terraform code, if that is your Infrastructure-as-Code tool of choice.

One of the best features of kops, is it's ability to be open ended in generating what the user wants! The private networking piece of Terraform codegen is right around the corner, and rumor has it, it will be used in some new API tests

ECS tasks now support IAM roles; how does this work for K8s equivalent? Do you need to use IAM user credentials for app credentials?

IAM credentials for ECS tasks are implemented with a transparent proxy to the metadata API, and there are similar projects for Kubernetes.

The one that's most popular I believe is kube2iam: https://github.com/jtblin/kube2iam

Using IAM roles then boils down to setting an annotation such as `iam.amazonaws.com/role: my-db-access-role` on a pod (assuming the correct trust relationship has been configured).

kops is nice... but too tied to AWS.

I'm really liking the direction that kargo is taking. It leverages a well known toolset - ansible - to build out its functionality.

kargo is compatible with everything from bare metal to AWS... with the caveat that when using it in AWS, you have to use a different provisioning tool like Terraform.

This is a feature.. not a bug.

Kops actually isn't tied to AWS, though we only support AWS right now. We'll likely be adding GCE support next; user requested features on AWS have taken priority so far though...

@justinsb - oh I didnt know that. For example, we are struggling to use kops on bare metal. However, I quite understand your AWS feature priority.

So right from the start I wanted to make sure it wasn't locked to AWS. We used to have GCE support enabled (and continue to have it on a branch, though obviously it has bit-rotted as we've added more AWS feature support) But we built GCE support early so we could be sure that the abstraction wasn't AWS specific. And we have bare metal support on another (more experimental) branch. Bare metal is much more challenging, largely because on clouds we can run etcd in HA mode with automatic instance replacement, which isn't really possible on bare-metal, so it looks we have to settle for HA with operator intervention when an etcd instance fails.

So I actually think kargo is a great choice on bare-metal - and it also does a great job of running on multiple clouds and bare metal. But the design of kops isn't particularly tied to AWS, though it does match better with clouds.

If you're looking for bare metal, have you tried using MAAS and Juju? https://jujucharms.com/kubernetes-core/

I don't know why projects like this don't use libcloud which allows you to use the many cloud platforms [1] through one API.

[1] https://libcloud.readthedocs.io/en/latest/compute/supported_...

Well I can't answer for other projects, but libcloud specifically is in Python, and kops is in golang. But you raise a valid question: why not use a cloud abstraction library like libcloud - for example embedding terraform. It would definitely short-cut things, but the trickier piece IMO is figuring out what cloud resources to create, rather than making the API calls. Embedding terraform might save a little time up front, but would also make it harder where terraform didn't work quite the way we wanted it to (the state store), or even if we hit any bugs. The Kubernetes experience with Salt was not overly positive in this regard (though this was likely more because of how we were using it than Salt itself).

It is possible to build a kops cloud implementation that doesn't do direct-to-cloud-API-calls, but instead just outputs terraform - that would be faster - but you would still have to figure out the right way to run Kubernetes on cloud X. Most of the work is in doing that!

I understand your point - just the vendor lock-in of AWS/GCP can be frustrating if you do not host with them.

There are also golang alternatives to libcloud one of them ->https://github.com/apcera/libretto

This is basically what we do at Cloudsoft. For the Kubernetes cluster deployment in Clocker [1] we use the features of Apache jclouds [2] and Brooklyn to support pretty much any cloud, public or private. At the moment, Clocker can deploy a Kubernetes cluster with an HAProxy load-balanced set of masters and an auto-scalable cluster of workers, using Canal (Flannel and Calico) [3] for networking.

1. https://github.com/brooklyncentral/clocker 2. http://jclouds.apache.org/ 3. https://github.com/tigera/canal

This is a nice write up, but we honestly needed to stand up kubernetes on-premise and in public/private clouds. This is why we've been using this: https://www.youtube.com/watch?v=B7nMFVaOOi8

Reply on news.ycombinator.com