Hacker News Reader

Comment by kshahkshah | Hacker News Reader

As someone who stepped out of the devops world for a minute and is now trying to convert my companies infrastructure to use these tools, this is very useful and I'm reading through the whole thing.

However, I'm still confused by how the tools in the ecosystem interact with the capabilities of various cloud providers. That is, we're using DigitalOcean and Docker and I want to get our infra to a point where I can easily spin up a brand new staging environment (staging-2 say) with an isolated postgres node using an attached volume which also say runs redis, a proxy node, a couple of nodes for application servers, and a couple of nodes for background jobs, all w/private networking, secure, w/non-root access, and run a quick task to seed dbs.

I just can't seem to find guides which put the whole thing together, just pieces and I'm lost in researching an overwhelming number of tools from Ansible to Terraform to Kubernetes to Helm, etc, etc, etc.

View on HN

I'm not a very versed DevOps guy, but I have used Ansible, Terraform and Kubernetes.

It should go a bit like this:

- Use Terraform to provision VMs, networking resources and storage from DigitalOcean. Basically, write the scripts that make your whole infrastructure available with a single command.

- Then use Ansible for anything you might want installed on those machines - Kubernetes, security packages, SSH keys for your team.

- Use Kubernetes to then deploy your application on top of your secure and replicable infrastructure.

Each of the steps above should be roughly one shell command. If you are disciplined enough to always provision machines, install packages and deploy your app via config files, this should be very much achievable.

>- Use Terraform to provision VMs, networking resources and storage from DigitalOcean. Basically, write the scripts that make your whole infrastructure available with a single command.

How does autoscaling work if terraform/ansible are involved before kubernetes can take over?

IMO it's better for autoscaling to bake VM images and use cloud-init or similar to add per-instance config tweaks if necessary. Using Ansible to do runtime configuration is slow and doesn't play well with autoscaling. I do like Ansible, however, and use it for building EC2 AMIs. I even wrote a tool for this purpose at https://github.com/cloudboss/bossimage.

There are generally two levels of autoscaling involved with kubernetes.

Firstly, kubernetes is able to create multiple instances of your app as load increases and scale them out over all the nodes.

Secondly, the nodes your kubernetes cluster is running on can also autoscale. With terraform you can set up an AWS autoscaling group for example to automatically increase the size of your cluster as load increases.

Hey just curious:

With a dockerfile is there any point to using Ansible/Puppet/Chef any more? I have used those tools to keep 'how machines are installed' in code in the past, but with dockerfiles these days I don't get the point anymore. Seems like the future will be having a few dockerfiles (staging, prod, dev, db, etc) then just finding a place to run them (digital ocean, docker swarm, aws container service - or your macbook).

You still have to bootstrap the instances/droplets/etc with Docker, Kubernetes, etc before Dockerfiles can be used. Unless you're starting with images where those are already baked in, Ansible/Chef/Puppet are good ways to accomplish this on a clean OS.

It's debatable. Ansible/Chef/Puppet themselves need to be bootstrapped as well.

Bash is perfectly fine if the machine can be installed with less than 100 lines of code and immutable infrastructure is being used (so no need for idempotent operations). Just make sure to add ShellCheck to the CI pipeline.

Ansible/Chef/Puppet themselves need to be bootstrapped as well.

Nope, Ansible just needs SSH access.

Remote hosts need Python, but you can use the raw command functionality to figure out what the remote package manager is and install Python as required.

Not when using AutoScaling Groups. Then the machine will be provisioned on boot and therefor ansible would have to be installed on the machine.

No, you'd use Ansible to prepare the new image once, then you'd use that image to scale. Bootstrapping everything each time a VM is created doesn't make much sense.

> Bash is perfectly fine if the machine can be installed with less than 100 lines of code

It is always tough to predict the future; something < 100 lines can easily grow over time. It is nice to start with a tool like Chef/Ansible/Puppet which can more easily grow as the project grows vs Bash.

I agree that it's hard to predict the future, that's why I'm not trying to add unnecessary dependencies. 100 lines of code are not hard to rewrite in anything else if necessary.

I'm doing just that (for a SaaS).

1. Build an AMI with Packer that contains Ubuntu + Docker + aws-cli.

2. Build and publish docker images for all the services that will run on the instances. (one image per service, not per environment like you suggested)

3. Configure the EC2 instance or AutoScaling Groups with userdata. The userdata usually just contains a `docker run ...` instruction.

I used to have Chef, run test kitchen, bake AMIs, cycle all the machines. Now I have a single AMI and do most of the testing on the docker side and then some on staging. In both cases I am using terraform to manage the overall infrastructure (and userdata scripts).

The only issue is how to manage secrets as userdata are readable with the DescribeInstance ACL. The best solution that I found is to upload the secrets on S3+KMS and use IAM Role on the machine to retrieve them. Or use Hashicorp Vault.

For single-provider solutions, I'd say that's debatable. You might be fine with <insert cloud provider here>'s PaaS spin-up services for Kubernetes. The instant you want to span multiple cloud providers, you'll rub up against all the minor differences between each of their implementations, and I think that's where the value in SaltStack/Ansible/Chef/Puppet is in modern times. Manually orchestrate your infrastructure once, then deploy it to any cloud provider.

I moved away from dockerfiles to Packer to build images for multiple environments. Even if you don't need that, ansible from dockerfile is a good choice to avoid writing hard to manage/reuse scripts in yml

I don't think it's a good idea to have different dockerfiles for your staging, dev and prod environments.

The point of using Docker containers is that you get to deploy to production your dev environment as is.

No, you don't need those tools anymore.

Cook a base VM image that has the docker engine and anything else you need. Do this once and spin up VMs you need after that. This is what images are for, and they're far easier, faster, more reliable and more secure than dealing with all this other junk.

How do you do the cooking of the base VM image?

All the major clouds already have API commands to convert a VM into a reusable image - so a simple startup script + this API command is all you need. Even better since you can script all this too.

If anything changes, update the startup script, then run this "make new base image" script and you're all set. Update VMs as needed. Really doesn't get much more complicated than that and using tools like ansible just to setup your VMs so you can run docker containers later is unnecessary.

You make Ansible sound like an ICBM launching system. In practice, you do a simple "apt install ansible" once (on your machine) and then you write a simple startup script in it. There's nothing extra to be "unnecessary", it's a 1:1 replacement.

Why do you need that then when you can just use the startup script directly with the VM?

Unnecessary = not needed to function. Replacing functionality is just preference, which is fine, but specifically not necessary.

Why do you need that then when you can just use the startup script directly with the VM?

My point is that's a loaded question. You could just as easily ask why do you need bash in every VM when you can just have Ansible in a single machine.

Check out Packer (https://www.packer.io/intro/)

Could you point to any articles that go into a bit more depth on this provisioning-configuration-deployment flow?

You can use Ansible for step one, too. And I think I'd recommend it highly over Terraform for production provisioning.

> I think I'd recommend it highly over Terraform for production provisioning.

I'm curious why you recommend this? I'd argue the opposite. With the declarative nature of Terraform, you can know exactly which part of your infrastructure is going to change before you actually apply it. With Ansible, this information is a bit more opaque. You basically just run the playbook and pray it goes as planned and things haven't drifted too much.

You can run Ansible in check mode and see what steps will make changes.

If you (or someone on your team) is truly enjoying assembling DIY solutions using the best of what OSS has to offer, the possibilities of how to accomplish this are endless.

But for a single-command fire-and-forget commercial solution, take a look at http://gravitational.com/telekube

It allows you to, basically, define your own "Kubernetes flavor" with pre-installed components like CI/CD pipeline, databases, redis, etc, call it "developer cluster foo" and have X replicas of it running on any infrastructure. Telekube runs a single process per box and manages dozens of parts (kube services, docker, systemd, terraform) automatically for you, does atomic in-place version upgrades of the entire stack, etc. It is basically a "kubernetes appliance" and that's how many of our customers use it: ship it in a box into on-premise environments.

Yes, yes. It was a shameless plug since I work at Gravitational. We believe that DevOps folks have better things to do that spend any time provisioning Kubernetes and keeping it (and a dozen of components it now consists of) alive.

Interesting!

Do you document what security options you use with Telekube anywhere?

One of the things I've noticed with the wide variety of k8s distros and installers that there are is that it's not always easy to establish what they've chosen to do in that regard, and as some of the options like RBAC are newer and have some operational overhead, not every distro uses them.

Yes, the security portion of our clusters is handled by Teleport, which is OSS and its architecture is well-documented here: http://gravitational.com/teleport/docs/2.0/architecture/

Teleport maps your corporate groups/roles to both SSH and Kubernetes.

thanks for that. I was perhaps more thinking of areas around how you configure kubernetes. for example, do you use RBAC, do you configure kubelet authentication/authorization, what certificates are used by the API server for etcd authentication? that kind of thing :) If there's manifests/pod speccs for the components that'd probably have the answers...

Amusingly, much of the backend of digitalocean runs via kubernetes. They way they do k8s secrets in hashicorp vault is pretty nice actually:

https://blog.digitalocean.com/vault-and-kubernetes/

You might enjoy this guide[1] to get started. It discusses how to run a cluster on virtually any cloud provider and offers fully automated provisioning using Terraform[2].

[1] https://github.com/hobby-kube/guide [2] https://github.com/hobby-kube/provisioning

Do you need to run staging-2 in a complete new cluster? If not, maybe namespaces can be what you need.

I would very strongly recommend this approach. It's worked quite well for me.

There are so many ways to split out application environments in k8s (namespaces being one, generated prefixes in a helm template being another) that you should really try to use your cluster as a single compute resource. Maybe segregate for prod if you're not confident in your ability to lock it down, but everything else should just be creating/scaling pods, then adding nodes to the cluster when it runs out of compute.

Agreed! The documentation is sparse, mostly blog posts and frequently out of date. I am finding some success with terraform style kubernetes cluster setups - there was a good one I came across for DigitalOcean that helped me trying to provision to Scaleway

Reply on news.ycombinator.com