Hacker News Reader

Comment by idan | Hacker News Reader

Naive question from someone who doesn't know the ecosystem well:

It seems to me like Terraform is good at describing desired deployment shapes and detecting drift between actual state and desired state.

Can someone clue me into why Terraform hasn't caught on as the abstraction above/that drives K8S?

View on HN

Kubernetes is very good at keeping track of state and maintaining it. You "just" tell it the state you want and it makes sure it becomes reality (this is the whole point of k8s actually). Terraform is doing the same thing, just for imperative (instead of declarative) APIs (like public clouds, etc).

The issue with Kubernetes is more expressing the state. Kube uses YAML, which quickly becomes verbose and hard to maintain. More on our blogpost: https://grafana.com/blog/2020/01/09/introducing-tanka-our-wa...

Tanka is trying to solve this issue by providing a more powerful language that overcomes these limitations hopefully.

My last company used Terraform to manage Kubernetes. The main issue is that the TF Kubernetes provider supports a limited subset of K8S object types, and of fields within those K8S objects. For example: TF didn't even support Deployment objects until sometime in mid/late 2019 (I may be wrong on timing, but it was long after they were the primary method for general scheduling of long-running containers).

We ended up using TF's Helm provider, sometimes with hacks like a helm chart which deploys an arbitrary YAML file (the so-called "raw" chart). At that point, Terraform is blind to what's actually happening inside K8S. You can still benefit from the ability of TF to pass data from your other infra automation into the Helm charts, of course, but it's really Helm actually managing the configuration of your K8S cluster. And that's the app we all love to hate.

The situation may have been improved, but my conclusion was that it would always be a somewhat incomplete interface.

The terraform provider has caught up a bit in the last 6 months. It is still missing things like CRD support.

For those things we use a direct kubectl yaml provider.

I wish there was an istio provider!

Terraform is really good at describing how infrastructure should be provisioned (VMs, load balancers, dns entries, networking, etc). Provisioning software on a VM and keeping it in a consistent state, however, is not something it's very good at. Userdata is very difficult to do anything complex with (limited size payloads, optimized for uploading a single shell script), and the provisioner system is explicitly described as a "last resort". This makes Terraform not so good at describing how software should be provisioned.

There is a bit of a movement, however, behind using it to deploy software by pairing it with Packer. You use Packer to create an e.g. AMI whose sole job it is to run your software (like a Docker container) then use Terraform to launch a bunch of EC2 instances that have juuuust enough resources to effectively run your software. That'd allow you to eliminate k8s from your stack, though it remains to be seen which stack would be more cost-efficient to run on.

This is interesting... not heard of this. Any pointers/links to more info on this?

Check out https://www.terraform.io/docs/providers/template/d/cloudinit... if you want to do something more complex than a single shell script.

I do not understand what you mean by "how your software should be provisioned"?

I have about 40 kubernetes services all as modules using the kubernetes terraform provider. I think I have 1000+ pods running on our one cluster all deployed through terraform.

It works very well because I can chain infrastructure resources into my service deployments. For example, I can create a dynamically named bucket and pass the name of that bucket as configmap/secret into my service to use.

Reply on news.ycombinator.com