Hacker News Reader

Comment by kleinsch | Hacker News Reader

Surprising that the author is writing a tool for managing your servers, so writes a post about how Terraform isn't great at managing your servers...

It seems like the root of the problem is that the author wants to use Terraform to manage their AWS state, but also wants to use the web console to directly change things, so Terraform gets out of sync. Terraform has a command to handle this - https://www.terraform.io/docs/cli/commands/refresh.html

View on HN

As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human.

The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up.

One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.

"every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language"

I constantly see soo many people step on the same rake, it's incredible. Tools like Tilt let you use python, it's a much more sensible approach.

The reasoning you put in also means kubernetes is unsuited to be controlled by terraform. Too many lifecycles (resources) to centrally control. kubernetes custom resources can have dependencies on others which terraform either needs to support as well. Which is not doable to maintain.. keep your kubernetes manifests outside or your terraform state.

My company manages a lot of Kubernetes manifests with Terraform without issue. Terraform is just generating the manifests in this case; Kubernetes is doing the reconciliation work. More complex than is ideal (i.e., if we were starting out with Kubernetes we probably wouldn’t use Terraform) but it works reasonably well.

The Hashicorp co-founders considered alternative approaches when originally designing terraform. The actor model was considered but dismissed. That's not a million miles away from the kubernetes reconcile loop model:

> Then we transitioned to an actor-based model where each resource was almost an actor, and there was a message-passing interface between them.

> This allowed the system to be highly concurrent the way Terraform is today, but also confusing for users to deal with and very difficult to build a programming model around, because the ordering of execution was so random and everything was happening concurrently.

https://www.hashicorp.com/resources/terraform-fireside-chat-...

They may still be right. Kubernetes' approach may seem more attractive but terraform is far more pragmatic in its design.

There are a lot of developments around using Kubernetes as an IaC platform for the reasons in your comment. The combination of a standard API model in CRDs + the controller model maps nicely to managing infrastructure and exposing resources to developers.

<https://crossplane.io> just graduated to CNCF Incubation and each of the cloud providers are working on K8s controllers and code generators (like Amazon Controllers for Kubernetes, Google Config Connector, and the Azure service operator).

Terraform accepts JSON format as an alternative to HCL.

I prefer CUE to JSON for TF and many other tools now

To be clear, the issue isn’t the HCL syntax. You could similarly use Cue to generate HCL. The problem is using Terraform’s dynamic features which were poorly designed.

Terraform is also working on allowing actual code: https://github.com/hashicorp/terraform-cdk

I know, I mention that in my comment :)

There’s also a refresh flag on plan. It’s always worth using before your apply CI step.

Reply on news.ycombinator.com