Writing an IaC Rosetta Stone | Hacker News Reader

Writing an IaC Rosetta Stone

briancaffey.github.io

34 pointszacharycohn38 comments

>I feel like Pulumi takes the best of the two tools and has a really great developer experience. There is a little bit of a learning curve with Pulumi, and you give up some of the simplicity of Terraform.

I'm familiar with Terraform, and generally like it compared to AWS Cloudformation, but do have some gripes. I'm not familiar with Pulumi - what have other people experienced with it? Do you like it better?

They introduced a Yaml interface last year, because, as I suspect, programming configuration in multiple languages sounds nice up front but does not scale. When you program IaC, you stop being declarative in the same way TF is.

That hasn’t been my experience; quite the reverse.

Programmatic IaC made it a lot easier to:

- turn our inventory DB into a IaC deployment for a hybrid cloud that needed IOT shadows per datacenter device

- shard our data processing pipelines to localized data centers so we can respect data protection laws

- automatically compiling infrastructure definitions directly from programs, to run as “cloud binaries”

What do you imagine the challenges at scale are?

The main issues with multilingual, imperative infra is sharing between teams who use different languages and building tooling for the stack. Another issue is understanding how the config / infra is assembled in high pressure situations like outages.

All of those things you mention have been done for a long time before pulumi or re-imperative infra. OG DevOps was largely imperative until TF came out. There was a reason for the switch. Not that I don't have complaints about TF, but I would never go back to the imperative days

I'm personally moving to CUE and outputting tf.json files until I have time to investigate crossplane. Want to see if their state management, permissions, and reconciliation is better by building on the k8s API S

> Another issue is understanding how the config / infra is assembled in high pressure situations like outages.

My experience is that this is strictly easier in a full programming language than TF, due to debugging tools.

> OG DevOps was largely imperative until TF came out. There was a reason for the switch.

There’s also a reason for the switch back: I’ve had to maintain those declarative templates at scale and they’re ghastly. I would much, much rather deal with an actual programming language any time.

I would never go back to the TF days.

> My experience is that this is strictly easier in a full programming language than TF, due to debugging tools.

Is that with code you have written? At scale, you are SRE'n something someone else wrote, because no one can be on call 24/7

I've never had to really debug TF in an outage, the tools have been logmon & requires visibility across the stack, from infra to applications

> There’s also a reason for the switch back

I suspect this is more about everything old being new again, TF having a few rough edges, and DevOps people wanting to feel like they are programming. Their are better places to spend programming effort in the DevOps / sre job

> At scale, you are SRE'n something someone else wrote, because no one can be on call 24/7

My teams had dedicated on-calls; we were platform teams for other teams. So — perhaps it makes a difference if you’re talking about that situation?

> DevOps people wanting to feel like they are programming. Their are better places to spend programming effort in the DevOps / sre job

Yawn.

You’re implying bad morals because someone has a different view than you — and such ad hominems are inappropriate.

You also clearly are ignoring the experience of the many, many people migrating this way — purely out of hand.

Maybe that’s why you don’t understand: you’re too stuck up.

you sink to exaggerated and derogatory statements when someone makes points you disagree with

what does that say about you?

Uh — did you get our posts confused?

> DevOps people wanting to feel like they are programming

[…]

> you sink to exaggerated and derogatory statements when someone makes points you disagree with

> what does that say about you?

I am one of these people. DevOps practitioners openly talk about feeling like yaml engineers and wanting to do more traditional programming. It is no secret nor derogatory, rather a reflection and expressed desire of many practitioners.

However there are better places where we can write code than in undoing the progress of declarative infrastructure.

You again show your assumptions and misplaced ridicule

> you stop being declarative

Pulumi uses a declarative model - actually the same model as Terraform.

Wrapping declarative in imperative undoes the declarative

No, it does not. The thing that makes the decision about what effects to carry out in Pulumi and Terraform is what makes both declarative, not the programming language used to configure that thing.

No... When you wrap config in code, you are instructing how to build a view of the world. Sure what you are outputting is declarative, but how you get there is not. You are not declaring what it looks like, you are writing instructions for how to produce declarative. The end user experience is no longer declarative, it's instructive, it's imperative.

Pulumi & Terraform... infra... is not the only place where declarative happens. The driver and medium does not determine if the system is declarative. It is the developer experience which does

This is quite simply false. Reducing your statement, using count in Terraform makes it imperative, which is patently false, since it is instructing the tool how to build a view of the world.

I could write a Prolog language host for Pulumi. Obviously that would not be imperative. It would present the exact same model as TypeScript or Go.

The engine is what matters, not the configuration format.

TF has indeed introduced some questionable semantics, though count does not make it imperative. On this you are wrong.

Yes the infra after the imperative step can have declarative layers, but the DX is still imperative. Declarative frameworks still have imperative operations underneath. You cannot argue that something is a declarative DX that is no longer a declarative input. Declarative DX is what matters. Pulumi reverses this so that the outer most layer is no longer declarative. Declarative as a term came to be because it is what the human is doing

> The engine is what matters, not the configuration format.

Code is not a configuration format...

> Code is not a configuration format...

Sorry, Lisp says otherwise.

It's clear I'm not going to modify your view of the world here, and you certainly are not going to modify mine, so I don't think there is any point in continuing this conversation.

Lisp is a special case, iirc, they think about it more as code is data. What if we only consider the languages most used with Pulumi?

Now let's consider another idea... can Pulumi prevent me from making arbitrary calls to an API or database to fill in the "declarative code"?

Honestly, reading the pulumi website ought to be enough to realize they think of it as programming and not something "declarative". They don't even use the word, because they aren't. They call it programming, programs, and code in many places, so they are positioning themselves differently from how you are describing them.

I have a vested interest in this argument, because it comes up all the time. From your argument, it's apparent you've never written a program in anything except a configuration language.

But by your own admission:

> You are not declaring what it looks like, you are writing instructions for how to produce declarative

If this was true, a Pulumi program would look like this:

    const bucket = new aws.s3.Bucket()

    if (bucket) {
      // do something
    } else {
      // do something else
    }

If you'd written anything in Pulumi you'd know that isn't how it works. The only imperative part of Pulumi is your ability to manipulate the DAG, for example:

    const config = pulumi.Config()
    const createBucket = config.reqiureBool("createBucket")

    if (createBucket) {
     new aws.s3.Bucket()
   }

Now, is there an imperative statement in there? Yes, but the only imperative part is your decision to include or exclude the bucket itself from the declarative end state. You do _exactly_ the same thing in Terraform but with a less intuitive syntax:

    resource "aws_s3_bucket" "example" {
      count = 0
    }

By that definition, if Pulumi is imperative, so is Terraform. It's not though, and you're wrong.

> From your argument, it's apparent you've never written a program in anything except a configuration language.

Way to assume, you know how that usually turns out?

Could I condition the creation of a bucket based on the even/odd value of a different resource's hash? What about the time of day or day of week? What other arbitrary things might I be able to do in a language that Pulumi supports? How can I be sure that if I run the code on a different day, that the outputs are stable?

Let's consider another idea... can Pulumi prevent me from making arbitrary calls to an API or database to fill in the "declarative code" that creates a resource? Would something like that be imperative or declarative? Does/can the order I write Pulumi code in matter? Does it for TF? Can you interact with the outside world with custom code in TF between the creation of resources?

In the end, are you writing the desired state directly or instructions for how to construct that desired state?

I invite you to consider these questions with the others you have discussed Pulumi semantics, as I assume I am not going to have success getting you to do so here...

You can likely do some of the above with TF, as it has strayed further from declarative. What you really want for configuration is a language which is not Turing complete and does not support dynamic values within the code, only via flags / args at invocation time. In other words, same outputs from the same inputs. I think TF & Pulumi both fail on the latter

(https://www.richard-towers.com/2021/01/03/terraforming-adven...) (https://cuelang.org)

What you appear to be saying is that Kubernetes is declarative if using YAML but not if using client-go, which is patently ridiculous.

The answer to your question is that you are constructing the desired state, not instructions for how to get from the current (unknowable) state to it. Using an API to construct declarative configuration is actually an incredibly useful pattern. Terraform can do it with HCL, too.

When you use client-go, you step away from the declarative interface to Kubernetes, much like when using Pulumi to step back from the TF interfaces. Kubernetes has itself strayed from being purely declarative and provides several mechanisms for transforming and modifying incoming resources during the admission and scheduling process, so you cannot actually be sure what you submit is what will be running anymore.

Again, declarative vs imperative is a DX issue. All the systems underneath are using both. So if we follow your logic, that the underlying system is what matters, every experience would be both imperative and declarative. Kubernetes and Terraform both use imperative processes to implement the declarative input. Using the properties of the underlying system to describe the exposed DX makes no sense. You don't get to add imperative on declarative or declarative on imperative and claim it has the property of the system you just layered on top of. The outermost layer, the one exposed to the end user, is the type of system (declarative or imperative) that the user is using. Pulumi and client-go are both imperative interfaces on top of other systems.

Go fish...

On a similar but slightly different topic, I've long wondered why someone hasn't created a "Universal Terraform Provider" that can work across the major cloud services. Resources such as "instance" and network are pretty general, an while there would definitely be gotchas, it would probably be helpful in the same way that cross platform GUI frameworks have seen a rise in popularity.

On a similar but slightly different topic, I used to think Terraform was the be-all end-all until I realized you don't manage Kubernetes with it.

To my knowledge, https://registry.terraform.io/providers/hashicorp/kubernetes... less people use this than use Helm charts, Kustomize, or some other Kubernetes-.yaml formatter

Then I get confused on where Vagrant provisioning fits into all of this.

Or Ansible/Chef/Puppet.

I just spin up a DigitalOcean droplet (I think I usually do it in the GUI to be honest, lol). Then I SSH into it, git clone a repo, docker compose up or (sometimes terraform apply) and I'm done.

Where does it grow from there?

I feel like based on the comments here, some people would even argue Docker is overkill there, or anything that isn't a monolith (small services glued together) is overkill.

I do manage Kubernetes with Terraform. I'm not a fan of Helm, however it is useful due to the community of available charts. Since I use Terraform to manage most everything else, I typically wrap the chart in a Terraform module too. This lets me do other things I usually need, such as creating IAM roles or other associated resources that the chart requires. Sometimes this means using the Kubernetes provider to add something needed by the chart, for example pulling a password from SSM or somewhere and creating a Kubernetes Secret from it.

The benefit is being able to build full stacks with only Terraform without needing to orchestrate multiple tools together. Building the cluster itself is done with Terraform, as is deploying the charts and other resources I need to build a "base system", all from a single root module. This also simplifies CI/CD as I only need a simple template to run IaC jobs and they all follow the same pattern of just running Terraform and not much else.

Could you show me an example somewhere on Github on how you do Kubernetes resources with Terraform?

To be clear, there is no YAML that you are writing?

I found this: https://github.com/hashicorp/learn-terraform-helm/blob/main/...

I made a small example at https://gist.github.com/rjosephwright/f1485cddcc12dd651ec965.... There is no YAML, I use HCL for values as they are easier to validate and less error prone, then convert them with yamlencode().

Thank you. What are the advantages to this over a light shell script that uses `kubectl + helm` CLI tools in your opinion?

You can just `terraform apply` and that's it?

My concern is it adds a "third" technology/abstraction layer (Terraform) to another underlying 2 (Helm + Kubernetes, which are also "abstraction layers" in themself, right?)

For me there is a benefit in having a single abstraction layer, the Terraform root module, to make IaC changes, and specifically one that is in the form of data rather than a script. The data incidentally contains everything required for reproducibility (all dependency versions are pinned, etc), and it always calls only versioned modules that are published independently. There is one Terraform wrapper script that runs from CI/CD and it is the only script, no matter what kind of change is being made, Kubernetes or otherwise. Teaching someone how to make a change is easy, they just need to make a change to the data and apply it. If the data layer doesn't support the necessary change, then work can be done to add the capability to one of the modules and then publish a new version. For simple needs this might be overkill, but in my environment it helps to keep things from getting out of hand. Helm and Kubernetes are abstraction layers, but very specific ones, whereas Terraform is a generic one that ties it all together.

> There is one Terraform wrapper script that runs from CI/CD and it is the only script, no matter what kind of change is being made, Kubernetes or otherwise.

Is it just basically:

    terraform init

    terraform apply -var "env=$ENV" -auto-approve

It also defines the state in a standard way, toggles CLI configuration if it is present in the root module, and does some validation.

could you share it/expand on “defines the state”? i thought state was captured in tfstate and then terraform goes and “diffs”/checks if it is accurate or not

I was referring to how and where we store it. Feel free to email me if you have more questions.

While it’s true that the cloud providers offer the same primitives, storage/network/compute, their implementations are different enough that it matters when you’re building any complex system. A Terraform provider that attempted to provide an abstraction layer over the major could services would end up having to either encode assumptions about how the user wants their infrastructure or it would have to be overly verbose in its configuration. In practice users want neither of those.

Terraform also aim to create a single workflow for all clouds, rather than provide a common abstraction over the clouds.

So, it’s a combination of that’s not what the tool was designed to do, and that’s not what people want in practice, given the current state of the major clouds.

If you want to take the time to deeply understand the answer to your question, grab one of those resources and set the Azure, Google, and AWS versions to each other. The VM instance equivalent is a reasonable starting point. The S3 equivalent isn't too bad either. Start lining up their similarities. Figure out what you would do to harmonize them. At least glance at the transitive dependencies, like networking... you don't need to fully work them out but you need to at least think about how that would be harmonized too.

If you do this honestly, and with some experience of having to tweak this semi-obscure setting and fiddle with that semi-obscure setting a few times, and you look at them honestly and realize that this is one of those "Microsoft Word" cases ("everyone uses 10% of Microsoft Word but no one uses the same 10%"), and all those settings are there for a reason and there isn't hardly one you can just drop without consequence, you'll understand.

At a 10,000 foot view it looks incredibly easy. When you actually try, you discover the lowest common denominator is so low that it would barely function. I'm pretty sure you can put an instance on "the internet" and probably load it with a common OS (though your harmonization layer is probably already doing some modestly heavy lifting just trying to have a coherent view of OS distributions) and get it to serve something, probably get some very simple port blocking done, but you're going to start being in serious trouble the moment you need more than that. The details will explode in your face, repeatedly.

But, I don't just mean this rhetorically. I'm serious. If you're really interested in this, this is a good exercise and you will learn something from the experience worth the time. Good for tuning the intuition.

There is already a couple of options

https://github.com/multycloud/multy

But a multicloud wrapper creates a lot of abstractions.

This would be both more honest and more comprehensible if it used the original title, instead of "Writing an IaC Rosetta Stone"

Sorry - the original title was a bit long!

Reply on news.ycombinator.com