Hacker News Reader

I've been in two production environments where IaC was done with Turing-complete languages (Google and Facebook). It's really not better (it just lets the truly impossible/unmanageable become possible) because they were both dynamically typed.

If anything, Terraform comes the closest to being a good IaC language because providers form an API to production resources as opposed to unstructured code/config like Helm or Jenkins/Puppet/Chef. Production in reality is well-typed, and unstructured configuration makes static analysis very difficult and makes diffs only a guess at what will actually be changing when code changes are applied.

What I'd like to see is Haskell (or similar) typeclasses as production API; make it impossible to construct malformed configurations in the first place but allow expressive functions to replace the limited templating functionality that current languages have (why can't helm templatize inside values.yaml? Why can't Terraform templatize backends and why aren't providers first-class objects?). It's very difficult to keep current IaC DRY, and if it's not DRY then it has bugs and there are misunderstandings about how environments are actually configured.

In terms of “Haskell type classes as production API”, I assume you’ve come across Dhall?

https://dhall-lang.org/

One can write good and bad code in any language. It just so happens that our daily driver - procedural or functional languages, happen to be ones where a Software Engineer's mental model + Existing tooling is evolved enough to make understanding it _easier_ (as opposed to just easy).

The problem with DSLs and Config languages is that someone will inevitably try to use them in a way they are not designed to be used, and then circumvent the way you're "supposed to" do things. Not only do you lose all the advances in tooling, but people will end up reinventing things that already exist in the other languages.

PS: Of course, I'll pick Static Typing every time.

Cuelang is a nice compromise: https://cuelang.org

Composition over encapsulation, expressive enough for complex configs, not expressive enough to be turing complete.

+1. Cue is actually very nice. I wish there's more adoption of Cue instead of most other config languages. Config languages is one of those fields where the problem looks easy and the barrier for entry is low enough that we have many different designs where they don't provide meaningful enough differences, yet the actual problem is quite difficult and subtle. Cue is one of few that has a clean theoretical foundation and looks very promising. Too bad we have YAMLs and friends everywhere.

> The YAML (or heaven forbid, jsonnet) does not help. TBH, I'm convinced if the IaC bits were done in a regular Java/Go/Rust SDK style instead of mixing and matching YAML glued together via labels and tags, I would have had a much easier time.

I have mixed feelings about this. I've seen some absolutely atrocious Python and JavaScript code, and i wouldn't want to use either of them, and the flexibility ( to shoot yourself in the foot) they provide. Go is better. On the other hand, YAML is just terrible.

I think DSLs such as HCL are better than either option, as long as they're flexible enough while staying limit in order not to get illegible/unmaintainable.

You are on the right track. I have exactly the same story, data engineer, working on automation a lot.

YAML, JSON and HCL should be purged from IaC asap. Luckily Dhall can be integrated everywhere we need YAML otherwise we would go nuts. The bigges failure of k8s is to understand that YAML does not scale to teams and beyond 100 lines, especially not good to nest it (hello ArgoCD).

A type safe not accidentally Turing complete configuration language or for that matter any langue (hello Pulumi) is much better than YAML ever will be.

Anyways, if I can give you an advise, drop k8s like its hot. I is much more trouble than worth. We have phased it out everywhere we could. The companies still using it are in 1 year long implementation cycles with it with 10+ consultants. It is going very well. For consultants, obviously.

https://dhall-lang.org/

Dhall looks interesting. Thanks!

I went into the rabbithole of terraform/HCL and came up with Terragrunt to do dynamic code generation. Combining it together with generous use of `make` keeps me sane, but it's not a long term solution.

Even now, there are things within the system where the boundaries should be separate, but they aren't.

We literally got exhausted and burned out writing our automation, and decided to leave it as it is and then slowly build it up over the next few months. :/

PS: You're 100% right re K8s/YAML. It feels like everything in the Kubernetes world is optimized for writing the code just once. Which, of course, is easy if you already know how all the various bits fit together (true in any system).

But Reading/Debugging kubernetes world is a dreadful experience.

Edit: typo

> But Reading/Debugging kubernetes world is a dreadful experience.

Exactly. It is abstraction, over abstraction, over abstraction for no obvious benefits.

"turning complete" : typo?

Yes, thanks.

Dhall or other typed languages like Cue definitely help a lot. It seems to me that the only acceptable amount of YAML you could have in deployment config is for a setup small enough that it doesn't warrant using Kubernetes, so any use case where Kubernetes is a good fit quickly becomes a major pain to maintain.

That said, you wrote:

> drop k8s like its hot [...] We have phased it out everywhere we could.

Could you elaborate on what you've replaced Kubernetes with, and for what kind of deployment?

AWS Lambda, EC2, Nomad, depending on the usecase.

I think the YAML issues center around a simple fact, few are taking e.g. 1000+ line YAML (or YAML for X) dev seriously. Complex YAML becomes code, and requires as much e.g. refactoring, idioms, reuse, static checking. At least it's not terribly hard to make a script output it as a format

could you give me a hint why you didn't like jsonnet? I dreaded k8s yaml when needing to (write) and use helm packages and think of indenting levels, but jsonnet via grafanas tanka gave me the option to be concise, writing functions and creating reusable bits. Of course then you're not far away from using a real language "sdk style".

As you so aptly put, k8s learning curve is beginner hostile and needs system knowledge before you go at it, so I'm not sure if this was also with being more familiar with the abstractions - after having looked at k8s for 2-3 years, jsonnet made me finally like interfacing with the k8s apiserver.

Yeah, I think the difference is that I was

a) Learning K8s and Helm Concepts (both different things)

b) Reading/Debugging K8s and Helm YAML (both different things)

c) Ending up at Helm Chart Configs (another config)

d) Figuring out how to set all this to helm me deploy the solution I needed and customize it acc to my needs

...all within the span of a few weeks.

So by the time I got to some GitHub repos which were using Jsonnet, I was basically done. I'm sure it has its uses and given enough time and experience, I can see how to utilize it more effectively.

yes, both different things. I guess helm is still the most common entry point for people to handle the contents of a cluster. I'm checking out other suggestions in this thread (dhall), but from my personal experience I can already tell, grafanas k8s jsonnet-libs was a second chance to kubernetes I'm glad I got to know (thanks coworkers).

> it makes the difficult things possible, and that's what we needed

Indeed, the problem of just saying "you don't need k8s" is you may accidentally invent an even worse in-house version of k8s.

Though for 80-90% og cases out there you seriously don’t need k8s, and when the small internal IT dept that used to “manage” a dozen virtual machines starts talking about “need to keep up with the times and use new tech” what they are really saying is “let’s gamble infrastructure stability on my personal self education in k8s for my next job” and it’s perfectly reasonable to give them a hard no.

> regular Java/Go/Rust SDK style

I thought that until I started working with Gradle (which is a build script tool who's syntax is nearly-fully-functioning Groovy). I miss Makefiles - they were strictly declarative and, while I've seen some hairy Makefiles, you'll never need a debugger to figure out what they're actually doing wrong.

If you haven’t heard about cdktf, check it out! It lets you do infrastructure as code but in full programming languages like Python, C# and TypeScript. It’s by hashicorp, and under the covers it runs Terraform, so you have the whole set of terraform providers at your disposal.

Ah, CDKTF seems to be the Pulumi equivalent for Terraform. I think I remember seeing it.

We're far too invested into Terraform at this point to learn yet another tool. Oh well. Hopefully the next time I have to implement something like this, K8s ecosystem would also have similar tools (or maybe I just missed something)

There’s a straight up conversion tool iirc, and you can do things like include hcl as well I believe. Cdktf is just terraform at the bottom.

If you havn't heard, check out Pulumi..

I don’t understand the appeal of CDK, it’s the most difficult to understand overly complex part of IaaC I have ever come across.

I want simple yaml/hcl that is easy to understand and write. I don’t need to feel smart writing TF, I need it to be easy to understand to reduce MTTR when things go wrong.

the issue with yaml is that is breaks down under real life complexities in larger/complexer environments.

Try to TF manage a large github organization with just terraform and get back to me. I'm having to wrap it with python code already, and my next step is the CDK.

The biggest issue is that parts of the TF for github requires references, and it fetches each of those references individually (which which quickly make you hit your ratelimit, while also taking 10+ minutes for just a plan). Wrapping this in python, and doing more efficient fetches ahead of time, and feeding the references in yourself can turn a 10+ minute plan into a 20 second plan for 1000+ repos.

DevOps specialist here and I agree with your complaints about IaC meaning JSON and YAML documents. sigh

any suggestions on learning materials?

You'll find plethora of learning documentation, but by far, the one talk/presentation that made it click for me was this one: https://kube-decon.carson-anderson.com/