Hacker News Reader

Comment by kaidon | Hacker News Reader

Genuinely interested -- why would I choose this over running Terraform as part of CI/CD?

If terraform crashes during apply, it leaves behind an inconsistent state by design: The lock is still set, and some resources which were created already are not yet in the statefile. Trying to re-run terraform after a crash during apply will generally lead to an error: Even if the lock is removed, resources may still conflict if they already exist.

In contrast, when a kubernetes controller or operator crashes, it can be expected to continue seamlessly where it left off.

It is easier to write kubernetes controllers that are able to continue seamlessly then to write terraform providers that do so, because of the granularity of the persistence of the state machine. Terraform locks the remote state, then applies all resources in the current root module, then unlocks the remote state again. In contrast, kubernetes operators can granularly update individual objects after each API call that is performed.

I have never seen Terraform crash in the years I’ve been using it. Optimizing based on a lottery event seems counter-productive

EDIT: Alright guys, good points all around. Mainly what I meant is the ratio of useful of Terraform vs its shortcomings. Not hating on the Cluster API pattern, but imho better to stick to standardized approach.

That's a very shortsighted comment. Just because you've never seen it happen, it doesn't mean that it doesn't. While terraform itself may not crash (maybe not in the manner that I assume you're referring), you are at the mercy of the implementation of the specific providers for which I've (and many others as well, I'm sure) seen plenty of issues. Even in "mature" ones as aws/gcp.

Besides that, the control-loop pattern, which the parent comment describes is a very sound design, which is employed not only in kubernetes and has nothing to do with "Optimizing based on a lottery event".

> That's a very shortsighted comment.

Is it though? GP claims to have been using terraform for years.

I have been using terraform for some time too, and from what i've seen well written providers will error out in a safe and controlled manner instead of making the whole thing crash.

> Even in "mature" ones as aws/gcp.

Can't comment for the gcp provider, but I've worked at companies where the terraform provider is used quite extensively and really can't recall a real crash that created serious problems. And i've been also using the openstack provider, again, quite extensively, with not much troubles.

So in conclusion I think that it's your argument that's really short-sighted, because it's based on FUD and situations that are theoretically possible but sufficiently remote in the real world that they can be ignored.

Doing a very quick search [1] on the aws provider github repo, produces `92` open, and `306` closed issues with various bugs in the provider. I agree that most of them might not count as creating serious problems, but I as well have been using it quite extensively and can definitely remember cases where I have had to manually deal with corrupted state in various ways.

I'm not sure which part of my comment was "FUD" - I never said that you shouldn't use terraform, I was pointing out that issues exist, always. I think of it as the nature of the work we're doing (I say we, as I imagine you are also in the SWE field based on your comments). Show me any piece of software without bugs, especially as complex as a tf provider, and I'm buying you whatever you want.

[1] https://github.com/hashicorp/terraform-provider-aws/issues?q...

Anecdotally I've had to untangle lots of half applied state. Sometimes a provider has a bug, sometimes somebody on the other side of your company changed your terraform plan without migrating the running state, sometimes there are network issues that prevent your plan from applying correctly, sometimes the cloud provider said it deployed the resources but it actually didn't, etc, etc.

It's extremely painful and delicate to do and it may not happen frequently, but in my experience it happens frequently enough when I'm dealing with a lot of terraform.

I've had Terraform break several times when my AWS STS token expired half way through a deployment. When that happens I ended up having to manually delete all the created resources because the Terraform state was corrupted to the point I couldn't reapply to create the remains resources but also couldn't delete the existing resource with Terraform.

That said, I agree in general that this wouldn't be the main reason to use the Cluster API over Terraform.

"Optimizing based on a lottery event seems counter-productive"

I think it depends on how often you "win" the lottery. If a lottery event happens once or twice a year and it's fairly hard to resolve, we classify them as land mines (or sea mines for the more maritime oriented people among us). These issues get a lower priority, but we'll try to work through them to avoid emergencies or lower their impact. So, maybe not optimizing, but increase resilience against lottery events.

That is an (incredibly) poor explanation of Terraform plugin separation from state management (or perhaps you’re using a weird locking backend that has more issues than correctly implemented ones). If created resources don’t end up in the state file, that is a provider bug (they failed to call `SetId` soon enough).

Source: was a core developer of Terraform and some of the largest providers for many years.

If the explanation is incredibly poor, why is this issue still open: https://github.com/hashicorp/terraform/issues/20718

If you don't need to dive into the rabbit hole that is Hashicorp and Terraform and can manage with just clustetctl and kubectl. Or if you prefer yaml with no templating :)

Terraform isn't a wrong decision, just that CNCF now has a more stable API/CLIs you can work with, with which Terraform also needs to integrate.

I guess it depends where you're starting from: I find the entire Hashicorp ecosystem much better documented, supported, more stable and generally less of a "rabbit hole" than the continuously shifting and expanding universe of k8s/"CNCF" projects.

Unsurprising because all the Hashicorp stuff is from a single vendor, and there is the risk of lock-in.

We use it to almost fully replace terraform (still use it for some basic stuff) for couple reasons:

* simplified tooling - you can use all the same tools you already use to manage your k8s configs

* more standardized setup for multi-cloud/onprem

* it’s based on vanilla kubeadm and cloud-init/systemd images - debugging problems is easier bc it’s all pretty standard, battle-tested tech

Many Kubernetes operators are sufficiently mature to use them to manage external infrastructure.

When it works, it's nice to be able to deal with one templating language and one system to manage all of the elements of your deployments. Deploying a single helm chart is a nicer developer experience than managing both a helm chart and a terraform plan.

my understanding is that the machinery underlying it are a bit more complex than what terraform would do.

if anything that would be because the kubernetes operators implementing the Cluster API for your specific cloud provider would be implemented in a continuously running control loop, thus picking up changes... whereas with terraform you would have to run terraform from time to time.

edit: plus of course all the other features coming from kuernetes like mutating admission, validation, rbac (eg: only allow some people to create clusters or allow people to create clusters without granting access to the underlying cloud provider) etc etc...

or over running Joyent's Triton?

Reply on news.ycombinator.com