Hacker News Reader

Comment by toomuchtodo | Hacker News Reader

I just wanted to offer the other side. I do devops, we use Terraform, and we've had times when Terraform took action that was not present in the plan generated before applying, in production environments. We've also had times when it'll fail mid-apply, be stuck in an inconsistent state (IAM profiles and autoscaling group configurations, specifically), and require manual recovery through modifying or removing resources via the AWS console (although this might be better in the latest version).

We'll continue to use Terraform at my day job, but I don't recommend it to colleagues nor clients if they use only AWS resources. Simplicity is the ultimate sophistication, and adding yet another abstraction does not contribute towards that philosophy.

View on HN

We consider these kinds of issues the highest priority and always work to resolve this ASAP. The resolutions very often include not only the fix but protections to ensure that unintended actions don't bypass anything in the future.

It's really unfortunate you had these experiences and I can't back up the statement when I say that this isn't normal and every release certainly does improve stability in a big way. Terraform is a very, very heavily tested (unit + acceptance) project and we fight very hard against regressions with multiple tests per bug in many cases at different levels.

I'd encourage you to continue giving future versions a shot and if you experience anything like this again, let us know and we'll react to that quickly relative to other types of issues.

Thanks for the reply Mitchell! Definitely don't want my criticism to come off as unconstructive. As I mentioned, we'll continue to use Terraform; I'm optimistic it'll improve as Hashicorp iterates. I'm just naturally conservative coming from an ops background.

I try to pick boring technologies that are battle tested through years of tech sector use; there's no glory for being on the bleeding edge, with only pain when things go south. So goes the ops struggle.

This matches my experiences, and my own recommendations. Multiple clients of mine have independently coined "terrafucked" for when Terraform frags its state. It's usually recoverable but it shouldn't under any circumstance happen, and regardless of my other beefs with Terraform (which, to be fair, aside from the quiet insanity of HCL, seem to be decreasing!) because of that risk profile alone I can't recommend it for AWS-only clients right now.

Personally, I use CloudFormation for my own stuff, because aside from reliability concerns, giving up on cfn-init to not write JSON (when I don't now, I use cfer, a Ruby DSL) is totally no bueno for me. Terraform offers no direct equivalent, though I've had HashiCorp sales people try to push Consul chewing-gum solutions that I'd have to manage much more directly than CFN metadata/cfn-init have been for me. Maybe in a couple years it'll be somewhere where I feel safer recommending it.

Yes to all this.

MIGBY, we have also been bitten in production in much the same way. If you are using only AWS resources there is no need to use terraform. Using it at all seems like a questionable decision to me since it's still firmly in "early adopters only" status.

If you're AWS-only then CloudFormation is the way to go. And IMO it will continue to be for a long time, until terraform is much more battle-tested and -hardened. Unless of course you're operating in an environment where you can afford to be exposed to the cutting edge of infrastructure-as-code. I definitely am not.

Reply on news.ycombinator.com