Hacker News Reader

The state management is a primary focus for us right now, and I think that's evident from this release: state management CLI tools as well as the beginning of import (which is primarily dealing with state in this iteration).

> ... it still breaks the second anyone ... bypasses Terraform and touches the world directly, so Terraform simply isn't resilient by design.

This shouldn't be true. If you create a new resource that was never under management by Terraform, then yes, Terraform will ignore it. This is by design so that you can use Terraform to manage infrastructure and use other processes to manage other infrastructure that perhaps isn't quite migrated yet OR doesn't fit Terraform's model for whatever reason.

However, if you change a resource under management or even remove it, Terraform will notice this. It is very resilient to detecting drift.

> I never understood the purpose of the state file.

The purpose is to map your resources to what exists in the world. There must be some way to make a mapping there to be able to do diffing, drift detection, etc.

Some cloud platforms provide interesting ways to work around this (AWS tags are often abused for this) and we actually experimented with that approach early on. Unfortunately, not everything supports tags and there was no way for us to separate what was Terraform managed and what wasn't.

One of the features we actually have planned for Terraform in the future is the ability for it to tell you what IS NOT under Terraform management (by comparing local state to global state). For companies that have everything under management, this will be a great check to find any rogue resources. For companies working on adopting Terraform, this will help find things that still need to be migrated/imported.

So, hopefully our state improvements in 0.7 and in the short term will help sway you. They're definitely a top concern. But I also hope I explained my way through some of the design here!

Thanks for the feedback and we hope to see you in the community soon!

I think that opening changes in state via a CLI is a fast path to making things worse.[0]

I greatly prefer the BOSH model: "give me a manifest and I will make the world look like it".

Documents are easily versionable. Commands aren't.

Documents narrow scope and can be more easily made idempotent as an automatic guarantee. Commands require us to remember to check the state ourselves, manually, before doing something.

That said, Terraform has a significantly different low-level model. BOSH looks at the size of the problem and imposes non-negotiable primitives to scope that complexity. Terraform is more ambitious. I suspect Terraform will be easier to adopt incrementally, but harder to manage on a per-unit-of-state basis.

> "give me a manifest and I will make the world look like it"

This is how Terraform works if you want a one-time setup. You can just throw the state away after that if you want. But Terraform is supposed to be used (and is very very often used) to do ongoing minimal changes to infrastructure, often dozens of times per day.

In this scenario, Terraform's configurations are completely declarative: it is a state of the world you want to reach, it isn't what to do next.

However, we require the state file in order to find the resources we own and then refresh the state of the world so we can do a diff.

This is a bug we just hit 5 minutes ago on 0.6: One of our CD builds failed because of a configuration issue - a variable wasn't properly set, leading terraform to continuously ask jenkins' stdin for a password.

We fixed the bug. But somehow, that previous build wiped the state file. When we pushed the fix, terraform freaked out and started deleting all our instances, including S3 redirects that weren't managed by it which we explicitly told it to ignore.

That sucked.

Always, always view (and save to disk) tf's plan. And if it looks good, apply the saved plan output.

Did view it... locally. It kind of defeats the point if it's not reliable enough to trust it'll behave the same from a different machine.

No harm done, we're still in beta to catch bugs exactly like those, but our initial enthusiasm with terraform has been waning very quickly. Like I said, I love the design, but it's just not there yet.

Do you keep a state file per environment or do you have one big state file for all your environments?

I would consider this a bug - if you can open an issue on the https://github.com/hashicorp/terraform repository, we will look into it prior to 0.7.1.

> But Terraform is supposed to be used (and is very very often used) to do ongoing minimal changes to infrastructure, often dozens of times per day.

I've seen BOSH used this way.

You just have to accept that editing a file and checking it into version control is a good thing.

After a while, infrastructure tends to stabilise into a predictable format for the given distributed system under management.

I think Terraform would definitely be easier for a fast-developing system because of its bias towards interactivity. But in the long run most systems cease to be fast-developing.

At Pivotal we're seeing more and more customers going all-in on BOSH. Not just for Cloud Foundry, but for pretty much every stateful service they can lay a hand on. They like that it's based entirely on versionable, auditable files.

> However, we require the state file in order to find the resources we own and then refresh the state of the world so we can do a diff.

BOSH will inspect the world when you ask it to, or you can ask it to do so continuously. As a rule, most operators prefer (like Terraform users) to do it manually.

I know why the state file exist, sorry; I really meant: The purpose of making the state file mandatory. In particular, it makes first-time setup incredibly onerous. You almost _never_ start with a clean slate.

It's great that you're working on fixing that, but it looks like it will be some time before it's quite there, given the seemingly slow pace of development.

It's nice that Terraform can support untaggable resources, but I don't understand why that's the general case and not a special case. I can't remember an AWS resource that isn't nameable or taggable.

>> I can't remember an AWS resource that isn't nameable or taggable.

ElasticIP is one among many. For a complete list: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tag...

> After all, it means that you have to start from scratch. If you any existing infrastructure — instances, load balancers, DNS, whatever — then anything you declare in Terraform will conflict, even if it's declared exactly identically.

One of the major features of the version released today is the ability to import existing resources in to Terraform state.

I wrote about that.

   That being said, 0.7 has an import command now, so they recognize the problem.
   It's extremely limited and is not a complete solution yet, however.

> A tool like Terraform, from my perspective, should be stateless. It ought to compare its target to the world, and then attempt to converge the world to match the target.

This is how BOSH has been doing it for years for IaaS setup and other systems like Salt/Puppet/Chef/cfengine for servers. Compare-and-repair is a fairly well-accepted, trustworthy model.

Disclaimer: I work for Pivotal, we donate most of the engineering on BOSH.

[0] https://bosh.io/

CloudFormation is stateless as well.