Hacker News Reader

Comment by dharmab | Hacker News Reader

An alternative opinion: When I worked at a large tech company, my team made a conscious decision to not use terraform. This gave us some key advantages- we are able to adopt new cloud features immediately, months before they were available in tf, and our direct cloud access let us build features that would surprise the teams using tf within the company.

If your core competency isn't dependent on your cloud platform tf is a great tool. But using cloud APIs directly was great for us.

View on HN

> But using cloud APIs directly was great for us

This is fine, I've done it extensively myself for some of the bleeding-edge cloud stuff, but the importance of things like tracking state, managing hierarchical resource dependencies, or retry/back-off logic shouldn't be tossed aside simply because there are gaps in what's available in the Terraform providers. Especially where change management is important (basically any enterprise company).

I'd caution others reading this against abandoning something altogether and writing bespoke IaC tooling simply because the stable approach doesn't cover every (bleeding) edge case.

You'll spend a lot of time reinventing the wheel, and while it's fine for certain situations (like when you only care about desired state, not known state, for instance), you'll move faster (and likely safer) by sticking with tools like Terraform for the bulk of your infra, and augmenting here there with cloud APIs/SDKs when needed.

Yes, we did have to implement our own state tracking, retries/recovery, etc- but since we were focused on a limited subset of the cloud API, this was pretty easy.

So you re-implemented terraform but worse most likely. Also you could have added those missing features and re-use the TF engine, it's very simple to include new API of an existing provider.

No, we used an entirely different architecture/paradigm not possible with tf, and had capabilities tf doesn't attempt to provide (such as coordinating migrations with both cloud and application APIs, or managing capacity while upgrading 10000s of CPU cores worth of compute).

Would the time spent re-implementing a specialised Terraform subset be better spent simply maintaining a private branch of the AWS Provider? You can add your secret/special API without having to do all the other heavy lifting as well.

This makes your own effort for customisation minimal, keeps your knowledge portable and because your added features can be separated in to different files and the provider API is stable you can also easily backport/fast-forward new changes.

Our custom code was fewer LOC, more robust and faster than tf. Honestly, it's not hard to beat OSS if you have a team of great engineers. This was far from the only time our internal projects were better than OSS- it gave us advantages for removing bottlenecks in the critical paths.

So when engineers left the company, was their knowledge portable? And attracting new talent, did they come with X years knowledge for the custom code?

With fewer LOC I'm not sure what you mean, the provider code is pretty small, smaller than custom ansible, salt or puppet modules. Smaller than CDK and Pulumni as well. Sure, you'll have to write Go, but that's about the only hurdle.

Everyone doing a round of NIH for internal tooling is ultimately not making the tide rise.

Edit: don't get me wrong here, writing internal tools to do a job the right way for the right needs isn't "invalid" or something like that, but people often dismiss the rest of the lifecycle of knowledge and maintenance when making something completely custom.

It usually takes 1-2 years for AWS to roll out their latest updates to the regions I use and by then Terraform is stable.

Same here. It's incredible how much efforts developers are willing to put on popular "devops" tools when the job could be done faster with 200 lines of Python.

This comment Gets It.

What new cloud features was not available in Terraform for months?

eg. AWS ChatBot not available in TF yet. TBH AWS haven't even added it to their Go SDK. So, I cannot blame TF. But anyway that's one of the inheret problems of TF plugin system.

Compare to kubectl. Where you can write plugins in bash/shell and mark with execute bit, put it in somewhere in your $PATH as kubectl-blabla and use it as "kubectl blabla".

It's not fair to compare imperative simple shell scripts with the things Terraform does. It has schema validation, state comparison, retries, failure handlers etc.

Also, just as you can write extensions to kubectl, you can write your own provider in Terraform if it does not exists. See https://registry.terraform.io/modules/waveaccounting/chatbot...

Also, Chatbot does not have a public API, that's why, it's only configured via Cloudformation. So the expectation is not fair either.

I've seen Cloudformation getting features years later. i.e

2021 - https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dy... 2015 - https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dy...

NAT Gateways is another notable feature that took CloudFormation months yet Terraform had on day 1.

If you can configure something via CloudFormation you can integrate it via Terraform et al also, since they have resources representing CloudFormation stacks.

This! Is not like you can't go beyond what Terraform offers by default. Running CloudFormation stacks from Terraform is a neat way of solving missing apis/integration. And that's exactly what my team did when Terraform was missing a lot of lambdas functionalities. We just declared the CloudFormation Stack for lambdas and then call it from Terraform.

There’s no reason you can’t do something similar with Terraform either - plugins speak GRPC and thus could be implemented in Python, with Node.js or with Rust.

However, if AWS have not published metadata for a given service to be used across their various SDKs, it’s hard to take that service particularly seriously, so I’m not sure I’d bother with this.

Private preview features for partnered organizations. We had access up to 6 months before the public.

Reply on news.ycombinator.com