On a related note, CDK for Terraform allows DevOps practitioners to use a variety of programming languages instead of HCL. I've really enjoyed modeling my AWS environments with Python using Terraform only as the engine.
More info here:
More info here:
_Full disclosure_ - Marcin at Spacelift here, and we chose to support Pulumi in addition to Terraform, so this opinion is necessarily biased.
I get Logic in yaml/json is less ergonomic but templing makes us for it
Tried it again last month and it's better, but I had to do a lot of trial and error. ie. (Try to figure out how to provision an EC2 instance with a root volume. They copied part of the docs from terraform but sections are missing.)
What showstoppers did you run into? Any insight welcome if that prevents me from hitting a wall I've not seen yet :)
Pulumi has a similar model where you build a resource graph at runtime BUT it's also got the execution engine built-in to the tool.
What this means in practice is that you can create resources (like a kube cluster) and then use them as providers (e.g provision state tracked resources with kube api) all in the same operation.
You can also (in your infracode or an importable module) define "dynamic providers", meaning you can easily extend the execution engine to do custom things related to your workload.
As an example, imagine you want to create a cluster, deploy an app, then provision (state tracked) some app resources like an admin user and group via the app's REST API. You can do that without too much fuss.
Neither terraform nor CDK can really do those things very well. TF is not powerful enough language-wise, and in CDK the execution phase is locked away from you.
Could you elaborate a bit more?
Let's say I have an app with API Gateway, Lambda, and DynamoDB.
I would provision them with one of those tools CDK or Pulumi.
How would these tools differ in their provisioning steps?
Under the hood different things are happening.
Your CDK program would run with all the resources you declared. That would generate CloudFormation script(s) that get submitted to the CloudFormation service for evaluation. The CloudFormation service (running inside AWS) is the "execution engine" and is responsible for creating and tracking the state of your resources.
Pulumi would run your code, build an object graph, then do the "execution" itself - invoke all the required AWS API calls to create your resources (the API Gateway, the Lambda, etc etc) from where the CLI is running. The CLI would also be writing out all the resource state to somewhere.
The tradeoffs are in line with what you might expect. The Pulumi approach is more powerful, but you "own" more of the complexity since it lives on your side of the responsibility boundary.
Some people prefer AWS to be the execution engine; they feel it's more reliable to let AWS provision resources and keep track of state. They like it that AWS is responsible and will fix bugs.
Others prefer the increased control of "owning" the execution engine. This means being able to debug it or extend it with third party or custom providers that let you provision new resource types. They're happy that they don't need to wait for AWS to fix things, they can do it themselves if they have to.
This is not the only difference between the two tools but it is one of the most fundamental ones.
Does Pulumi support stuff like Cloudflare Workers, FaunaDB, or Auth0?
What is CFN here? Cloudformation?
xyzzy123 already described the differences between Pulumi and terraform, but I want to add one key way in which they are similar:
Pulumi uses terraform under the hood. We get all of the reliability of terraform, but with a much more powerful runtime engine.
What I want: Use Terraform programmatically, i.e. call "cdktf deploy" or similar FROM node or python and give users some scripts they can use where I can abstract away some of the difficulties of learning to use Terraform natively for simple use cases (i.e. deploy an S3-based frontend host). Ideally, I had intended to distribute some npm-installable packages which would run this stuff.
This is actually exactly the use case we’ve designed the Pulumi Automation API (https://www.pulumi.com/blog/automation-api/) to support.
Allowing modern IaC technology (like Pulumi or Terraform) to be easily embedded into custom software solutions, instead of just being something humans work with directly, is a huge potential enabler for the next wave of cloud infrastructure management tooling.
Maybe not node/python, but I'm pretty sure you can use terraform as a package in go. If not, there is always the "make temp dir, write/download files necessary tf files, run terraform apply"
We have are actually working on : "How to manage Terraform resources". We ended up having a conflict dev <-> ops where dev teams are messaging the terraform guys to create resources. For example, to create a database, it will take only 1 hour writing HCL, but 2 weeks of emails to align on the specs.
We are currently building something on top, to have resources that can be created in a self-service mode by the devs themselves. (Behind the scenes, it uses Terraform modules to generate resources that will comply with the company policies).
For the moment, it's a bunch of Jenkins pipelines. Having a CDK can actually help us a lot there. (Can plug it to a CMDB database, have a UI on top, etc)
No... the high level programming language really just serves as a bridge or translation layer to a Terraform compatible JSON file. Those sorts of evaluations don’t happen to the actually plan/apply. However, you may find it useful to make direct API calls to your cloud provider in cdktf stacks. For instance, I mostly use data lookups but if I want to perform string operations on that sort of data I would use boto3 instead.
Out of curiosity what is it that people generally don't like about HCL?
Terraform is super fast, has great docs and vast coverage for managing just about any type of cloud resource.