After that perhaps you vowed to take better notes, so the next time you do it that way, but then 6 months later you find that you missed some detail, or there was some changes in between that were not recorded.
So then you decide you're going to do everything by API and save the commands, so you write a bunch of bash scripts that execute against the AWS CLI. Over time you add several more for different operations (eg. adding an instance, configuring new ssl cert, etc), but the list of scripts grows very long, and you find that each one makes a lot of implicit assumptions about the state of the infra when its run, so you end up with varying degrees of confidence in the scripts depending on how often you run them.
Now you are primed for Terraform. At this point you realize the hard part about cloud configuration is state management. Furthermore, you realize there are some common patterns of how different components and APIs interact with regard to serial dependencies and operation idempotency, however the specifics vary by service and by use case. Terraform gives you a standard substrate for state management, and a framework for developing service "providers" that know how to interact with APIs and map them to state. All of this happens in code which is declarative, can be code reviewed, and state which can be centrally tracked and shared among a large team for a clear audit trail.
There's definitely a learning curve, but once you learn it the overhead is pretty small compared to the benefits, even for small teams IMHO.
You need to write idempotent ansible from the start. No excuses.
Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait)
Then after a while with all your resources created with your idempotency implemented in ansible, your code will roughly first check ( with timeouts, retries) if you actually can skip tasks since there is nothing to do. The checks increase, your playbook takes longer and longer to execute. Maybe you try to parallelize it in some way. In the meantime a colleague of yours executes the same playbook in their machine or pipelines, and now you see weird side effects. Maybe you start implementing some locking behavior.
At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?”
This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource.
So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code.
So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform.
If you are alone and don’t want to follow the above best practices, you still have to be very good in writing idempotent, resilient ansible code.
My hypothesis is that the intersection of people who write high quality (idempotent, resilient) ansible code but people who do not care about code review/testing, is quite small.
Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time.
You could even call ansible from terraform if you need some ansible integration, where ansible has a nicer api for you. But let terraform handle the retries, state management and ansible could just be the executing part. So ansible is more like a fancy bash script/function which you call in an orderly manner.
Absolutely. And code reviews, etc. just like code.
> Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait)
Yes, Ansible modules have a retry built in or a wait_for statement. You can also control the serialization serialize: (or was it parallel), or run_once or when. You have a couple of tools at your disposal.
> At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?”
Why not? We usually start with some diagram in README.md or Confluence or a Design Proposal before the code. Code never gets released without a merge request.
> This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource.
state: absent, you can use in about every module.
> So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code.
I believe terraform hides way too much and is too magic. Troubleshooting when terraform goes wrong is really really hard. You have two states to look on -- the cloud state and the TF state.
And now you need to know two tools too, since you will likely need Ansible anyways to configure and operator the hosts later.
> So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform.
Beg to differ, 100s of engineers sharing ansible galaxy roles with semantic versioning and whatnot in a very good way at my org.
> Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time.
Maybe you're right. If I'm doing freelance I will likely use Terraform to interact with Cloud. IMO Ansible, just like coding, requires a bit of a community around it with best practices, reviews and etc.
I understand we both have our experiences when using ansible compared to terraform. Good to know that this scales
Regarding your point:
> state: absent, you can use in about every module.
What I meant is that you probably would create 2 pull request. The first one to actually delete things for your infrastructure and then a second pull request to delete your ansible code. I am still not sure how changing things work in this manner, where you destroy and recreate things in a single logical Pull request/commit for example. I guess tracking multiple pull requests in a single ticket might work.
> And now you need to know two tools too, since you will likely need Ansible anyways to configure and operator the hosts later.
I agree. Well, I would use ansible beforehand with Packer to build immutable images/AMIs which are launched by an auto scaling group and a final configuration via cloudinit. Therefore I would not have a chance to apply ansible during autoscaling.
Thanks for your input and valuable discussion. Stay safe
I'd love to have a single tool, but Ansible's style seems like a poor fit especially compared to Terraform.
This is a feature, not a bug. Terraform is a tool to (reproducibly) enshrine your infrastructure in code. 'Minor manual fixes' are often left undocumented and known only to 1-2 people, and suddenly become major problems when your infrastructure comes crashing down and nobody knows why it used to work.
For me, the biggest thing is, when I go into AWS I struggle to find everything that is intrinsically linked to another resource. Say you have a lambda, to find which iam is linked to it, and what permissions it has is 2 separate tabs, then another for e.g. security groups, probably more tabs for other things. While using aws-cli makes it slightly easier, it's still a lot of effort to do this effectively.
With terraform I can look in one repo that has all the above, often in the same file too. Finding out what your infra looks like is a lot easier.
Regarding the state, you should not be touching your infra outside your code, if you do (e.g. while you're testing in dev), you should make the same changes in tf once you've confirmed it's what you want, and otherwise you undo those changes.
With further automation (e.g. tfcloud) you can even enforce these things by auto applying workspaces which ensures manual changes are always undone.
In my opinion if you're doing manual fixes you're doing it wrong. Let's say you do your manual fix in your Dev environment. Do you remember to do it in Prod/whatever other environments you have? Are you sure you did the EXACT same thing? Did you change 5 other things trying to fix it first?
You end up with so many different deployment environments that are unique 'snowflakes', and when something breaks in one it might not affect the others cause they're in totally different states.
It's a nightmare.
In my opinion, infrastructure as code is the only way to do it in a serious environment.
* Make your changes by hand
* Right afterward, run "terraform plan" to see how Terraform would undo your changes
* Edit your Terraform config to reflect those changes, and run "terraform plan" again to make sure you caught everything. Repeat until it's a no-op.
Now you've got a log of what you've done in a Git-ready format, and you can repeat it elsewhere, and you've learned how to make that console change in code.
For example, at my work this is what i do to apply changes to our AWS setup:
1. Fetch the latest version of our git repo.
2. Create a new git branch named after the Jira ticket im working on.
3. Solve the jira ticket by modifying the terraform code accordingly.
4. Submit a pull request and assign one of my colleagues as reviewer.
5. They review my solution, tell me to correct some issues that there might be, or straight up approves my solution.
6. My PR is merged into master.
7. I download the latest master version and apply the codebase.
This way we always have at least two people verify any changes to our infrastructure, minimizing the risk of fuckups and ensures solutions are as good as possible.
Now, managing changes in code doesn’t look too far from dealing with kubernetes’ json/yml config and applying them to the current cluster, provided it would be trickier when expanding to multiple cluster or doing complex orchestrating.
I guess TF makes a lot more sense for on-premise, bare metal VMs ?
There seems to be some kind of https://github.com/aws-controllers-k8s/community in early beta, but I don’t know how much it supports so far or whether it can diff a resource and modify it in-place.
No, not necessarily. If you have an API for orchestrating your on-premise infrastructure (openstack etc), then sure.
Where terraform shines is managing resources in your cloud provider.
I.E. creating the GCP project and GKE cluster that you need before you can apply your k8s YAML. Or creating the cloudsql databases, GCS buckets, etc that the apps running in your k8s cluster need.
Once you have a cluster up and running you can just use kubernetes yml to manage the cluster.
It is possible to also use Terraform to manipulate kubernetes resources (or AWS ECS or another container platform) but I personally like a clear separation between infrastructure (bringing up the environment using Terraform) and scheduling work on a cluster (using kubectl)
One part of the cluster building that caught me off-guard with GCP is that cluster options effectively change over time, as the k8s versions march forward.
For instance some feature go out of beta and become the default, and trying to rebuild a cluster exactly similar to one built year ago would require specific “don’t use that new feature” flags added to it. Or using these new features require adapting the other resources to the new configurations.
I guess TF is still useful in that it helps audit and reproduce the same infra inbetween changes, and will be nicer as an interface than bash scripts, even if it won’t save from dealing with nitty gritty of the platform.
Yes, the "provider churn" is real: defaults on cloud platforms can change and platform-specific TF providers can change as well. The way I usually deal with that is to make sure my TF repo's stays in shape by applying them regularly. If you haven't applied a TF repo in over a year chances are real that your repo has rotted. In a similar way to an old Ansible playbook that has rotted because some apt packages have new dependencies. Keep applying them regularly and manage changes in small chunks instead of once a year.
Auditing is really useful. I would recommend preventing developers from applying TF to production directly, and have it all managed by a CI pipeline.
Also with some careful planning and structuring of your TF repository. It's pretty straightforward to duplicate environments: for example: spin up an extra development environment to experiment with a newer K8s version, or to validate some infrastructure changes.
Sure. And also for Kubernetes json/yml. You don't just create infrastructure but it will also change -> delete/create.
And establishing a process how to delete your Kubernetes json/yml files in some form of codereview or in some way so you know what it actually deployed, you end up with a tool like terraform.
That's why I like Terraform.
Edit: "Also, I created a thing 2 years ago and now I need another one. Oh, here it is in Terraform." It's incredibly nice not to have to re-learn how to make that thing, and remember what its quirks and dependencies are.
In short, I consider Terraform exactly like I consider shell scripts: it's not perfect, not by a long shot, but once you have it working it tends to stay working and you don't have to invent it again the next time you need to do something.
Let me add, this thing was changed 2 years ago to add an option...why? Git blame to the rescue...
I probably use git history more in my tf code than in my regular programming.
The naive approach that tries to do this without state in code goes something like:
a) invoke the remote API to look for something that should exist b) if it doesn't exist, invoke the remote API to create it c) use the property of the thing to do something else
What Terraform does is cleanly separate your state (don't look stuff up you already know exists) from the API (which is handled by an open-source library, aka the provider) from your configuration (which can now be declarative). Because the state and API calls are abstracted away, the configuration itself is much more clean, easy to reason about, and easier to maintain.
I use Ansible to manage multiple clouds (Openstack, AWS...etc) using a mix of custom modules and public collections. I don't need a "state", I couldn't care less if resources exist or not, upgrades between versions are smooth, module/collection upgrades doesn't interfere with all the existing resources we already have... every time I run a playbook I know that everything will end up just like I want it to be. Not bad for a naive approach I guess.
Terraform feels like infrastructure management to me. We use it to provision underlying resources: Networking, Clusters, Nodes, Alerts, etc. All of the actual code deployments are entirely separate.
Ansible is more of a configuration management right?
Edit: seems limited to ec2 and s3: https://docs.ansible.com/ansible/latest/collections/amazon/a...
Although you could provision other services with some custom modules using the aws-cli.
Here's a comment from the author of ansible when terraform was first released: https://news.ycombinator.com/item?id=8100036
> One of the things shown in the Ansible examples are how to do a cloud deploy in one hop, i.e. request resources and also configure the stack all the way to the end, from one button press, and can also be used to orchestrate the rolling updates of those machines, working with the cloud load balancers and so on, throughout their entire life cycle -- all using just the one tool.
You have to add code to as only to be sure to remove the non-longer needed resource. But how long does that code need to stay there.
Ansible is supposed to engender a decorative approach, but it’s very easy to slip into procedural code. Whereas terraform is much more declarative.
There’s also a risk that your legacy environments only work because some dangling resource wasn’t cleaned up, and a new clone of the environment will fail.
What if instead of a dangling dns record, its 15 large EC2 instances?
Yes, you can come up with examples of trivial dangling resources, but it's just as easy for me to come up with non-trivial examples of dangling resources.
The number of articles I've read about someone who left a non-trivial number of resources running unused in AWS and were later surprised by a large bill would seem to be a counterexample to that point.
You should care. This opens you up to subdomain takeovers, which have real security implications.
https://developer.mozilla.org/en-US/docs/Web/Security/Subdom...
If you do this, or in fact anything with Ansible, be REAL careful about double-checking what your tags actually match before committing. Since it doesn't track state, anything in your cloud environment is fair game.
I was not careful once, and that was a bad week for me.
I use tf and ansible regularly. I wouldn't call ansible upgrades exactly smooth, they deprecate features just like anyone else.
> It's just drivers for the cloud API in question
Exactly, and if you are on gcp, both the ansible modules and the terraform modules are even generated from one code base: https://github.com/GoogleCloudPlatform/magic-modules
The point of using TF for many is reproducible infra and change approval workflows. So avoiding manual changes via the web consoles is what people are striving for.
I think tf really shines when you start using multiple providers to manage things outside of the cloud though. In my last gig I was for example using the workspaces feature and auth0 provider to have separate auth stacks for our different envs, being able to use values created by one providers resources in another's was nifty
>manually fix
You can if you need to. TF can ignore changes to certain aspects of resources like desired count of ECS tasks which might autoscale up and down and you can always make the change manually and then update the code to reflect the change. That's a bit of a no-no though. Like pushing code without a review.
Like WTF? Do they think I have better chance reaching them than them? Tried then TF and it was much simpler and everything worked as expected.
On AWS though, seriously, TF is inferior.
I find this to be the best advantage to Terraform. These small changes make your infrastructure harder to re-produce in the future; and unless you have strong documentation rules, you lost what is there in your infrastructure.
I had an old boss who fixed minor mistakes in stored procedures on the live server and then wondered why they broke again when builds went out...
`terraform refresh` does this.
(Disclaimer: I am an ex-core-maintainer of Terraform)