We have dozens of completely independent applications made up of hundreds of micro-services comprised from mergers and acquisitions, different market segments, etc.
Going from wildly disparate stacks to a unified core, while improving reliability and _real_ DevOps/GitOps/SRE principals (which are very different from old-school System Operations practices), we've independently adopted a model with elements of this.
> This article describes a systematic way of applying terraform at scale. At scale refers to:
>
> 1. high technical complexity: deploy infrastructure to any number of accounts and cloud providers
> 2. high organizational complexity: enable multiple teams of developers to work collaboratively
It's making sense of the complexity that requires strong adherence to _reusable patterns_. We also use Terraform for "Monitoring-as-Code", so we use it in non-cloud provider use-cases as well.
Avoiding monolithic Terraform does come with an additional layer of complexity -- absolutely -- but we also don't have cookie-cutter services. We modularize the reusable bits, adopt _dependency injection_ (as opposed to tightly-coupled modules), and have adopted a more _composable_ approach rather than building large monolithic modules that do all the things. There are similar concepts in more mature software engineering organizations (e.g., SOLID principles). Updating a Lambda should not require executing the same monolithic Terraform as what manages your RDS instance. So we break apart the execution of Terraform into chunks which solve a single orthogonal problem.
Every module is well-tested using Terratest (Golang), and we leverage Terragrunt to solve issues with state management, reducing interdependency, and being able to use the output of one unit of Terraform as input to another unit of Terraform _in an entirely separate run_. This helps de-risk our infrastructure changes, and allows infrastructure to deployed _entirely separately_ from the application containers, while still leveraging things like ECS Capacity Providers in AWS (which uses container traffic to dynamically scale the underlying EC2 instances).
But for this to be successful, we've had to democratize the tools. We will never have enough headcount to manage all of the day-to-day ops work for all services. So we've adopted what Netflix calls "full cycle development" where dev teams have taken over day-to-day operational needs (deployments, AMI updates), which has freed-up our SRE team to focus on tackling the harder R&D reliability projects and providing solutions as software. It also moves us from a "man-to-man" model to a "zone-defense" model (to use a sports metaphor). It allows us to scale our team resources better.
Development happens in the open (internally) with public (internal) repos, and we encourage discussion, issues, and healthy disagreements to flourish (as long as those disagreements can come to a resolution -- we don't leave things hanging for too long). It becomes an interactive experience between dev teams and SREs to _work together_ to make sure that what we're building is solving real problems that engineers face.
We're not perfect, and there is always more to do, but this has been an absolute godsend to helping us improve how we do things at our scale and with our resources. By building best practices into the base AMIs, shared Terraform modules, Monitoring-as-Code, and other tools, it helps the overall engineering culture move in a healthier direction. We've been able to _dramatically_ cut our AWS costs as a result, leading to more jobs — even during the pandemic!
For anyone reading this blog post and kinda freaking out, I would recommend setting aside your biases and thinking through how a different kind of model may actually be able to _help_ you. Because it worked for us.