rcrowley

Glad we cleared up our terminology! I agree that “root module” risks ambiguity, just like you point out.

I just realized I never responded to the very last point in your original comment. I don’t have, and I don’t think Terraform has, a complete solution to dependencies between root modules. Fortunately, data sources will at least fail if they don’t find what they’re looking for. For me, these failures never come up in production since I’m never using `terraform destroy` there. It does come up in pre-production testing and that’s an area that seems rich with patterns and products on top of Terraform that are controlling order and dependence between root modules.

PS thanks for your work on Terraform and Kubernetes.

I stuck with my typical term, root module, synonymous with how folks are using “stack” and “state” in various parts of this thread.

A module is any directory with Terraform code in it. A root module is one that has a configuration for how to store and lock Terraform state, providers, and possibly tfvars files. Both modules and root modules may reference other modules. You run `terraform init|plan|apply` in root modules.

I think my comment makes sense in that if you mix two services into the same root module (directly or via any amount of modules-instantiating-other-modules) you can end up with changes from two people affecting both services that you can’t easily sever.

Happy to clarify further if I’m still not addressing your original comment.

(Hi, I’m one of the authors of the article at the root of this thread.)

Considering your hypothetical stateless microservice change in the same root module as stateful services, problems arise when _someone else_ has merged changes that concern the stateful services, leaving you little room to apply your changes individually.

It’s also worth remembering that, even if a stateless service and a stateful service are managed in the same root module, applying changes is absolutely not atomic. Applying tightly coupled changes to two services “at the same time” is likely to result in brief service interruptions, even if everything returns to normal as soon as the whole changeset is applied.

(Hi, I’m one of the authors of the article at the root of this thread.)

I’ve gone back and forth on workspaces versus more root modules. On balance, I like having more root modules because I can orient myself just by my working directory instead of both my working directory and workspace. Plus, I feel better about stuffing more dimensions of separation into a directory tree than into workspace names. YMMV.

It’s set in Computer Modern, the font Donald Knuth designed for TeX and which you most often encounter in academic papers.

All rings true, to me.

We created a second GSuite for GCP at Slack because we didn’t want email and other corp IT assets to be mixed into what could’ve (but didn’t) become production infrastructure.

Honestly, having lived a parallel life to the Windows ecosystem, TIL about “red forest.” I do think, though, that cross-account AWS actions are much more first-class than it sounds like jumping between forests ever was.

The same link’s in the second sentence of the article. But, sure, I forgot.

In the meantime, check out Substrate <https://src-bin.com/substrate/> and don’t worry about waiting for AWS to improve.

Interesting. Thanks for the detailed response. Another, positive way to look at one aspect of your architecture is that the AWS account boundary prevents most cases of dueling configuration management, with two tools changing the same resource back and forth forever.

I’d love to hear more about your experience with shared VPCs. What’s inherently unstable about them?

That UX is atrocious.

Substrate [1] instead presents you with a list of all your accounts with a link to assume your role in that account in the AWS Console (and parallel tools for assuming that role in a terminal, too).

[1] <https://src-bin.com/substrate/>

AWS recently added the organizations:CloseAccount API (albeit with some caveats discussed elsewhere in this comment tree).

Actually, as of Friday, that’s no longer true. \o/

<https://aws.amazon.com/about-aws/whats-new/2022/09/aws-updat...>

Curious / product research: Are your 38 accounts all in the same organization? Do you have any human IAM users left or is it all IdP, all the time? Do you use Terraform or anything like it?

Also, yes, a pox on the single-player AWS Console. I’ve at least found a way to logout from one account and login to another in the same motion but it’s still a poor experience.

Service limits for regional services like EC2 are regional.

Service limits for global services like Organizations appear to only be manageable from us-east-1.

This is a bummer, yes. I haven’t looked but I wonder if that 10% is a soft limit.

At any rate, this is a good reason to use accounts for architectural divisions, not teams, and certainly not individual engineers.

Control Tower is cool if the problem is “I need lots of AWS accounts.” Substrate [1] is cool if the problem is “I need to accomplish something and I’m cool with using lots of AWS accounts to do it.”

[1] <https://src-bin.com/substrate/>

Actually upgrading your Support plan is one of the very, very few things you still need to break into the root of each account to do. However, if you’re big enough you just sign an EDP contract that forces all your accounts onto Enterprise Support anyway.

You’re absolutely right, I’ve fucked up plenty. That’s why I believe so strongly in making the right thing the easy thing. I think I’m doomed if it’s critical for tags to be perfect because they always drift, doomed if IAM policies must be least-privilege because sometimes that’s impossible, and doomed if I can’t adapt what I built yesterday to what my business needs today.

PS good job partitioning your EKS clusters, even within one account. That’ll save you some sleep one day, I’m sure.

As the other commenter notes, no, you still have just one bill.

Better, though, that one bill is broken down by account so you can see where the money’s going.

Author here: I have settled on account per service per environment with a couple of exceptions.

Sometimes I run multiple services in one account if they’re so tightly coupled as to be useless as a group if any one is down. (This has practically come up when two services are codesigned to multiplex TCP connections to support tens of millions of clients.)

Sometimes I run a single stateless production service in two accounts and route 10% of traffic to the canary account and 90% to the other one.

Substrate [1] is meant to lessen the investment required to use lots of AWS accounts. Would love to know how it looks to you.

[1] <https://src-bin.com/substrate/>

Substrate [1] is meant to help folks not make a mess of lots of AWS accounts. Would love to know if it feels less enormous.

[1] <https://src-bin.com/substrate/>

actual lol

Synchronizing ~/aws/config gets harder and harder as your team grows because there are both more people who need to receive changes and more people making changes.

I think the human-readable names for AWS accounts need to be part of the account, not part of the laptop. Substrate [1] does this so that you can type commands like `substrate assume-role -domain example -environment production -quality beta` [2] to get where you're going.

[1] <https://src-bin.com/substrate/> [2] <https://src-bin.com/substrate/manual/moving-between-aws-acco...>

I have thought a great deal about whether I also want EKS clusters to officially support nodes in multiple AWS accounts. On the one hand, having the option to create that additional low-level isolation would be lovely, even and maybe even especially if I didn't always take it. On the other hand, isolating two things from each other but then tying them to the same Kubernetes cluster upgrade schedule feels wrong.

In the end, I decided that if I care about isolating two things enough to put them in separate AWS accounts, I'm willing to spend the $75 per month that it takes to have separate EKS clusters, too. (This opinion perhaps obviously doesn't fit will with hobby/side project budgets.)

I'm not honestly sure how AWS Organizations interacts with the AWS free tier.

Rest assured, though, having lots of AWS accounts (and using AWS Organizations, their service designed to _help_ you use lots of AWS accounts) is _not_ against the terms of service.

Author of the article here: I agree. GCP projects are a better abstraction. I still think AWS is, on balance, a better cloud.