A tool that could manage complex TGW setups across many accounts would also be cool. Terraform can do that but it’s not the most elegant solution.
This has a couple of benefits. The first is that you don't need to coordinate IP address ranges in different VPCs, which is necessary if you're peering them together (unless you're using NAT). IPv4 address management in large networks can be challenging.
The second benefit is that although clients can communicate with the service, there isn't an actual route between the two networks. This better achieves the principle of least privilege at the network level; it avoids the security risks of connecting networks together such that any machine can connect to any other. When you design an application network to use PrivateLink from the ground up, you can start with the expectation that the VPC doesn't need any connections to any other networks, not even the Internet; it can potentially be an isolated network bubble, with only PrivateLinks allowing traffic in and out (using e.g. Session Manager for administration).
There are also practical limits on how large a federated network of peered VPCs can grow (probably helped by Transit Gateway), but there isn't any limit I'm aware of on the number of client/server connections that can be established with PrivateLink.
For these reasons I prefer to use PrivateLink when building systems that communicate across VPCs, falling back to network peering or Internet-facing endpoints in scenarios where it doesn't work.
However, one downside compared to network peering is that you need to set up a PrivateLink connection for each service that each client needs to call, though this process can be automated during infrastructure provisioning. Services need to have specifically onboarded with PrivateLink; you can't just deploy a service on the network and have it work.
But definitely a great tool in the toolbox.
> For low latency and fault tolerance, we recommend using a Network Load Balancer with targets in every Availability Zone of the AWS Region. To help achieve high availability (...) you can enable cross-zone load balancing. Cross-zone load balancing enables the load balancer to distribute traffic across the registered targets in all enabled Availability Zones. For more information, see Cross-Zone Load Balancing in the User Guide for Network Load Balancers.
When establishing a connection, the client can specify multiple subnets in which it would like to create an endpoint [2], and those subnets can be in any AZs supported by the service, if I recall correctly. The client can thus establish a connection that spans multiple AZ by requesting endpoints in subnets in those AZs. In most scenarios it's fine if client/service traffic can cross between AZs; I've only found it necessary to partition traffic by AZ in specialized applications (usually applications that are themselves trying to provide a zonal availability model, or that follow cell-based architecture at the zone level -- most business services don't fall into these categories, though low-level infrastructure services sometimes do).
[1] https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-se...
[2] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...
Aviatrix popped up on my radar recently trying to solve the operations side of this. Handles cross account TGW attachments and simplifies the routing.