AWS networking concepts in a diagram

Pretty tedious work though. I'd far rather feel I was actually doing useful work instead of dealing with this stupid crap.

Yeah this has been my observation.

CTOs bought the "fire all your sysadmins" literally, and now we have a 20% overhead on all app devs who are supposed to learn all this stuff.

It's crazy to me. It's literally the opposite of every management training / book about focussing on highest value work / where you add value / where you provide expertise.

I never saw it marketed that way - but even so, anyone stupid enough to fall for it deserves every NAT gateway charge they've had since.

For some APIs they return encrypted payloads with actually useful debugging info in, but there seem to be plenty where you just end up in a black hole of "computer says no"

They could put better information in Cloudtrail, at least.

Unfortunately with Azure, while authentication may be more obvious, 0x80003007 ERR_UNKNOWN type things aren't.

Also, what's up with the way almost every any change in Azure takes some random period, sometimes hours, to take effect and there's not necessarily any indication that anything is happening.

> I can grant a function app access to a SQL database by simply using the literal name of the function app (qualified by the object's identifier if aliased). No passwords, certificates, etc. involved anymore. It literally just works

I can't comment on the general correctness of your comparison (never having used Azure), but this particular one is flawed, since the same is true in CDK. That said, there's enough arcane weirdness in AWS' configuration requirements that I suspect your point is _broadly_ correct.

> It literally just works

In only a few toy scenarios!

For example:

Azure SQL Database, Managed Instance, and SQL 2022 in a VM all support AAD authentication.

Okay.

SQL 2022 supports only user accounts and not managed identities.

SQL Managed Instance requires an additional global AAD permission button to be pressed post-deployment or it won’t work. This can’t be templated.

The first two have AAD permissions controlled via their template (ARM API) — but only exactly one full admin group. All other permissions must be assigned via T-SQL. Again, can’t be templated via ARM. (Microsoft has better data plane RBAC for random open source products than their own flagship database!)

User access audit logs are easily collected for Azure SQL Database, needs manual T-SQL for SQL MI and is missing in action for SQL Server.

This is just SQL, which now at least supports user-assigned managed identities… if you include the latest SQL client which has a breaking namespace change and hence won’t work with third-party code. It also drags in 500 MB of dependencies.

Lightweight functions, you were saying? Hmmm…

Anyway, I got fed up with the whole thing when I tried hooking up a Storage Account in the same way and discovered that it didn’t support UMSI in connection strings — you need two code paths for dev-time and runtime!

Up until recently the whole thing was an unstable mess. Authentication would take 5 seconds in some scenarios and then wouldn’t cache the tickets properly.

It turns out that every Azure team is implementing the whole thing separately! It’s not at all like traditional AD where the platform provides the authentication functions in a uniform way and apps can just pick up the ambient tokens.

No, in this shiny new world every service does RBAC differently, connection strings differently, auditing differently, caching, config, and on and on.

All of this within 100% Microsoft products and services.

It’s a poorly thought out mess.

Exactly. The tools exist, they just don't make them available to customers.

I share this perspective. Most boxes in this diagram are AWS names for a thing existing in normal datacenter or TCP/IP networking. I am an old schooler who personally never worked a lot with AWS, but I think this is pretty clean.

I thought the other goal was to make it easy to get in but very difficult to leave so you are practically forced to keep using their services. Customer capture.

AWS used to be not only sane but elegant: every instance had an entirely-arbitrary internal/private IP address, some could optionally have a second public address, and which instances (including external IP addresses) could talk to which other instances was entirely and solely defined by security groups (as well as, of course, any OS-level firewalls that you'd generally disable), which were pretty much just flexible and reusable firewall rules where the concept of a "security group" replaced entirely the concept of a "subnet", which became an obsolete legacy concept.

They needed to support multiple adapters per instance, which they later added (maybe with a separate security group per adapter, which they might support now but I don't know off-hand); and they also needed hierarchical security group inheritance (the same way traditional subnets can nest into each other), which they didn't add but I guess you can now simulate them (though this sucks and I think is part of the downfall of the elegant stack) using multiple non-hierarchical security groups (which was not supported originally: security groups were permanently fixed in a one-to-one relationship with an instance).

This original elegant cloud-first model of instances and groups made network engineering pleasant for once... even fun! I remember thinking how great it was that all of my arcane physical networking and routing knowledge might soon be obsolete: that I could now think in terms of the abstractions of instances and how they talk to each other, drawing abstract circles around them without having to think about limited address spaces, and that they would assuredly fix the only two shortcomings of the original model...

...but then the network engineers showed up in force and ruined it all. There is simply no good reason for all of this VPC IP-address subnet focused insanity once you go cloud: they are just re-instating all of the frustrating limitations that come up when doing real world network engineering, presumably because they weren't willing to throw away their knowledge and realize all of that stuff is obsolete.

Like, seriously: we want to be able to replicate some enterprise network? That's madness, and it makes it all worse for everyone that this is even a supported goal. This is all virtualized networking: we don't need to be thinking in terms of subnets and gateways, we don't need to be manually configuring our egress... if you have a ton of hubs and routers and have to run cable all over the place, it makes sense, but this is the cloud!

And so now we all actually had to brush off all of that networking knowledge I was happy to give up as Amazon deprecated and fully removed "EC2 Classic" and have forced us all into this VPC insanity; and maybe if you never really tried to grok how AWS worked 15 years ago when it wasn't pretending to be a pile of legacy networking equipment you just shrug and accept that this somehow is all necessary, but it really isn't.

What they need to make are standards. Then it becomes technical knowledge for civilisation and not just single-company product knowledge. It would be great if cloud companies made cloud networking standards. I mean big competitors used to collaborate and make standards before but all that has petered out over the last couple of decades.

My suspicion is that they have distinct names/skus because (a) they're billed differently (b) they represent different security choices. If I grant a developer's account the ability to create "virtual router" entities, that means said developer can do all kinds of things I may not approve of. If I grant them only the ability to create "Internet gateway" entities, I know exactly what I'm getting into and (more or less) what I should expect the bill to be for that

I do agree with you that having a separate "egress-only Internet gateway" for IPv6 is dumb and confusing. I'm sure there's some number of pizzas which explains this Conway's Law instance

And that’s part of the problem IME from the customer side. Many of the largest pockets come from those with the worst technical debt and cultural issues of past mistakes and trends all done precisely while trying to avoid said mistakes. While part of the point of many different vendors is for resiliency as an enterprise much of the complexity is frankly various forms of proprietary vendor lock-in tactics trying to entrench themselves in some way to these high value, low churn accounts which starts to become more of a politicking or spear phishing exercise to the C-suite than anything about the solution itself. Essentially, every other bad technology decision in hindsight comes from many of the underlying problems that cause people to develop rather poorly thought out, shallow opinions unrelated to technology - confirmation bias.

As such, sales to me is one of the ultimately lucrative forms of social engineering attacks that oftentimes leaves the victim feeling like they got something when they really are just paying to feel like someone’s solving something for them.

Can't agree more, if decision makers would more often ask themselves 'should we do it' instead of 'how we can do it'. But engineers gotta engineer and managers gotta manage or they are obsolete.

But then we would have fraction of projects and work and just evolved systems created decade or two ago, with much lower IT costs. A total nightmare I say.

Either Lightsail, or just the default VPC that all AWS accounts come with.

If you think your use case needs private/public subnets, NAT, complex routing tables, etc. you can still add that to the default VPC later.

A few years ago, when what we now call EC2 classic was a thing, there were no sophisticated networking options at all. I assume VPC and its services were introduced because customers asked for something that feels like traditional datacenter networking.

> Isn't this sort of what Amazon Lightsail is? (Haven't actually used it, but am under the impression it abstracts away networking config).

Yes. Many services abstract networking. App Runner, for example, even gives you a public URL with HTTPs. Lambda also can do that nowadays. Most managed services just expose endpoints.

> I'll probably get downvoted, but isn't global addressing + firewalling basically available by creating a VPC with only public subnets, and using security groups as your firewall? (Best practice is public/private subnets with NAT gateways, but it's probably not impossible to rely only on security groups)

I will also disagree with what others are saying. You can keep things simple in AWS if your use case is simple and never care about 99% of what's on the AWS VPC dashboard.

If your use case is complex (you need to connect several internal private apps, expose private things behind a single IP / FQDN, do load balancing, use private DNS, connect with other clouds and on-premises, segregate, monitor and authenticate traffic...) then using physical appliances would likely require more hard work and expertise.

A fun exercise to folks complaining about AWS complexity: go to Fortinet's web site and try to find out which appliances you should use to secure a small company (~ 100 employees, couple of public internet facing services, couple of internal apps), how much it costs and how can you buy it.

It may be possible (though there are practical impediments as well, such as the poor IPv6 support), but the expectations and culture in the AWS world are so ingrained that you'll be looked at funny and constantly have to defend the "unorthodox" choices.

And the network failures happen more with wider impact?

Just use normal routable v6 space.

Thanks!

Yes the book I made the diagram with had lots of these drawings. IDK I preferred a mind map to solidify my understanding :)

> What I don’t personally get with Azure is why the “default” settings for enterprise isn’t that everything is not on the internet. Then when you need to do something on the internet, you’d need to open up.

Because every major cloud provider made the mistake of using IPv4 networking for backwards compatibility with their customer's software from the 1990s.[1]

What they should have done is used IPv6 by default, and provided each customer subscription/account with a /56 in each region sharing a common prefix, e.g.: /48.

All of the private networking complexity just evaporates and turns into trivial firewall rules.

The "allow our vnets only" rule becomes a single /48 subnet allow rule. Want to trust a foreign tenant? Add their single /48 prefix and done! No NAT!

Service Endpoints and Private Endpoints are required only because IPv6 RFC1918 private ranges are too small, and hence overlap between customers. Remove the overlap and you remove the need for any kind of tunnelling or SDN magic.

No more split DNS either: the AAAA records are always the same for every name, whether "inside" or "outside" the network. At most, you might need "private records" that resolve only if the request query comes from the customer's /48 prefix.

And on and on...

IPv4 ought to have been an optional compatibility add-on, a side show to secure-by-default IPv6.

[1] Sadly, there's still software being written today that doesn't work with IPv6.

yea idk if you heard, there is this new thing called "zooming" were you essentially just project a portion of the image onto your whole screen.

sadly it is forbidden by the creators of the website

ClayWS

Or, they can be considered as a great way of understanding the tools in your toolbelt when working in that environment.

You don't need to. Solution Architect associate and upwards cover what you need to know.