AWS Customers Cannot Escape IPv4
tty.neveragain.de
tty.neveragain.de
The Principal PM in charge of the "regionalization" effort was asked in a Q&A "why didn't we just switch to IPv6?".
Her answer was something along the lines of "The number of internal networking devices we currently have that cannot support IPv6 is so large that to replace them we would have needed to buy nearly the entire world's yearly output of those devices, and then install them all."[0]
It's easy to presume malicious intent on the IPv4 front from Amazon, but with so many AWS systems being on the scale they are at, I find it easy to believe that replacing all of the old network hardware may just be a project too large to do on a short timescale.
[0] - At least, that's my memory of it. I'm sure that's not an entirely accurate quotation.
surely they started the process...
right? i cannot imagine AWS just sticking head in the ground and ignoring this...
0 is also a number.
About 18 months ago, the requirement came that federal agencies are required to be IPv6 Only, dropping the dual stack. IIRC they have until 2025 to do that. This has the neat effect of forcing all vendors to make IPv6 a first class citizen. The extra little fun from this is that it applies to the military JWCC contract that all the major clouds have been trying to land. The timescales of JWCC meant that initial offerings are pretty bare, but that won't be allowed to last.
I work a federal entity tied to DoE and that's the biggest workstream cut out for us. 90% of our environment is either dual stacked or IPv6 native. We would love to kick IPv4 out under us and go full IPv6. Problem is that the vendors who are largely private don't have the same mandate so there's varying degree of "we support IPv6" which makes planning bit more difficult (especially at the discovery stage).
They get to decide how much that sweet federal $$$$ is worth to them. For most vendors, it's hopefully worth too much to ignore.
Sounds like a perfect opportunity for a market upstart to start out v6-only...
You're right about the cost and need to replace network equipment being one of the strong reasons why they didn't. Amazon used its own in-house designed and built network gear for a variety of reasons (IIRC there's a re:invent talk about it), which I'm sure is probably still the case. Every single one of those machines had fixed memory capacity and would need to be replaced to bump up the memory sufficiently large enough to handle IPv6 routing table needs etc. What they had wouldn't even be enough if they'd have chosen to go IPv6 Only (which you couldn't get through except via dual stack IPv4/IPv6 anyway).
I'm not privy to details, but I recall once when a mandate was issued to a Java platform to remove an outdated encryption protocol (mandated by Amazon Infosec). The change was made and rolled out with little fanfare.
A few weeks later, a large outage of Amazon Video (which used said platform) occurred on a Friday evening. Root cause? The network hardware accelerators were only setup to use that outdated protocol, which in turn meant that encryption was happening in software instead. Under load, the video hosting eventually caved.
Might be specific to the hardware used for Amazon retail, but it reinforces the point of their home grown (and now aging) stack.
I think I switched teams before that was resolved and moved to some greenfield work where we didn’t have to worry about scale for a while, but I do believe they eventually figure it out.
I’ve got a slight suspicion you were given some bullshit or at least a creative treatment of facts e.g. everything had IPv6 support but FUD-filled network engineers didn’t want to turn it on.
Most network devices I’ve encountered were dual-stack way before anyone I knew seemed to care about actually using IPv6 — I always assumed it was added for US government/military requirements.
Also, AWS now have the majority of NICs and switches built in-house I imagine. The underlay network could be IPv6 or totally custom for what we know (but probably is IPv4).
As for AWS, I tend to agree with the sibling post and your supposition about IPv4. Everything out of the Amazon organization is aggressively, err, "minimal."
There were also other reasons given, like the amount of internal software that used e.g. IPv4 addresses. Also, AWS likes to have 'lots of small things' instead of one big thing (regions, AZs, cells, two pizza teams, no (official) monorepo) so regionalization was part of that.
Another big reason for regionalization, other than IPv4 exhaustion was that AWS promises customers that AWS regions are completely seperate, but with one big giant network, it turns out there were all sorts of services making calls between regions that nobody had realized. I have a couple of funny examples, but that might make me too identifiable :)
so they don't own 2 pizzerias? :(
Pepperoni.16xlarge
IIRC, early on on that project the gateways would get overwhelmed at the volume of traffic they were handling between various VPCs and had to be rolled back several times early on.
Of all the transitions I dealt with at Amazon, snowfort may have been my least favorite (though the ACL/role migration was pretty frustrating as well).
I'm not saying there aren't v6 issues (for some vendors, resource exhaustion might have come into play) or bugs, but there's no way it's that massive a problem. There are huge and complex all v6 networks all over the planet that have more stringent requirements (by law) than AWS DCs.
It was indeed a lot of work. But worth it.
* When I was there we still had a handful of weird things that couldn't be made IPv6. If you needed to access such things you could get a dual-stack dev server.
The mitigation was to sort routers by hostname which began with the regional airport codes (iad, pdx, etc.), and pause for 15 minutes each time the first three letters changed to give folks on-call time to react.
EDIT: But maybe bugs, IDK.
FUD sounds like a mean way to say unproven in production
Tables were relatively large internally because AWS was all in on clos networks at that point. And the devices used to build those clos networks were running Broadcom ASICs, not Cisco or other likely vendors.
Talk about nominative determinism...
The world is bigger than your apartment.
Often IPv6 and IPv4 paths were entirely different and latency on IPv6 was much bigger, so we had to measure latency between nodes on both. Also, sometimes IPv4 was a symmetrical, but IPv6 wasn't. As a result, we had to buy tons of IPv4 addresses.
Our control plane was on IPv6, but data-plane had to be on both.
There is no reason any company of any size should run out of IPv4 addresses internally, IF they are doing proper IP management. If I were to wager a guess I'd say there was a lot of waste going on, issuing /24s or larger to teams when all they need are /29s etc. It adds up over time. Once they exhaust private IP space they can always buy more at auction. They are Amazon after all, there's no shortage of money. This is just mismanagement of resources.
If you wanted to assign a single non-routable IP in the 10/8 space to each of those cable modems, they would be 13 million IPs short.
I'd imagine few service teams at Amazon would get very far with a /29, let alone a /24, if they have to put all their stuff on that.
If that is the case, then Amazon should hold off on charging for IPv4 on a short timescale until they have replaced all the old hardware and can support IPv6 internally everywhere.
The better reason is the regionalization was probably a way to decrease blast radius in case of a service failure.
Also, AWS definitely did not regionalize all their services in 2016. IAM and certainly not DNS/Rte53 (part of the reason why they had their massive failure in US East 1 2-3 years ago)
In the world of their recommendations, even the concept of a "public ip address" is a red flag, and AWS even recommends (for an added cost of course) tooling to flag and "mitigate" them. These provide a strong lock-in effect when customers spend effort to build the complex infrastructure for them in the name of security, even though in reality they hurt security through unnecessary complexity, addressing ambiguity, etc.
There is no such typo i could make with my single shared public ipv4 address because it’s just one address. Saying “allow” by accident isn’t enough, I’d have to somehow accidentally configure the particular ingress port to NAT to a particular internal machine, and even then it would only affect that machine and no other.
(Full disclosure, i actually like IPv6 and am in full favor of everything moving to it. This is in spite of the above, but i at least recognize that the above is the case.)
IPv6 isn’t amazing, but it makes many of these problems simply disappear. Of course [0] networks should be isolated, but this should be achieved with a firewall that, by default, disallows connections between the public Internet and private networks. And that’s about it — every VM has a globally unique address, routing just works, one company (if permitted) can connect to another company’s endpoints, firewalls can be deployed where they make sense instead of being forced to exist exactly where inconsistently-addressed networks meet, etc.
The entire mess of designing and negotiating allocation of extremely limited IPv4 addresses for private systems simply disappears!
[0] Beyond corp has something to say about this.
Private networking is good. IPv6 doesn't help here at all.
I'm not saying that running a private network doesn't provide genuine security value, only that it drastically complicates your networking architecture for very little security benefit. Organizations can decide whether that trade-off is worth it, for organizations with deep threat models like militaries and banks, it's probably worth it. For 99% of the private sector, it's folly.
In the first case, you need a firewall for IPv4 anyway. In the second case, that would also work with IPv6.
Disclaimer: I know nothing about Azure/AWS internals.
Hopefully one of the big cloud providers figures it is in their best interest to have a much bigger address space and make all this busywork sinecure obsolete.
And why are unique MAC addresses a problem?
(Though that WP page seems also to have self-coined the "private network" phrase and I don't think it's an estabilished term in this meaning. The first and second references off the leading paragraph talk about "private internets" and "unique local addresses" respectively).
If I know the external, publicly addressable IP address of your router (e.g. 135.77.9.106), and no firewall whatsoever, there’s nothing at all preventing me from doing `ip route add 10.0.0.0/8 135.77.9.106`, and voila, I’d have a route to your “private” network.
Using private addresses vs globally unique offers no security benefiy whatsoever.
This only works if you are on the same L2 segment as 135.77.9.106, or control and install this route on every router between you and it. Otherwise, 10/8 will get routed to the next hop for 135.77.9.106, i.e. your local gateway, which won't know anything about the intended 135.77.9.106 destination and will route it normally (which likely means dropping it).
It's true that firewall rules should be in place to prevent this attack from your direct neighbors, but it's not possible to perform it over multiple hops that you don't control.
If your (default) gateway from one network segment to another network segment only has one rule, default-deny, then it's not a problem. If you think that's not enough, then use IPv6 ULA (fd00::/8).
But why should the incompetence of some customers limit what all customers can do?
There's nothing preventing you from having a private network using unique address space that's either blocked from accessing the internet via a firewall on a router or just plain not even routed. You could even use ULA networks with stateless prefix translation to avoid using GUA addressing for your private network.
The sad part is that IPv6 support is abysmal on every cloud so just migrating to it imposes serious limitations as addressed by the blog author.
I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale.
Literally none of those products would be required if they had just made IPv6 work properly.
Instead, they NAT IPv6, so you can't even use it to avoid the NAT forced upon you by IPv4. They also release new products -- in 2023 -- that don't support IPv6 and likely never will.
There is still an entire page[1] of listed limitations in the docs: https://learn.microsoft.com/en-us/azure/virtual-network/ip-s...
Think about how insane it would be if this was the IPX -> IPv4 transition. Imagine if this page said "IPv4 limitations: All VMs must include at least one IPX address, etc..."
Sounds nuts, right? I was migrating customers to IPv4 from IPX in 1999, and IPv6 support materialised in about the 2001-2003 timeframe. It's been decades, but it still feels like 1999 and migrating Novell NetWare where we had to have IPX+IPv4 because "not everything supports IPv4 yet".
[1] This is definitely not all of the limitations. Most of their PaaS products don't support IPv6. Hence, any IaaS+PaaS solutions must use (mostly) IPv4.
Service Endpoint: Allows a PaaS service (that itself uses public addresses) have firewall rules for overlapping private vnet addresses. E.g.: You can have have two VMs both on 10.0.0.123 addresses (in separate VNets) using individual "Allow" rules to the target service. Essentially Azure tags the traffic at the VXLAN level with the source network ID on top of the IP address, making it a "fat IP address" that is unique within Azure and can be used in firewall rules.
Private Endpoint: Makes a PaaS service appear on a private network address range instead of the default public range. This allows your on-prem firewalls to isolate your specific PaaS instance from other customers -- otherwise the traffic gets "blended in" with everyone else in the same public service tag ranges. This also allows you to use your ExpressRoute fibre links to route traffic from on-prem to the public service.
In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks.
Fundamentally, all this exists just to enable the ability to firewall things. With overlapping IPv4 addresses and small shared blocks of IPv4 addresses with NAT behind them, it would be impossible otherwise.
With IPv6, using firewalls would be much simpler because overlapping addresses aren't needed any more. Similarly, PaaS services could trivially allocate IPv6 addresses per customer instance, so that customers could apply selective firewall rules.
That's the whole selling point of private links. How could you possibly missed that? That's exactly why companies onboard onto the service. They say exactly this exactly on the marketing brochure. That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild.
What kind of confusion reigns in your mind to come to the conclusion this was some obscure conspiratorial gotcha?
It boggles the mind how you felt the need to come up with absurd conspiracies involving IPv4 to arrive at a claim that the marketing pamphlets show front and center as their whole reason of existence: avoid traffic to go through the internet, and instead pay extra to go through private pipes they own.
It's as if the name actually means something!
I believe their point is that it doesn't flow through "the wild" in any case, it is probably routing within the Azure AS(aka Microsoft controlled networks). However, as you say, people line up to pay for it, likely for reasons having to do with their architecture/security model/compliance requirements.
My understanding is that if you don't have a private endpoint, your traffic to an Azure cloud service won't be routed out to the "big bad internet" per-say, but it will be routed within the Azure AS as mere IP traffic.
If you have a private endpoint to an Azure service in your virtual network, that means Azure has provisioned you a virtual NIC with some private IP address, and presumably alters DNS resolution within your network for that Azure service to resolve to the IP address of the NIC. The NIC provides (presumably encrypted) link layer transport out to the Azure service.
Compliance for some customers may dictate that there aren't any routes out to the public IP address space from within a network. If you still need access to cloud services, private endpoints are a necessity.
All that to say, I think Private Endpoints provide more than just a means of firewalling traffic/changing the IP address associated with a service; the actual transport from client->cloud service is fundamentally different.
If you want port forwarding, they recommend that you do... something. It's not clear what; what you can find on the internet is mostly just people complaining that they insisted to customer support that they needed port forwarding, customer support said they'd do that, and port forwarding still doesn't work.
But, intriguingly, it turns out that a Comcast router will also assign every device on your network at least one public ipv6 address. They also firewall all incoming ipv6 traffic, but unlike the situation with port forwarding, you can disable that firewall on the router admin page. (You can't put up your own firewall.)
Sounds like the CGNAT experience, possibly blanket applied even if you have a globally routable public IPv4 in order to have consistent behaviour and reduce network management / support case complexity.
I bought my own (plain) modem and manage WiFi/routing myself.
I get the direct Internet-facing IPv4 and IPv6 addresses.
No monthly equipment fee!
I'm not sure what you've been reading, but the concept of a private link has absolutely nothing to do with IPv4 vs IPv6. In fact, practically all your remarks don't involve the issue at all.
The most charitable interpretation of your comment is that you're making the mistake of conflating any application involving a virtual network as something caused or involing IPv4.
Hint: it's a complex workaround for insufficient IPv4 addresses.
One of the primary usecases and design requirements for this service is regulatory compliance. They say right on the tin that the service is designed to send traffic over private networks, including AWS's own global network. The whole point of private link is to ship data through the pipes you own, instead of routing it through the wild. I don't know how you could have missed that.
More importantly, you really need to want to use private link connections. This is a value-added service. You need to want to go out of your way to avoid your traffic to go through the internet to onboard both ends of your services to private link.
Not only are your conspiratorial hypothesis completely out of base, even your baseless assumptions have absolutely no relation with what version of the IP protocol is in place.
I'm the first to join in on any good old fashioned AWS/big cloud provider bashing, but these should be grounded on reality.
That is, when you have a customer in some network and a provider in another network, you had to implement full connectivity between the customer and the provider
With private link, you can remove all that connectivity, and instead expose the provider' service to the customer The service, nothing more, so just one endpoint
This is really good from a security point of view, but also for managing your stuff (especially if there are multiple teams in the compagny): because you now have a resource, you can easily list the services you expose to other people, and whom are your customers
1. get list of customer netblocks
2. setup "internal" service(s) for customers
3. setup firewall rules to allow customer <-> service allow list
4. setup DNS records
5. tell customers DNS and API targets
Any new competitor would have to cough up a huge upfront cost for IPv4 addresses, if they could even get them at that scale.
Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink
I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing...
Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain.
That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.
In the early days of IPv4 many big companies did not NAT IPv4. I was at a company that did this. Our workstations all had routable public IPv4 addresses. There are still some companies that do this. One of these was a company that had 20 managers and VP's on a call when I just wanted to give their network engineer a CIDR block and a pre-shared secret for a network VPN. I suspect they will be using public IPv4 addresses internally forever. And I doubt they will ever sell their /8's unless people comment on their Vogon poetry.
Amazon is probably one of the exceptions as they have so many geographically disperse configurations it would be harder to continue using RFC1918 address space. Not impossible, just difficult. I can think of a few other big companies that do manufacturing that are spread out around the world that would probably run into walls with RFC1918 at some point. I've seen some of them take over public address space for internal routing which then breaks access to some things on the internet thus requiring double/triple NATs. 1/8 assorted, 25/8 MoD, 26/8 DISA are a few I've seen.
A lot of big universities did this and even still do this to a large degree. They got huge IPv4 allocations early and there was no scarcity.
I don't know how companies are doing it but they are able to track your IPv6 changed daily or not.
Eh… If I was a company that wanted to use IP addresses to fingerprint users, IPv4 vs IPv6+privacy extensions both seem identical to me. Multiple requests from the same IPv4 address mean “someone, perhaps more than one person, from the same household/wifi”. Whereas multiple IPv6+privacy requests from the same /64 prefix means the same thing.
ie. You just consider the first 64 bits of the IP and can assume the same amount of information you already would assume from the IPv4 address. Just ignore the trailing 64 bits because it’s expected that they’ll be randomized/shuffled even from the same client.
Having NAT and a firewall gives you a better illusion of privacy. Sure you can track devices from the outside world, but its pretty hard.
If you have v6 configured in a certain way, then your IP address is basically a UUID for your machine. Plus you can't really just stop ICMP anymore so you can trivially ping it (caveats apply)
There are some annoying operational issues around it as well, common one: DNS hostnames for devices that only do SLAAC
If your network card breaks you switch it out, and you make sure your IPv4 settings apply to the new card.
If you fully rely on IPv6 you'll get a new address.
And if your devices self-update DNS then you have to make sure they pick the right address, as there can be many due to privacy extensions.
Lastly, combining privacy extensions plus hosting stuff is hard, as you don't know on which address a certain port is bound.
Not saying this is what you should do, just a common rationalization.
Pretty sure that's enough time to test it in development...
I'm curious how people do it btw, if you have tips to share, I'm all hear. Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.
> Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.
Yes you'd limit by /64 or slightly larger.
The live set of IPs shouldn't be very big.
If you go the extra mile and simultaneously track /64, /56, and /48 with moderately increasing thresholds, you'll probably end up causing less collateral damage when you block someone than with IPv4.
It's possible that cloud providers assign smaller ranges to their customers, so you may need to allocate more bits for granularity in that case; on the other hand, one might naively assume that cloud providers are more responsive to abuse reports than ISP's.
Cox & Spectrum only hand out /56. I'd hate to be banned because my neighbor did something bad and we happen to be in the same /48.
People see a 10.x and instantly know it can't be reached from the public internet. IPv6 is much harder. For internal-only stuff there is the fd00::/8 block, which AWS actually does use, but there is no equivalent range for outgoing-only connections.
Turns out it's a bug in the kube stack when containers have both ipv4 and ipv6 addresses.
This was last year.
I suspect the ipv6 cases aren't being tested very well across the software stack.
I continue to do that on all my personal equipment.
Learning a new framework doesn’t (often) break things for the end user in a way I can’t diagnose/reproduce. IPv6? Totally different ball of wax.
Also my ISP, a fiber gigabit provider and the best available in my region, doesn’t support IPv6. Until they, and others like them, get on board fully I don’t see customers having a snowball’s chance in hell of working cleaning with IPv6. I can only imagine how long it will take the shitty ISPs to fully support it.
AWS has no incentive to support IPv6 from a sales perspective.
Owning IPv4 addresses is a requirement of AWS’s core business. Unless people stop using IPv4, then AWS cannot sell those addresses. There is no incentive for the addresses to increase in value.
Further, if people continue to use IPv4, then AWS has to continue to acquire even more IPv4, and AWS wants the price of those to go down so that acquiring them is cheaper (or wants people to stop using IPv4 so that they can stop spending money on them altogether).
But if people stop using IPv4, the asset (billions of dollars by some accounts) becomes worthless... AWS are passing on the cost for public IPv4 addresses now, so there's even less incentive.
Stateful NAT is a real burden on bigger networks and is at least a chore on smaller networks. It at least doubles the complexity of managing a network especially when you have a DMZ that should be used from some "private" and some "public" endpoints.
How do you hurt security by preventing external access to your internal services?
In the v4 world, one can easily accidentally allow access by inadvertently sending traffic toward the wrong group of colliding addresses or otherwise messing up any of a number of things that ought not to be necessary in the first place.
Anything else is amateur hour madness.
Okay, in real life I need private addresses because I connect to things that are only available over IPv4. So there’s some negotiation to make sure that my private network does not have an addressing conflict with the other network, there are NATs in the way, and traceroute gives output that is every bit as bad as you would expect. The ACLs that everyone (arguably quite reasonably) sets up suck are fiddly because the clients don’t have well defined address ranges. When people allocate /24 subsets out of IPv4 private space, the probability of collision is annoyingly high. Amateur hour indeed.
I would take globally unique but “private” IPv6 addresses, over private links, with private routes (dynamic or static), and ACLs that actually make sense any day. Heck, I would happily go IPv6 only!
(a) addresses that are globally unique but not globally routable. (These are extremely common in IPv6. These are not so common in IPv4 because IPv4 addresses are expensive, so people try to minimize usage, so people will try to avoid using paid-for globally unique addresses for non-routable purposes.)
(b) addresses that are in ranges that are, per spec and actual usage, only even defined within an organization and are not globally unique. For example, 192.168.0.1.
(a) and (b) are not the same by any useful definition. Sorry.
Even without this, it's a lot simpler administratively. I have about 20 AWS accounts, all with their own VPCs, all using the same 10.0.0.0 block because... well.. nobody thought about this. What do I do if they need to communicate? (They probably won't, but...)
Private addresses offer no security benefit whatsoever. If you have no firewall, nothing at all prevents me from doing `ip route add 10.0.0.0/8 your.routers.ip.here`
You got faked out :)
IP is silly and refers to next hops by IP address, which fundamentally makes very little sense, because IP routing actually works by sending packets toward either whatever is on the other end of a point-to-point link irrespective of its address or toward a certain destination on a certain link, where that destination is addressed by a link-specific address (generally a MAC address). In common usage, the sole purposes of a next hop IP address are to identify the link (implicitly, while configuring the route) and to tell the router what IP address to ask for via ARP / neighbor discovery so it can actually route there.
With that in mind you are (on Linux, anyway) very much prevented from this particular mucking around:
$ sudo ip route add 1.2.3.4/32 via 5.6.7.8 Error: Nexthop has invalid gateway.
Because it's not actually possible to route a packet via a host that isn't locally reachable.
You can try to send packets using various encapsulation schemes to try to convince an intermediate router to decapsulate the packet and forward it to an attacker-controlled address, and someone manages to pull this off every now and then. Actually getting the evil packets in question to traverse the public Internet can be challenging but is not necessarily impossible. So the actual point stands -- relying on a private IPv4 address range to be unreachable by the general public merely by virtue of being private and without using an ACL is a mistake.
It isn't.
> and even obtain certifications
No, not really.
> just to set up everything required for private networking.
Frankly, no.
I'm tempted to agree only in one aspect, which is apparently you are completely unfamiliar with the topic, and the degree of confusion you are showing in your comments suggest you would benefit from an introductory course on the topic, or in the very lease a 5-minute read through the service's documentation.
Certification wold only help because you would need to learn the basics to pass those, and learning the basics would be enough to prevent you to fill in the gaps in your understanding with fabricated nonsense.
> The complexity of a properly configured service mesh in AWS is staggering, extremely hard for newbies to get right, and easy to fuck up big time down the road. That hurts security.
People being way over their head because they can't even grasp a FAQ will definitely hurt security, but the root cause of this failure mode is sheer incompetence.
As the saying goes, poor craftsman blame their tools, and here you are with a tool-blaming fest.
In case not: it's about doing it the wrong way (excess complexity and ambiguity -> hard to understand/analyze/monitor). Using globally addressing is orthogonal to controlling access to your internal services - you can do it using firewalling or various other means. Eg on AWS you get a default-deny firewall.
People never interact with these protocols directly and use a layer of indirection such as a HTTP/2 client for HTTP, and the same applies for IPv6: use DNS (or your hosts file).
And of course the need to maintain two sets of IP addresses and two sets of IP address prefixes - even and especially in DNS itself - is probably the number one factor slowing down the deployment of IPv6. That and far too many places, far too many interfaces, far too many protocols, and far too many APIs (notably Berkeley sockets) that are not transparent to which network layer protocol is being used or what the address format is. The wire format, transfer format, configuration format, and administration of DNS address records is a case in point.
Unfortunately, such changes are quite common in networking; Linux networking has many moving parts these days, there was the move to iproute2 and nftables, and the like, so one can only try to best keep up with the changes.
And for reverse DNS, PTR records? What should we use there?
More like it gets rid of band-aids
Computers do.
IPv6 doesn't seem "human usable" sometimes in large part because you aren't actually using it. People adapt. The human skills in pattern matching are robust: there are new tricks to learn, but there were always tricks to learn. (IPv4 addresses aren't "human usable" either if you sit down to truly assess absolutely how many RFCs are involved to build the patterns "everyone" has internalized that seem "easy". They are easy because they are familiar, because you use them often, because you've already adapted to them.)
I don't understand why, but until some large tech company starts pushing for end to end addressability as best practice, I have no choice but to follow the conventional wisdom to avoid throwing up red flags.
> I don't understand why […]
Excluding Microsoft, all the others find it easier to have as a checkbox to make it easier to confirm that "internal" hosts are actually (theoretically) internal since RFC 1918 isn't allowed outside.
Of course most companies' firewall and NAT rules are probably all sorts of complicated once you get to a certain size (never mind stale open-rules which were never cleaned up), so a bunch stuff is probably accidentally exposed. Also, most attacks are probably from compromised clients nowadays, so even internal hosts need to be locked down as the castle-and-moat security model isn't (as) valid.
But having "internal-only" hosts is low-hanging fruit on the security checklist.
I will resist the urge to be snarky at your expense and politely point out that exposing your LAN to public routing tables is madness, from all perspectives.
It brings no benefits and carries huge risks.
And I don't understand why people think that.
You are exposing a /64 network. That's 2^64 addresses, no one can scan your LAN if that's what you fear, nor can anyone reach your hosts if you build a stateful firewall that denies incoming connections - you know, just like NAT. But minus the packet modifications.
Are we really back to security by obscurity? Please don't tell me you are serious.
Anyways, you can't rely on ISP's handing out sufficiently large network ranges to make your security-by-obscurity scheme work.
/64 is the smallest network your ISP can hand out, of course you can rely on that. Even my mobile phone is getting a /64 from my ISP.
The inside of my service is not the internet, even though my service may be exposed on the internet; why would I want internal implementation details externalized?
Just let me strongly associate identities with my workloads and apply policy indicating which workloads should be able to send data with which other workloads.
How data gets from one workload to another should not even be my concern, just make it happen.
I don't think it's that crazy, it's just formally standardizing where we're already going.
One can use some public NAT64 services, but that's not very reliable for anything serious. https://nat64.xyz/ . AWS chargers arm and leg for NAT gateways traffic, and I don't think it's possible to configure them so that they only intercept traffic to ipv4-only hosts (please let me know if i'm wrong).
Other than this being IPv6-only in AWS works flawlessly and is cheaper (free egress gateways for private networks). As long as you don't care about IPv4 of course - that's given.
(Technically S3 does have a separate dual stack endpoint, however it doesn’t really help as I have to change application configuration anyway to deal with this change.)
That's because all mobile data connections are on IPv6
Thank endless lobbying that make legally mandated monopolies a thing in this arena. They did it with phone companies too.
It's not "just Comcast" by happenstance. It's "just Comcast" by legal regulation.
Most people don't realize there are two IPv6 internets right now, the Cogent side and the Hurricane Electric side. Both are equally sized and refuse to connect to each other, so you need to know that and either buy transit from both or buy transit from a network that buys from both. At least one major provider I know of is still running v6 over tunnels. In many places your v6 traffic is taking suboptimal routes, whereas an enterprise network may have v4 connectivity at each datacenter, v6 all gets sent to that one box under Dave's desk.
But we continue to measure v6 adoption at places like Google and Cloudflare where dedicated teams make sure packets arrive and pat ourselves on the back.
Cogent engages in peering spats on IPv4 too; this dynamic is not new with or unique to IPv6, or limited to Cogent/HE. The lesson here is to not go singlehomed under Cogent, not to reject IPv6.
Currently v6 is like connecting to the late 90s internet. It dosen't work as well as people think.
Maybe Starlink is another option. It has some drawbacks like reduced performance during heavy rain, but I've seen some positive reviews as well.
Funny that you complain about Comcast, when I was with Comcast I actually had native IPv6. With my current provider I only have IPv6 through NAT46.
[0] https://blog.devopstom.com/ipv6-only-ec2/ [1] https://nat64.net and http://v4-frontend.netiter.com
Maybe not in AWS, but there are Unique Local IPv6 addresses in fc00::/7 and NAT66 if you really love NAT!
I just recently heard that MS apparently has built everything IPv6 on Azure around NAT. This is so weird.
The headache are vendors that still refuse to properly implement IPv6, in 2023.
IPv6 just works. Amazon, Github, and Azure don't. That's not really a problem in most cases (very few people go IPv6 only because it's just not necessary with CGNAT, and even then network translation tricks can put up IPv6<->IPv4 bridges easily). In Amazon's case, they don't even need to bother setting up a real network, they could abuse an fd00::/8 network to mimic their 10.0.0.0/8 network if they wanted to.
Amazon is terrible at implementing modern standards. Just look at how long it took them to support DNSSEC on their domains, and even that didn't exactly roll out great the first time.
Only via the herculean efforts of a bunch of people having to literally reinvent the world to deal with it. Everything needs IPv6 support specifically. It's such a mess, if IPv6 has just been identical to IPv4 but with larger addresses we would be on it by now. But no they had to make it their religious crusade to eliminate NAT (and now we have NAT66 so clearly a winner) put IPSec in there which is hilarious in the era of Wireguard and eliminate DHCP which is actually insane and makes a stupid number of assumptions about hosts being able to communicate with one another and actually complicates DNS registration.
Can you imagine how trivial it would have been if you could support both v4 and v6 by just supporting v6 and having 0::v4addr be literally equivalent to ipv4? It would be more difficult to not support v6.
Remember the error "This program cannot be run in DOS mode"?
IPv4 and IPv6 are unidirectionally compatible using NAT64. The fact that you can make an IPv6 to IPv4 TLS connection using a packet-level translator without breaking the endpoints is quite remarkable, and it wouldn't have been possible if the protocols were too dissimilar.
What you need to support dual stack? Who would do that…
And how are you supposed to get the packets back when the client has an address outside that range? You still need to add support everywhere, or have NAT gateways into the areas that lack support.
Automatic mapping of IPv4 addresses exists but it requires support infrastructure just as much as any other method of allowing access to IPv4 devices.
There is a reason well resourced companies like google cloud have been slow w IPv6 - and it can be even more hair pulling in smaller settings.
Then you pick your shortest prefix length and use that for your network configuration, no? Nothing in NPT is forcing you to use a /48 or a /56, if your failover uplink only provides you with a /80 for some stupid reason you'll still be able to do translation.
DHCPv6 is supported just fine by everything but Android (for some annoying reason). Even with SLAAC, IP addresses shouldn't rotate, unless you enable Privacy Extensions on your server.
If this were the bakery just around the corner we're talking about, I would've accepted these problems as illogical to even try to overcome, but these are billion dollar companies selling network access. When networking is one of your major streams of revenue, I expect better.
Every other episode seems to be about a different new RFC that's replacing another RFC because the original ended up having a bunch of holes and edge cases. That's somewhat understandable for a new protocol but the protocol have been around for almost 30 years and is just so overly complex that it's rife with these situations.
As an example the most recent such episode was on rfc6724[0] which describes these convoluted algorithms systems are supposed to follow to determine which of their many assigned IPv6 addresses to use for a particular connection and also which of many possible destination addresses to use. Just reading the introduction makes your eyes water with how overly complex and prone to nasty failure cases (what if the source address isn't what you expect and somehow the connection routes around your firewall?) the whole situation they've created is.
I use IPv6 since 2006 and I just can't see how it can give you "a massive headache". I read a blog post about how overly complex HTTP/3 is. Better ignore it forever and never implement it then. ;-) Also, which successful RFC protocol doesn't have a see of follow-up RFCs?
* if the app specifies a source use that.
* do a route lookup, if the route has a src use that
* if you get no hint, use the first address.
That said, EVs are mostly better than ICEVs - "mostly" because performance-heavy applications like long-haul trucking and tractors still benefit heavily from fossil fuels, and in most other applications EVs are still more expensive than ICEVs.
Yes, for some who understand and have tested it, we do not like or want IPv6, it has no privacy when the device's IP is public on the Internet 24/7. No privacy extensions fix this. Test it, it's not difficult, disable IPv6 in your home router, wait a few hours, the kids will be complaining their search results are messed up, that's just the start of the indication that the advertisers now have a veil where with IPv6 they had clear fully trackable results.
I have a WireGuard server that was dual-stack. I turned off IPV4 to see what would happen, and it kept chugging along very nicely.
This is the definition of cloud bloat. The fact there are tons of systems abusing that kind of architecture probably justifies charging for IPv4.
How nice it would be if you could just create a bunch of load balancers and all that actually meant was that it was just adding config profiles to a single physical load balancer and kept them truly isolated? Right now it's really annoying because load balancer config is global state and everyone has to either be kind neighbors when adding themselves to it or manage them top-down.
You have to set a load-balancer-name annotation https://kubernetes-sigs.github.io/aws-load-balancer-controll... to tie everything together to one load balancer. There is a downside where you have to have a few other annotations be the same value across your ingresses, but once you work around that, you're good to go.
https://kubernetes.io/docs/concepts/services-networking/ingr...
If you are going to mention gateways or other methods make it work please just stop. No end-user is going to do that, or rather no appreciable amount of end users are going to do it. If your fix starts with “why don’t you just…” then please stop living in a fantasy world.
I was excited for IPv6 when it was announced, I was excited years later, I was excited a decade later, now I’m just tired of it. 2024, year of IPv6 and and the Linux desktop, ok sure. My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support.
While the idea of every device having its own public IP address was attractive to a younger me, I look at it with a bit of horror now. The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default. I’m not going to exceed the 16M+ limit of 10.0.0.0/8 so I don’t see why I would ever want to use anything but IPv4 internally for my sanity. Are STUN/TURN servers fun? Is needing some central server ideal? No but the alternative (everyone can talk to everyone directly) makes my head hurt with the implications and footguns.
At the end of the day I’ve started disabling IPv6 as a matter of course. Leaving it on is a landmine I’m laying for my future self. I’ve dealt with too many issues directly myself or for clients/customers which end with “let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done. Something drastic would have the happen to get me to change that thinking and seeing how it’s been over 2 decades and major websites I use daily still don’t support IPv6 I’m not holding my breath.
Every time someone brings up this point, I have to assume that they know nothing about IPv6 but the superficial things.
If you work with IPv6 long enough you will remember the addresses, we all remember 192.168.0.* through years of typing it repeatedly and looking at it. Not because it is easily to remember. I can already recall 2606:4700:4700::1111 or 64:ff9b::101:101 from memory.
>My ISP, literally the best available in my area and fairly cutting edge in every other aspect, has zero IPv6 support.
This is almost exclusively an Euro-American phenomenon. I am not sure why you are lashing out on IPv6 when it's the ISPs' fault. In most East or Southeast Asian countries we are looking at double-digit % of IPv6 deployment, the moment you click on the IPv6 checkbox you get IPv6 connectivity here.
>The privacy/security aspects alone are staggering and you rarely want your device to be publicly available by default.
Another one who mistakes "having a globally unique address" with "public accessibility". Boo.
>“let’s try disabling IPv6, oh it’s working now?” (on my end or theirs) that I’m done
Just say that you are lazy in fixing IPv6 problems. I have found that lots of old networking guys would say "it's defo my fault somewhere" when IPv4 fails but when it comes to IPv6 it's always IPv6's fault somehow. Protip: most of time it isn't.
ECS does support dual-stack IPv6, but most other services do not support IPv6 at all.
AWS does make it easy to fuck up with its default settings. Subnets that auto-assign EIPs for every instance attached to them should not exist, period. And neither should RDS instances or anything else be reachable from the public Internet by default.
This distorts the market in eyeball networks and hosting - the former are under little pressure to offer v6, and new entrants to the latter can only offer v6. Competition law in the EU works (I think?) on the principles of consumer benefit and market fairness. On that basis, I'm left wondering why this has never been pursued by the EU's competition authorities.
To me, the question is: what stops me today from spinning up an IPv6-only website and having 99% of the world's browsers use it? If the answer is "nothing", then AWS shouldn't be forced to offer IPv6 (or only IPv6) - IPv4 is just part of what they offer customers. If the answer is "these 7 things" then those 7 things need to be fixed[0] before we pay civil servants to try and force companies to do things that they barely understand.
[0] E.g. in the UK, it's some of the big ISPs that don't do IPv6, so there's no point forcing someone way upstream (and way more optional) in the process to do something https://www.ispreview.co.uk/index.php/2021/11/update-on-ipv6...
The European Commission did advocate for IPv6 use, but, the EU being the EU, motivated their recommendation by complaining that law enforcement had issues tracking down people behind CGNAT, and made clear that they wanted every IP address to point to a specific person for law enforcement reasons.
So, yeah, I don't think we should let the EU deal with the specifics of network infrastructure just yet.
I think it's hard to make an economic argument for IPv6. Yes, it's obviously a superior technology, but ISPs can CGNAT for cheap, consumers can still access every server, and the €40 per year a business needs to pay for an IPv4 address isn't exactly breaking the bank either.
Perhaps the EU should force the issue, but I think countries like Lithuania ,where there is practically no IPv6 available (0.58%, according to https://stats.labs.apnic.net/ipv6-zoom, but who knows how accurate that is), will protest any mandate that will force their ISPs to buy new networking equipment.
Perhaps the enterprise side of networking is better about this stuff, but I doubt it if my experiences with other enterprise products is anything to go by.
The packets routed by these devices will end up at their destination, but at very low speeds.
Not really that cheap. While CPAEX is CAPEX, OPEX is still a thing and operating CGNAT requires efforts. Also some (most?) CGNAT implementations are buggy and is not a good user experience, even for users who don't understand the concept of IP at all.
They're a pain, especially when you're visiting a website with CAPTCHAs, but the money they save on buying IP space seems to be worth the bad experience from an ISP point of view.
Even here in the Netherlands, with its relatively high wages, a fiber ISP decided to use CGNAT on their new fiber networks as a cost-cutting measure. Luckily, customers can disable CGNAT in their online control panel, but the cost cutting measure seems to be worth the annoyed customers from that company's perspective at least. Of course they also didn't roll out IPv6.
The fact that they’re owned by an investment fund also makes them probably very focused on profitability.
As a point of comparison, the other players aggressively rolling out fiber (KPN, ODF/Odido) have been nationwide ISPs since the 90s, and they aren’t doing CGNAT AFAIK (so they probably aren’t hurting for IP space).
Ziggo's DS-Lite, which also CGNATs IPv4 traffic, is annoying but at least you get a normal IPv6 subnet. This would've been a much better solution looking forward.
Dutch ISPs in general have plenty of space. Dutch ISPs has 53 million IPv4 addresses for a country of 18 million according to the first result on Google. Every person in every household can have a home connection and two servers without anyone lacking IPv4 addressing if these addresses were all pooled together.
However, there's no guarantee that things will stay this way. Like I said, Ziggo already does a form of CGNAT, and as the price of IPv4 addresses keeps rising, I expect more cheap providers to start selling off address space. KPN will stick to normal IPv4 for a while, but I don't trust super cheap companies like Odido to have the benefit of the consumer in mind, especially after trying to route all traffic through their affiliated German exchange instead of AMS-IX a while back. Odido is owned by an American fund as well (which is why they had to change their name), as is VodafoneZiggo.
The problem is customers don't like CGNAT. You can't run Animal Crossing on Nintendo Switch in network mode as a host if you don't place the Switch as a catch-all in the DMZ.
Wish I were joking here - especially due to the security risk involved in running something in all-ports-open on the Internet - but Nintendo doesn't seem to (want to) run STUN/TURN servers.
CGNAT brings tons of issues, but following Amazon's pricing model, I don't think consumers would be willing to pay $4 a month to rent an IP address. Better to sigh and shrug at the two of three games and programs that don't work than to spend $48 a year, especially with the current cost of living being on the rise.
I agree with you, but it doesn't change reality... Nintendo doesn't give a fuck and (from hearsay) people with Nintendo Switches make up a huge proportion of service calls from customers that want CGNAT disabled and pay for a legitimate IP address.
Example: we run a bunch of endpoints on ipv4, but get ipv6 IPs in our logs. How? Are there 6-to-4 translators out there at ISP edges?
Unknowns in networking are bad.
Proxies. Your logs are too trusting of X-Forwarded-For headers.
That said, until the cost of IPv4 becomes really huge, few organizations are going to suffer the effort-cost of embracing IPv6.
I would argue that the IPv6 sales story is unmemorable|unclear|weak. Also, it is arguable that most IPv4 addresses are wasted.
This means that everything else in your VPC has to be dual stack if the lambdas talk to them.
AWS Lambdas don't actually run in your VPC, there is an ENI that is exposed to your VPC that belongs to the VPC-equivalent in the Lambda space.
If you have to run dual-stack just to accomodate lambdas, then you may as well run IPv4 anyway.
I actually keep a cheap Comcast connection as a second WAN just to get IPv6 enabled on my home network. (And also because I live in the woods in New Hampshire and having two ISPs means I have fairly ok uptime)
I currently use Hugo and my blog is in Markdown in git, but the theme is pretty heavy-weight, and I like this look of the page in OP; Looking at the source, it's so minimal!
Virtually every single application at the company I work at deploys into VPCs without public IPv4 addresses - this seems like a ridiculous claim.
If your target VPC has neither PrivateLink nor public IPv4 connectivity somewhere, I'm not sure how that would work; I'd love to learn how that was built.
This is not so easy to do with a IPv6 address, AWS tends to want to keep it the same
Who is a good provider of ipv6?
For servers, there are plenty; AWS Lightsail, Hetzner and Vultr both provide IPv6 out of the box. If you don’t have an ISP which provides IPv6, you could use a server and set up a wireguard tunnel for IPv6 connectivity.
Read the burst/throttling page carefully before you choose this product.
https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amaz...
I don’t know that this is true based on google’s IPv6 adoption data: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...
It seems like developing nations have the worst IPv6 adoption, at least by a cursory look of how there’s very little green in Africa, for instance.
I think ISP’s in countries with small IPv4 blocks just use CGNAT.