Hacker News Reader

Comment by jiggawatts | Hacker News Reader

Azure is copying this wholesale.

I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale.

Literally none of those products would be required if they had just made IPv6 work properly.

Instead, they NAT IPv6, so you can't even use it to avoid the NAT forced upon you by IPv4. They also release new products -- in 2023 -- that don't support IPv6 and likely never will.

There is still an entire page[1] of listed limitations in the docs: https://learn.microsoft.com/en-us/azure/virtual-network/ip-s...

Think about how insane it would be if this was the IPX -> IPv4 transition. Imagine if this page said "IPv4 limitations: All VMs must include at least one IPX address, etc..."

Sounds nuts, right? I was migrating customers to IPv4 from IPX in 1999, and IPv6 support materialised in about the 2001-2003 timeframe. It's been decades, but it still feels like 1999 and migrating Novell NetWare where we had to have IPX+IPv4 because "not everything supports IPv4 yet".

[1] This is definitely not all of the limitations. Most of their PaaS products don't support IPv6. Hence, any IaaS+PaaS solutions must use (mostly) IPv4.

View on HN

A lot of IT folks are still fearful of IPv6. I've been on calls where people disable IPv6 as a matter of "best practice." It's sad. People will gladly learn the latest flavor of the month web framework but won't take time to gain experience with a fundamental protocol.

Why? What's the (supposed) fear?

It's simple fear of the unknown. Many folks haven't learned it, so they just turn it off... Ignore it, kick the can down the road.

It's individuals responding to the incentives before them. If something goes wrong because they're using IPv6, it's their fault. If you never upgrade anything until you're forced to, then you can never break anything by upgrading, and you're never seen as a "stuff breaker".

This is it. Having broken stuff by upgrading myself, I can say it does not make you popular. This is especially the case if what the upgrade did was tighten up some bug that it turns hid a bug in your team's code...

Why? What's the (supposed) fear?

Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink

I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing...

Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain.

That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.

The biggest issue is IPv6 is a privacy, wide open wild west, there is no privacy on IPv6. Every device's IP is literally public, on the public Internet, 24/7. All so called privacy extensions or improvements do not change the lack of privacy of IPv6 and one more thing, the address structure sucks.

FWIW this does not have to be true for companies that do not wish to expose internal nodes. I'm not even talking about the privacy extensions. I realize that people beat the drums that one must not NAT IPv6 but it can absolutely be a NAT just like IPv4. I would actually expect in most companies that they don't even add IPv6 inside their datacenters, rather they just put a block of IPv6 addresses on some load balancers at the edge and then point their edge devices L4 and L7 mapping to IPv4 nodes in their datacenter. The same could apply to VPN/WAN configurations in some cases much like how mobile networks are configured nowadays. There would need to be some of this regardless due to ACL's required between companies that don't use network VPN's for restricted end-points. e.g. PCI to PCI environments.

In the early days of IPv4 many big companies did not NAT IPv4. I was at a company that did this. Our workstations all had routable public IPv4 addresses. There are still some companies that do this. One of these was a company that had 20 managers and VP's on a call when I just wanted to give their network engineer a CIDR block and a pre-shared secret for a network VPN. I suspect they will be using public IPv4 addresses internally forever. And I doubt they will ever sell their /8's unless people comment on their Vogon poetry.

Amazon is probably one of the exceptions as they have so many geographically disperse configurations it would be harder to continue using RFC1918 address space. Not impossible, just difficult. I can think of a few other big companies that do manufacturing that are spread out around the world that would probably run into walls with RFC1918 at some point. I've seen some of them take over public address space for internal routing which then breaks access to some things on the internet thus requiring double/triple NATs. 1/8 assorted, 25/8 MoD, 26/8 DISA are a few I've seen.

> In the early days of IPv4 many big companies did not NAT IPv4. I was at a company that did this. Our workstations all had routable public IPv4 addresses.

A lot of big universities did this and even still do this to a large degree. They got huge IPv4 allocations early and there was no scarcity.

All of the early Internet companies I worked at were like that, up until roughly 2000: public IPs direct on the desktop. My 90's home network was also like that. I had a /24 block from the old class C "swamp" space. I still have it, actually. It's legacy space, no ARIN fees.

In my current company (related to academic institution that introduced internet to my country) every office workstation has FOUR public (but firewalled of course) IPv4 addresses. And every user has an unique VPN IPv4 address on top of that.

I remember the days of non-NAT IPv4, though I'd forgotten until you mentioned it. I'd be OK with NAT IPv6, though the addresses are still ugly and difficult to reason about.

CIDR IP addresses are based on binary number prefixes. If you think it's easier to reason about them in decimal than hex, then you probably don't really understand binary.

IP-based "privacy" is an illusion. With IPv4, your public IP (NAT router IP address) may not change for months, years, and possibly not until your change your router/MAC address. With IPv6 privacy extensions, your address changes regularly. This seems like an improvement.

The IPv4 at my router yes. That's where tracking ends. IPv6 privacy is an illusion, try the test I described, remove IPv6 from your router at home, wait a few hours or few days, the family will complain search results are odd or messed up and that's only the beginning of it.

I don't know how companies are doing it but they are able to track your IPv6 changed daily or not.

They probably just track the IPv6 /64. With prefix delegation, the /64 would rarely change, unless your provider delegated a new block. This is similar to your IPv4 router changing its address w/DHCP: it happens, but is relatively rare.

Also just cookies and browser fingerprinting.

> With IPv6 privacy extensions, your address changes regularly. This seems like an improvement.

Eh… If I was a company that wanted to use IP addresses to fingerprint users, IPv4 vs IPv6+privacy extensions both seem identical to me. Multiple requests from the same IPv4 address mean “someone, perhaps more than one person, from the same household/wifi”. Whereas multiple IPv6+privacy requests from the same /64 prefix means the same thing.

ie. You just consider the first 64 bits of the IP and can assume the same amount of information you already would assume from the IPv4 address. Just ignore the trailing 64 bits because it’s expected that they’ll be randomized/shuffled even from the same client.

This is wrong. Every IPv6 interface has a link-local address (which is not routable outside the LAN) as well as a global address. Global addresses are just addresses that come from a block allocated by the IANA. It has no bearing on whether the interface is reachable from the public internet. Just like with IPv4 networks, a stateful firewall will prevent unsolicited inbound connections.

Why is it wrong? You don't seem to be engaging the parent's point at all, which is that IPv6 addresses are a more specific identifier than current IPv4 addresses. The existence of link-local addressing has no bearing on that argument, because those addresses are non-routable by definition. Nor does stateful firewalling prevent your unique device address from being broadcast over the Internet, you'd need 6-to-6 NAT to achieve that.

Every Ipv4 device has a loopback (well most do.) thats not what they are on about.

Having NAT and a firewall gives you a better illusion of privacy. Sure you can track devices from the outside world, but its pretty hard.

If you have v6 configured in a certain way, then your IP address is basically a UUID for your machine. Plus you can't really just stop ICMP anymore so you can trivially ping it (caveats apply)

It’s a big scary number and the letters just break understanding.

There are some annoying operational issues around it as well, common one: DNS hostnames for devices that only do SLAAC

You wouldn't want to do SLAAC and hostnames.

If your network card breaks you switch it out, and you make sure your IPv4 settings apply to the new card.

If you fully rely on IPv6 you'll get a new address.

And if your devices self-update DNS then you have to make sure they pick the right address, as there can be many due to privacy extensions.

Lastly, combining privacy extensions plus hosting stuff is hard, as you don't know on which address a certain port is bound.

It's definitely "best practice" to turn off features you don't need. Who knows, maybe in five years someone will find a bad bug in the implementation of the IPv6 stack, then you'll be glad you decreased your attack surface.

Not saying this is what you should do, just a common rationalization.

It's also "best practice" to learn a new, foundational technology (like IPv6) sooner than later, perhaps less than 20 years after it was first available.

Right and what is better way to learn than deploying random features to prod!

Did I say they had to deploy it to production? If they deployed it in dev and test environments even a few years ago, they might have some experience deploying it in prod today.

It's only been ratified since 1998 and available as far back as Windows NT4

Pretty sure that's enough time to test it in development...

I know IPv6, just don't feel the need to use it. I prefer to have one firewall and public network interface to worry about. I can't disable IPv4 yet, so the logical solution is to disable IPv6.

I have a reason: we do per IP rate limiting. It's easy enough for IPv4 when the number of IPs is necessarily not too big to fit in a small redis for example, but for IPv6 everyone have at least a /64.

I'm curious how people do it btw, if you have tips to share, I'm all hear. Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.

When a bunch of households or cell phones are on the same IPv4 do you have any measures to compensate?

> Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.

Yes you'd limit by /64 or slightly larger.

The live set of IPs shouldn't be very big.

We put limits high enough that it's far enough for any expected usage, including a bunch of users on a single IP. If we see rate limiting happening in practice and it doesn't seem to be an attack, we revisit.

Well it sounds like you'd do fine tracking the IPv6 blocks that are currently very active, without needing any significant amount of resources.

If you go the extra mile and simultaneously track /64, /56, and /48 with moderately increasing thresholds, you'll probably end up causing less collateral damage when you block someone than with IPv4.

You treat ipv6 /64 just like /32 in ipv4

Answering myself: I found this interesting article https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/

Given that the only routable IPv6 address space is in the 2002::/16 range (is 2003:: in use yet?), and the standing recommendation for ISP CPE endpoints is to allocate a /48 per customer (a customer can't do any local subnetting if only allocated a /64), the effective address space for rate-limiting is the exact same size as the current IPv4 address space: you only need to track bits 16-47.

It's possible that cloud providers assign smaller ranges to their customers, so you may need to allocate more bits for granularity in that case; on the other hand, one might naively assume that cloud providers are more responsive to abuse reports than ISP's.

While the recommendation is to hand out a /48 to each individual customer it's definitely not the standard.

Cox & Spectrum only hand out /56. I'd hate to be banned because my neighbor did something bad and we happen to be in the same /48.

Because there's a lot of shit that still doesn't work well with IPv4. Logging is one good example - a lot of software that uses its database for event logs has the database column for remote_ip defined as VARCHAR(15), you can guess the rest of what happens when deploying that with IPv6 enabled.

These things take time. I mean, it's only been 25+ years since the first IPv6 RFC was released...

Mainly that everything is publicly routable.

People see a 10.x and instantly know it can't be reached from the public internet. IPv6 is much harder. For internal-only stuff there is the fd00::/8 block, which AWS actually does use, but there is no equivalent range for outgoing-only connections.

In the case of my corporate VPN, Teams doesn’t work with IPv6 enabled. Something on either the MS side, or the SSO one.

I was troubleshooting some bad routing in a service. Some endpoints worked fine most of the time but would occasionally lose packets for a bit.

Turns out it's a bug in the kube stack when containers have both ipv4 and ipv6 addresses.

This was last year.

I suspect the ipv6 cases aren't being tested very well across the software stack.

Some current Apple products still have problems with IPv6, and the standard practice is to disable it entirely.

I continue to do that on all my personal equipment.

I probably fall into this bucket but I’ve seen disabling it fix very odd issue entirely too many times to write this off as just “sad”.

Learning a new framework doesn’t (often) break things for the end user in a way I can’t diagnose/reproduce. IPv6? Totally different ball of wax.

Also my ISP, a fiber gigabit provider and the best available in my region, doesn’t support IPv6. Until they, and others like them, get on board fully I don’t see customers having a snowball’s chance in hell of working cleaning with IPv6. I can only imagine how long it will take the shitty ISPs to fully support it.

Comcast at some point stopped letting you administrate your own router. You can log in to it, but port forwarding is no longer available through the administration interface.

If you want port forwarding, they recommend that you do... something. It's not clear what; what you can find on the internet is mostly just people complaining that they insisted to customer support that they needed port forwarding, customer support said they'd do that, and port forwarding still doesn't work.

But, intriguingly, it turns out that a Comcast router will also assign every device on your network at least one public ipv6 address. They also firewall all incoming ipv6 traffic, but unlike the situation with port forwarding, you can disable that firewall on the router admin page. (You can't put up your own firewall.)

> port forwarding is no longer available through the administration interface.

Sounds like the CGNAT experience, possibly blanket applied even if you have a globally routable public IPv4 in order to have consistent behaviour and reduce network management / support case complexity.

One cannot unplug the Comcast issued router, power cycle the modem, and plug in a customer owned router?

All I see are combo units these days.

You have to bring your own modem. Then you bring your own combo unit or bring your own modem that you can plug a router into.

All the modems I see support bridge mode. When enabled the Comcast device doesn’t do any of the routing at all. Your own device gets to do that instead.

Yeah, even at my father's place (I've had different providers than Comcast for personal reasons, but FWIW same story) I've never had a problem just plugging in a router of my choice and using that instead. Makes it a lot easier to handle quirky setups between modem swaps anyway.

I’m also a CrapCast subscriber.

I bought my own (plain) modem and manage WiFi/routing myself.

I get the direct Internet-facing IPv4 and IPv6 addresses.

No monthly equipment fee!

It's almost as if those big cloud providers want to prevent a transition to IPv6 as relying on IPv4 will solidify their oligopoly.

Any new competitor would have to cough up a huge upfront cost for IPv4 addresses, if they could even get them at that scale.

Wait a second. A private link means that a service endpoint is public so a part of your traffic goes through the Internet, which is supposed to be insecure (even if you have encryption in transit?), so you don't want to do that and they will happily route your traffic internally so it is not exposed to the bad Internet - for a fee. All these VPC Endpoints etc. cost money and you are charged by the our. I don't think IPv6 would change anything here, they would find a way to charge customers for "security" anyway.

That's what I thought at first, but that's not quite right.

Service Endpoint: Allows a PaaS service (that itself uses public addresses) have firewall rules for overlapping private vnet addresses. E.g.: You can have have two VMs both on 10.0.0.123 addresses (in separate VNets) using individual "Allow" rules to the target service. Essentially Azure tags the traffic at the VXLAN level with the source network ID on top of the IP address, making it a "fat IP address" that is unique within Azure and can be used in firewall rules.

Private Endpoint: Makes a PaaS service appear on a private network address range instead of the default public range. This allows your on-prem firewalls to isolate your specific PaaS instance from other customers -- otherwise the traffic gets "blended in" with everyone else in the same public service tag ranges. This also allows you to use your ExpressRoute fibre links to route traffic from on-prem to the public service.

In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks.

Fundamentally, all this exists just to enable the ability to firewall things. With overlapping IPv4 addresses and small shared blocks of IPv4 addresses with NAT behind them, it would be impossible otherwise.

With IPv6, using firewalls would be much simpler because overlapping addresses aren't needed any more. Similarly, PaaS services could trivially allocate IPv6 addresses per customer instance, so that customers could apply selective firewall rules.

What a shit show. Seriously I can never rant enough about how awful Azure networking and their bullshit concepts is. Like they don’t know how to do networking, so they’re gonna introduce a bunch of shit and pretend that nonesense makes perfect sense because of their own ineptitude.

Most of the complexity is a side-effect of having to do IPv4 far past the scale where the ~17 million private addresses might be sufficient.

Some of the complexity seems to be the security theater of the lowest common denominator of customer demands. Companies invest too much money into incredibly expensive Palo Alto firewalls then demand Azure route through those too so that their cloud operations are as theatrically "secure" as their main network because look at all those amazing sunk costs invested in it.

> In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone.

That's the whole selling point of private links. How could you possibly missed that? That's exactly why companies onboard onto the service. They say exactly this exactly on the marketing brochure. That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild.

What kind of confusion reigns in your mind to come to the conclusion this was some obscure conspiratorial gotcha?

It boggles the mind how you felt the need to come up with absurd conspiracies involving IPv4 to arrive at a claim that the marketing pamphlets show front and center as their whole reason of existence: avoid traffic to go through the internet, and instead pay extra to go through private pipes they own.

It's as if the name actually means something!

> That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild.

I believe their point is that it doesn't flow through "the wild" in any case, it is probably routing within the Azure AS(aka Microsoft controlled networks). However, as you say, people line up to pay for it, likely for reasons having to do with their architecture/security model/compliance requirements.

> In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks.

My understanding is that if you don't have a private endpoint, your traffic to an Azure cloud service won't be routed out to the "big bad internet" per-say, but it will be routed within the Azure AS as mere IP traffic.

If you have a private endpoint to an Azure service in your virtual network, that means Azure has provisioned you a virtual NIC with some private IP address, and presumably alters DNS resolution within your network for that Azure service to resolve to the IP address of the NIC. The NIC provides (presumably encrypted) link layer transport out to the Azure service.

Compliance for some customers may dictate that there aren't any routes out to the public IP address space from within a network. If you still need access to cloud services, private endpoints are a necessity.

All that to say, I think Private Endpoints provide more than just a means of firewalling traffic/changing the IP address associated with a service; the actual transport from client->cloud service is fundamentally different.

You’re exactly right that the main point of private endpoints is to allow customers who aren’t allowed to open firewalls to public internet to still connect to their public services like Azure Storage, Key Vault, or SQL

> I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale.

I'm not sure what you've been reading, but the concept of a private link has absolutely nothing to do with IPv4 vs IPv6. In fact, practically all your remarks don't involve the issue at all.

The most charitable interpretation of your comment is that you're making the mistake of conflating any application involving a virtual network as something caused or involing IPv4.

I replied to someone else's similar misconception of what problem Private Link solves here: https://news.ycombinator.com/edit?id=37609614

Hint: it's a complex workaround for insufficient IPv4 addresses.

I think your comment shows a high dose of ignorance and complete lack of research on the topic. The whole point of private link is to not route traffic over the internet, and instead flow traffic between private networks through private pipes.

One of the primary usecases and design requirements for this service is regulatory compliance. They say right on the tin that the service is designed to send traffic over private networks, including AWS's own global network. The whole point of private link is to ship data through the pipes you own, instead of routing it through the wild. I don't know how you could have missed that.

More importantly, you really need to want to use private link connections. This is a value-added service. You need to want to go out of your way to avoid your traffic to go through the internet to onboard both ends of your services to private link.

Not only are your conspiratorial hypothesis completely out of base, even your baseless assumptions have absolutely no relation with what version of the IP protocol is in place.

I'm the first to join in on any good old fashioned AWS/big cloud provider bashing, but these should be grounded on reality.

If you think that either Azure or AWS loop terabits of customer traffic between two of their own services "out to the Internet" and back just because the IPv4 octets don't start with a "10", then you're the one who's missing the big picture.

I am at a total loss to understand the link between this post and the prior one - can you explain?

Yep. Pick your poison. You're either nickle-and-dimed for a "NAT gateway" or overpaying for "VPC endpoints". Often, it's both. I preferred the EC2-classic days, honestly.

Private link is a clever way to implement real network segmentation

That is, when you have a customer in some network and a provider in another network, you had to implement full connectivity between the customer and the provider

With private link, you can remove all that connectivity, and instead expose the provider' service to the customer The service, nothing more, so just one endpoint

This is really good from a security point of view, but also for managing your stuff (especially if there are multiple teams in the compagny): because you now have a resource, you can easily list the services you expose to other people, and whom are your customers

The IPv6 version of this would be ...

1. get list of customer netblocks

2. setup "internal" service(s) for customers

3. setup firewall rules to allow customer <-> service allow list

4. setup DNS records

5. tell customers DNS and API targets

There is no "ipv6 version of this", private links have no business with layer 3 stuff

If you're not using token ring, it's not a real network.

I read that as "Tolkien ring" - which would also be appropriate, since "...and in the darkness bind them" is a good summary of cloud vendor lock-in.

"Nine /8's were gifted to the race of cloud providers, who above all else, desire power"

Reply on news.ycombinator.com