Thinking about our passive exposure to IPv6 issues
utcc.utoronto.ca
utcc.utoronto.ca
It's also incredible that, despite putting all their users behind CGNAT, they don't provide IPv6.
I mean technically they are no _worse_ than their major competition here (Virgin Media which has failed to move on IPv6 for 20 years, why start now). But it's baffling that a _new_ network would be built without IPv6 provision!
More specifically, T-Mobile US only assigns IPv6 addresses to devices, and to reach the IPv4 world you have to go through a translation mechanism.
Presentations from T-Mobile at NANOG and Rocky Mountain IPv6 Taskforce:
FWIW Verizon FWA does NOT use CGNAT for IPv4.
I’ve seen Comcast business also use CGNAT.
IPv6 Transition/Co-existence Security Considerations, https://datatracker.ietf.org/doc/rfc4942/
Operational Security Considerations for IPv6 Networks, https://datatracker.ietf.org/doc/rfc9099/
2 Gb (but it only performs to 1.1 Gb) is $100 USD (£81).
Still a great deal, except for needing to pay £5 extra pcm for a routable static IPv4 address.
Yes, ideally, an infrastructure provider should not dictate what kind of IP packets the commercial ISP can use, and there are dozens of technologies that would allow them to provide transports that don't care about the routing layer, but in practice, these providers (that we never really hear about as end-users) often make very bizarre choices, for example setting up a complex DHCP interception schemes that feed both the OLT to "open the port" and a BGP route server to announce connected clients at the collection point, of course not supporting IPv6.
In one of the ISPs I work for, we literally have /48s assigned to all of our clients, but the infrastructure provider's doesn't route them to us : IPv6 traffic is rejected at both ends of the transport. The ticket has been open for years (IPv6 was supposed to be part of the network they were commissioned to build), and insiders told me that they had difficulties adapting their transport system for IPv6.
We may end up having to tunnel IPv6 through IPv4 to our clients, but we try to fight against that so that the infrastructure provider doesn't declare the problem "fixed" and decide that the proper way for ISPs to provide IPv6 is to tunnel it through IPv4 so that they don't have to change anything.
I think the biggest pain point with IPv6 adoption is probably the large number of customers with a little netgear router that doesn’t know how to pass IPv6 through it. I also think local administrators that still purchase stuff that doesn’t support IPv6 are just shooting themselves in the foot. After all, the writing has been on the wall for over two decades.
If that's everything IPv6 has going for it it's no wonder its adoption is still so low.
Edit: don't know why I'm being downvoted, I guess the network gurus here don't research
But then you need NAT piercing everywhere for even "basic use". How is not being able to connect to things "usable"?
By that same metric, you also get a bare minimum, naive count of of 144 bits of IPv6 address with 2 layers of link-local address + all ports.
144 is much larger that 96.
Even the naive 64 bits "just use one layer of link-local" is still larger that the entire current 32-bits of IPv4.
And, of course OP is wrong about 56 anyway, but so it goes
Also, that sentence continues in the article with: "[…] but the pragmatic question is how much of this space can be exploited in a cost-effective manner such that the marginal cost of exploitation is lower than the cost of an IPv6 deployment."
IPv6 has numerous things going for it. But it also has numerous downsides. I suspect that's why the adoption is so slow.
I am in the ARIN region but I did not want to pay the ARIN registration fees for an ASN. It was $550 + $150/year, so I went with a RIPE LIR.
Your ISP should give your router a /56 prefix via PD, which you can divvy up into 256 networks internally. Alternatively, if you don’t need globally-routable addresses, just pick a ULA prefix and have your router advertise it internally.
Outside of obscure hypotheticals you make do or switch providers.
So it's reasonable to wonder how people deal with that problem when needing multiple subnets. Many locations don't have a realistic choice of affordable providers, or none of them provide IPv6 with a shorter prefix anyway.
I'd love to have to make do with a /64.
Given that the IPv6 standard was written by enterprise network people with no thought how it would get deployed to the masses, lots of software is still a PITA to work with if you got an ISP like mine.
Thankfully, contrary to GP's claim, you can just use smaller subnets to mitigate on links where this might be a problem.
That IPv6 has been around so long and is still not ubiquitous means that writing is faded and partially obscured by graffiti.
> the underlying need hasn’t gone away
That's 100% true. I think almost everyone sees the problem that IPv6 is addressing.
Exactly!
For home IoT, Matter uses IPv6, and may start picking up this year. (I worked on some IPv6-related things for it for my then-employer.)
The current state of support and adoption is frankly abysmal. The attack surface is non-negligable. And there is no end in sight.
The idea of deploying a new version of the internet protocol (which lacks backwards compatibility for reasons) in parallel to the existing version should never have left the drafting table.
Outside of the world of smartphones, though, adoption is awful.
Precisely so.
> all browsers use Happy Eyeballs and it tends to pick IPv6 because it’s often faster.
I know. This sort of thing causes me trouble every so often. I wish I could make browsers not do that.
The most important thing is probably just pushing equipment vendors to set better defaults since that’s what a disturbing number of places will be using.
I hope I can ignore it up to the point where I can make a complete switch, use only IPv6 and stop supporting IPv4 everywhere.
I wouldn't be surprised if that point never comes and I can be a happy "IPv4 only infrastructure" person forever.
Or IPv7 comes out before running an IPv4-only infrastructure becomes a problem.
I think the mistake of IPv6 was to not be a superset of IPv4.
Every firewall I've come across has a default deny rule for incoming IPv6 traffic, giving the firewall the same properties as any IPv4 network. Host firewalls are the same; anything ranging from Windows Firewall to UFW and firewalld have presets to block all traffic except for the applications you've whitelisted. Once you get to huge enterprise routers managing routable IPv4 addresses and IPv6 addresses the situation may become different, but it's still not that much overhead.
The biggest problem with securing IPv6 seems to be ignoring it assuming that makes it disappear. If you configure your firewall to drop all IPv4 traffic not on a whitelist but somehow manage to forget to add the same rule for IPv6, you should re-evaluate your networking knowledge and maybe get up to speed with how the internet has changed since 2015.
Its also all kinds of code that interacts with the internet in all kinds of ways. Extending all that code to two kinds of IPs, writing tests, setting up two types of IPs in development, staging and production, monitoring real life implications ... that would be a huge cost with no benefit at all.
In my experience, IPv6 Just Works (tm) with modern software. There are some mid 00's frameworks for blacklisting abusive hosts that can't parse IPv6 addresses, or don't understand the /64 subnet you need to treat as a single IP address, but that's all I've ever run into. If anything, that gave me an excuse to finally get rid of an old Perl network filter running on my server.
I'm not sure how many tests the average piece of software needs that deals with the type of address family connecting. I suppose it matters if you want to test your rate limiting middleware or your logging library? That should only matter for the vendored code of course because modern libraries all have those tests themselves already. It's not like you need to run and write every test twice, only one or two very specific subcomponents if any.
If you're writing firewalls or kernels or router firmware then yeah you'll have your hands full with this stuff, but that's far from the standard developer experience. In those cases, IPv6 is a reality as much as TCP and UDP are.
AF_ANY is a thing, and it’s a best practice.
gethostaddr (iirc) was the old interface, but nowadays getaddrinfo is almost the default and supports AF_ANY.
So now you’re getting a record with multiple ip addresses, some of which are ipv6, but ipv6 is blocked… there you go with random connection delays and possibly timeouts.
Ipv6 exists and it’s getting more and more adoption, no matter if some people keep their head under the sand…
RFC6555 “Happy Eyeballs” discusses this.
In some environments that is maddening and I don't blame people for just deciding not to either at all or only translating at WAN.
Internet facing IPv6 infrastructure is usually much cheaper that their equivalent IPv4 enabled peers.
So supporting IPv4 can be the huge cost with no benefit at all, if all your clients/peers can use IPv6
Including the case where you have something else in the middle already - for example, if you're fronting a website through cloudflare, then you can only have IPv6 on your server and still support dual-stack for clients:)
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1
In fairness, neither of those things would be unreasonable stances, given those conditions.
The only issue I have is the same issue found in the article, but in reverse: now that I have a securely configured ipv6 network, how can I ensure that my hosts are fully prevented from communicating over a rogue ipv4 net?
And the example revolves exactly around filtering and firewalls -- on IPv4 they drop outgoing packets whose source address isn't a part of their subnet -- they prevent source spoofing of other addresses.
But on IPv6 they do not do that. I noticed this when their router somehow lost the prefix route to me -- I was informed that their DHCP server adds router dynamically to routers -- and I could still send packets. But I couldn't just send packets from my prefix, but also from any other address, including stuff like 2000::.
Perhaps the good network engineer that got IPv4 working right left before they could implement IPv6? Or maybe they don't provide enough of a training budget for their engineers to stay up to date on networking technology?
Either way, if their core business is doing networking, I'd expect them to be better at this!
for the curious, that's (in most cases) DHCPv6-PD relay agent operation, https://www.rfc-editor.org/rfc/rfc8987.html#name-general "Delegating relay":
Delegating relay:
A delegating relay acts as an intermediate device, forwarding
DHCPv6 messages containing IA_PD and IAPREFIX options between the
client and server. The delegating relay does not implement a
DHCPv6 server function. The delegating relay is also responsible
for routing traffic for the delegated prefixes.
That RFC also goes into detail on problems with these setups. What you experienced was probably the loss of state described in section 3.2.
0: https://packetpushers.net/podcast/ipv6-buzz-123-why-you-need...
Let me guess, you live in and/or operate setups in the US?
Yeah, the US has disproportionately many IPv4 addresses.
Meanwhile if one of our services doesn't need IPv4, it doesn't get IPv4. And if it does need IPv4, it's increasingly common to be IPv6 behind an IPv4 reverse proxy.
And as a result, due to the extra reverse proxy, you'll increasingly just get worse performance on IPv4 than native IPv6.
When you do it near the service being provided, and only for your own services, it's called a Reverse Proxy.
You are right that these two things are similar, but they aren't identical; CGNAT attempting to handle you trying to talk to who knows what on the Internet (e.g. game servers, VoIP) is a much harder problem to solve than a Reverse Proxy handling a known set of protocols you want to expose.
And, yes, an unloaded CGNAT or Reverse Proxy is not noticable in terms of performance. However, both of them have load limits where you need to scale them up, and particularly CGNAT frequently degrades (due to larger tracking tables) before completely falling over.
..if only. :(
* https://bgp.he.net/AS239#_prefixes6
It's just that particular department may not have bothered.
[1]: https://utcc.utoronto.ca/~cks/space/blog/tech/UniversityMone...
There's a lot of requirements that apply to institutions as a whole to clean up their acts and mature a lot of their security processes to even be eligible for grants. A lot of the wild west days are coming to an end, and I've seen a push in several institutions to wrangle unit-level IT groups into one massive IT organization for their entire campuses. Not only is there a lot of security vulnerability there, but it's also a huge cost center that can be squeezed to flatten the curve of tuition, but not have to reduce the salary of three dozen associate deans
I don't think we could possibly be anywhere near the top of the things harming the transition when we regularly encounter people actively encouraging others not to use IPv6 for either security or convenience reasons!
A miserably common human behavior pattern.
Anyone have a copy of the DSM-5 handy, to tell us the technical name for it?
2) When was the last time you fixed networking issues by turning off IPv6?
For me, the answer to 1) remains "never", the answer to 2) is currently "Late winter"
In 2017. The problem was that file copying via scp was very slow between the servers (one in Germany, one in China). I set up a he.net tunnel in China, got an IPv6 address, and it went much faster via IPv6.
About a year ago, when I noticed wikipedia still doesn't have IPv6 reachable DNS resolvers.
4) When was the last time you fixed networking issues by turning off IPv4?
A few weeks ago, IPv4 was just slow for some reason I couldn't quite determine.
2. Never, because altering my network in a way that I'm no longer able to reach my own servers isn't really a "solution".
I've used IPv6 to help users behind CGNAT; enabling IPv6 for client -> server often means those clients no longer go through carrier stateful firewalls and their sometimes tragically low idle timeouts (I've seen timeouts as low as 10 seconds).
I've also used IPv6 to help in a proxy config when I couldn't get the upstream servers configured for more listening ports; it's super easy to get extra v6 source addresses for the proxy servers and harder to get v4 addresses. With high volumes and low session times, you run into port reuse issues at around 10k-20k sessions per {source ip, dest ip, dest port} tuple; theoretically you can get to 64k, but TIME_WAIT takes time to clear and you don't want to be too aggressive.
Probably less going forward, but v6 and v4 often have different routed paths, and sometimes you want to get off of one path and onto another. If you're using a tunneled v6 connection, you almost certainly have a different path for most destinations.
> 2) When was the last time you fixed networking issues by turning off IPv6?
My previous DSL modem would crash when receiving fragmented IPv6 packets. My current one doesn't crash, but adds a lot of internal latency. Temporarily fixed by disabling IPv6; permanently fixed by using the modem in bridge mode and doing PPPoE on my own personal equipment (at great cost to my own sanity, PPPoE is gross, and I built a multi-node redundancy setup, so PPPoE sessions can be transferred between my redundant routers, bleh)
2) Never
2) 20 years ago? Late 00's
This blog implies that IPv6 might simply happen and that is bad. If it does "happen" for a particular site then it will "happen" in the same way as IPv4 does already. Routers generally fail safe and so do PCs. In general a router will by default block inbound and allow outbound, regardless of IP flavour.
Yes it is a bit of a pain to maintain two lots of firewall rules but when you think about it you already have to worry about protocols too so IPv6 simply adds one more dimension to the already considerable combinations. A firewall rule generally worries about inbound/outbound IP/port/protocol. All IPv6 adds is IP version.
You'll also be surprised how quickly you can become accustomed to the stupidly long addresses. You may have forgotten how long it took you to get to grips with dotted quads when you first encountered IP(v4).
IPv6 does just work - it does have issues with multi ISP and that is a major issue that was never considered at the initial design stage. There are workarounds but they are shit unless you do full on BGP which is hardly a home user solution. It is what it is.
I imagine this may be a problem for other providers.
Thankfully no CGNAT, so I have a couple VLANS that use he.net in the meantime.
Sure, we'd save a little bit of time from the extra features, but I don't see how anyone's deployment approach would be different.
(An octet is a byte btw, so it'd be an increase from 4 to 8.)
But if you've changed things so radically that it discourages moving to the replacement protocol, isn't that a problem?
> The changes made with IPv6 actually simplified deployment compared to IPv4.
Perhaps, but they also make transitioning from IPv4 to IPv6 orders of magnitude more difficult.
And that's not even beginning to talk about supporting services such as RA, etc.
IPv6 has been available much longer than IPv4 was when the Internet went mainstream in the mid 90's. Yes, it's work to deploy. You might need to even upgrade to a router built in the past 10 years. Organizations have had decades to learn about it and deploy it. People just don't like change... especially corporate IT departments.
Well, all change causes pain. People don't like change when there isn't an obvious benefit to offset the pain it brings. When they do see the benefit, though, they tend to embrace change. I think the mistake that was made with IPv6 was a social one: it's a huge amount of change (pain) for a benefit that is invisible or irrelevant to a whole lot of people.
IT departments, like most departments, dislike expense more than they dislike change.
In reading the discussions leading up to IPv6, it was clear that the actual experts involved saw a few ways to maintain such compatibility. It was decided not to do that, though, because tradeoffs are (as always) involved. There are features they wanted to include that wouldn't have been possible while maintaining compatibility, and it was thought that a breaking change was worthwhile now as long as the IPv6 protocol was good enough that no breaking change would be required in the future.
For applications a subset of the IPv6 address space is set aside for IPv4 connections. So applications only need to support IPv6 and can communicate with IPv4 destinations using ::ffff:<ipv4-address>, so for example ::ffff:8.8.8.8 is a valid ipv6 address. But that still requires the host to have a IPv4 address for this to work (it's only so applications can support both while only requiring support for one address family).
For IPv6 hosts connecting to IPv4 hosts there are different methods that translate IPv6 to IPv4 (NAT64, SIIT, etc.), but most (if not all) just embed the whole IPv4 address space inside IPv6 (and again you can have something like 2001:db8::8.8.8.8) and then translate based on extracting the IPv4 address from the IPv6 address. And in combination with DNS64 you can have ipv6-only hosts that can communicate just fine with IPv4-only hosts (all ipv4 traffic gets routed via NAT64).
The problem is however the inverse. IPv4 is not forward-compatible. A IPv4-only host can only ever form outgoing connections to other IPv4 hosts, since there is no way to put more than 32-bits of information into the 32-bit destination field of the IPv4-packet. You can of course expose certain ipv6-only hosts to the ipv4-internet using for example SIIT, but that requires a 1:1 mapping between the IPv4 and IPv6 host. If you don't have that explicit mapping, how would a gateway know which IPv6 address you meant?
Every time IPv6 negativists come to the thread they make it look like you need to be a rocket scientist to configure two hosts for IPv6.
> You can't incrementally shift toward IPv6
Yes, you can, you don't need to throw out your lovely IPv4 networks and addresses. Yes, it's dual-stack.
> you have to run [..] tunnel, which adds at least another layer of complexity to everything
No more complexity than running a PPPoE/PPTP tunnel to get IPv4 connectivity.
> And that's not even beginning to talk about supporting services such as RA, etc.
Oh God! You need to read another 20 pages on top of the previous 20 pages you read to understand IPv6! So, so hard!
BTW you obviously forgot what there was a time when you didn't know nothing about IPv4, including what is the default router, what is ARP, what is CIDR, netmask, routing etc, etc. Yet somehow you know that now. Same applies to IPv6. You had 20 years to learn it, yet in 2023 you still complaining what to use a technology you need to learn.
I'm describing various reasons why there is such slow adoption of IPv6. Ridiculing and being mad at people for avoiding it isn't likely to make them feel any differently. Understanding the resistance might be helpful in terms of getting people over that hump.
You are not a devil's advocate nor a public defender, yet you defend those people, using the same nonsense reasons they use to defend their position.
> I'm describing various reasons why there is such slow adoption of IPv6.
For the most part those reasons boils down to 'I have my way and don't do a shit to learn something new'.
> Ridiculing
Yes, they deserve it.
> and being mad at people
A grown-ass person who can't find the reason and time to learn something new? I'm not mad at them, but I don't like when people defend them.
> Understanding the resistance might be helpful in terms of getting people over that hump.
See 'I have my way' part above.
v6 isn't radical in the slightest. It works very similarly to v4.
If I had a time machine and had to take a relatively naive crack at the problem, I'd make IPv6 really really simple: it'd just be IP-in-IP.
On the internet you're routed with an outer address, and the local router forwards it using the inner address. Speaking v6 over v4 is free because v4 routers ignore the inner packet. Speaking v4 over v6 just doesn't include the inner packet. Other local hosts are just contacted through v4.
Maybe.
> If I could go back in time to 1995 I would kick some people at IETF meetings.
Sure.
But how is any of that relevant enough to bring up today? Now, IPv6 is what we have, and the standards are what they are, flaws and all.
The only reason I can think of is psychological: People don’t want to learn new things, so they find reasons to dislike the new thing to be able to pretend they don’t need to learn it.
Yet IPv6 is very complex with tons of moving parts compared to IPv4, and a lot of it works very differently so I can't rely on my existing IPv4 knowledge. In fact it frequently a hinderance.
Which IPv6 does pretty well even if you aren't a network engineer. More than 40% of all requests to google are from IPv6 and the vast majority of those connections are from people that never heard of IPv6.
I'm not just talking about connected to Google, I mean to each other. So ULA, DNS, DHCPv6 vs SLAAC etc is in play.
Outside of smartphones (in the US, apparently), moving from IPv4 to IPv6 is genuinely painful and often requires deeper knowledge of networking to accomplish.
Yes, if you're using a smartphone or your needs are otherwise trivial (for instance, you don't have a real network at home but just have one or two devices connected to your modem), then none of this is a big deal. But outside of that, things can be very different.
Right. Those are the "trivial" cases I mentioned.
That was the only point I was making.
That's not an endorsement on my part at all, just a recognition of what has happened.
I confess that I'm doing the same thing in my home network -- I won't move to IPv6 until I have no other choice, because that move will take a ton of time, sweat, and tears.
Just to stave off misguided attacks on me: I am not saying IPv6 sucks and we shouldn't move to it. Not at all. But moving to it is a very expensive proposition.
Now I'm interested on what kind of "home network" you have that makes v6 that hard.
I'll start by acknowledging that it's larger and more complex than a whole lot of networks. I have three subnets (not counting a second WiFi AP for guest use that just routes directly to the internet and bypasses my network entirely).
I have a subnet for my 4 internet-facing servers, a subnet for devices that for one reason or another aren't capable of working through my VPN, and a subnet for my general use. This is entirely encrypted using a VPN server I also run.
I have a total of around 70 different computers and devices on the network.
When I played with moving to IPv6, two things became immediately obvious to me. The first is that it will break most of what I have going on, so I'll have to go through and debug most of the systems to make them work right again. The second is that I'll need to rethink the entire network topology (except for my "lame devices" subnet -- those devices also can't use IPv6, so I'll have to maintain an IPv4 network just for them).
Rethinking the topology is probably the part that I'm least excited about, because it seems that I'll have to essentially become an IPv6 expert to design and implement that sort of change.
In any case, I anticipate at least a week of having a broken network ("broken" meaning just basic internet connectivity) while I figure all that stuff out.
> I can't rely on my existing IPv4 knowledge.
So you are a network engineer, but you just don’t want to move with the times?
In my experience, that depends highly on what you want to accomplish. For example, tell me how much less complex it is to inform a device on the network of what the address of the NTP server is in IPv6 compared to the IPv4 address. Assume you rely entirely on SLAAC, because that's the preferred way, or so they said.
> So you are a network engineer, but you just don’t want to move with the times?
I don't call myself a baker just because I can make a loaf of bread, but I then I've never been one to over-inflate my CV...
It's not like I don't want to move with the times, but I find IPv6 quite complex and rough around the edges in practice, especially for a homelab network.
Who's "they"?
I think RA for addressing/DNS and stateless DHCPv6 for everything else is the way to go these days.
But SLAAC makes me equally nervous. If/when I shift my home network to IPv6, I'll probably just go with static routing to avoid both of them.
If you're that paranoid then you also need to worry about IPv4 ARP and IPv6 ND attacks. Again, managed switches are required to drop unauthorized ARP and ND replies.
Yes, thank you. I'm aware of these features.
I never claimed that my nervousness is rational. But it's very real. This is complex stuff, and since I haven't been working with IPv6 for decades, I can't shake the fear that I've made a configuration error and am not aware that I've made a configuration error.
It is? It sure seems more complex to me in most ways. There are a couple of things that are simpler, but not many.
Of course, neither is true.
I am not opposed to IPv6 in any way, but I'm also delaying changing my home network to IPv6 for as long as possible.
It's not from fear of change or not wanting to learn new things at all (I know IPv6 as well as I know IPv4). It's just that making that change is a whole lot of work, and IPv6 doesn't bring any benefit to me that I care about. I'll put in that effort when the situation changes and I get some benefit from it.