IPv6 Is A Disaster (but we can fix it)
matduggan.com
matduggan.com
If any "new" computer technology has been around even half as long as IPv6 ( https://en.wikipedia.org/wiki/IPv6_deployment#Major_mileston... ), with even a tenth of the "you gotta start using this!" push from the Big Boys - and yet still is very widely avoided/resisted, and the older-tech alternative commands a price premium due to widespread demand...gosh, that "new" technology must absolutely suck, eh?
The things that IPv6 would enable (direct end-to-end connectivity) is now seen as a negative by the industry that has since pivoted on rent-seeking, walled gardens and restricting user's potential. The industry is now even legally making money on many things that would've been considered outright malware just a decade ago.
People being able to host things themselves, or local-first apps that communicate directly without the need for any middlemen is a negative for the industry. The industry wants there to be a technical need for a middleman, so they can provide that and seek rent over it.
There is no user-level demand for IPv6 because the industry is no longer making any apps/devices/services that would take advantage of end-to-end connectivity (even if it was available now - let's say in a hypothetical world where IPv6 adoption is 100%) since it's more profitable not to, so as a result there is no pressure on ISPs to offer it.
Sounds a bit over the top. Can you name some examples?
Likewise with analytics - tracking every single action you do in an app (along with generic metadata such as IP addresses - which often leaks your general location and your relationship with anyone on the same network since you'd be sharing the IP address with them) would have been considered spyware.
When there were talks of tracking people for ad targeting in the early days of the internet people (rightfully) freaked out, even though that tracking was really primitive by today's standards.
All of those things are now considered legitimate and are routinely done.
I suspect that if IPv6 limited itself to just increasing the size of IP addresses, IPv4 would largely be a distant memory by now.
The proposals that seemed backwards compatible were just aggressive CGNAT consolidating even more power in the hands of IPv4 address owners. That doesn't seem like a sustainable fix in the long run.
True, but it would limit that break to a single thing. That's much easier to deal with than the whole basket of things that IPv6 brings with it.
In general, despite the complex vocabulary about most of it, in many ways IPv6 is simpler than IPv4. Its header has fewer fields. Its QoL/QoS fields aren't accidental hacks on top of old debugging fields but intentionally designed fields for that very purpose. SLAAC is a simpler protocol than DHCPv4, though the algorithm sounds more complex at first. (DHCPv6 is basically as complex, but fewer devices and fewer subnets should need DHCPv6 in the first place.) Much of the "basket of things" that IPv6 brings with it are designed to remove complexity that has concreted around IPv4.
They ripped the bandaid completely off with the backwards compatibility break that they made with IPv6, and apparently a lot of people loved the cute stickers they had applied on top of the bandaid. But at this point it is probably better for the skin below to heal without the bandaid than to continue to sticker and bandaid over that and let all that unnecessary glue fester in place. (To push such a metaphor almost to its breaking place.)
It's not really about whether or not IPv6 is simpler than IPv4, though. It's about how painful moving from IPv4 to IPv6 is. And it's very painful. If the only thing that changed between the two was that the IP address space is bigger, it would reduce the pain of changing.
I'm certainly not going to claim that my experience is representative of anyone except for me, but the reason that I'm not going to shift to IPv6 until I literally have no other choice is because doing so is an enormous undertaking. Since IPv6 doesn't bring me any benefit that I care about, there is no reason to do so unless I simply can't get on the internet without it anymore.
Please note: I am not bashing IPv6 here, and I'm not saying that a change isn't needed. I'm just expressing some of the reasons I've seen why people resist changing to it, and that I think adopting it would have happened within a reasonable timeframe if it weren't as ambitious.
The only pain I've ever seen is in corporate networks where all the tooling around the network management are IPv4 only but those would break even if you add a single bit to an IPv4 address.
Mostly because of the number of machines that I have to fix up and the fact that updating each of them involves a fair bit of time. My estimation is I'm looking at at least a week's worth of work, during which my network isn't fully functioning.
It's also complicated by the number of devices I have that aren't possible to make work with IPv6 at all, which means I have to maintain some IPv4 segment and deal with making the two work together in a seamless way.
I'm also very concerned about security. I'm not confident that I know enough about how to secure an IPv6 network adequately, so I need to set aside a fair bit of time for study before I even start.
All in all, it's a large project with a lot of friction that wouldn't be as large if I didn't have to rethink everything.
But the last time I mentioned half-measures here on HN, I got dumped on pretty harshly.
If they're IPv6 and I really need to reach them anyway, then I could imagine writing a special NAT that keeps a database equating IPv4 addresses (that I make up) with real IPv6 addresses on the net and doing a NAT that way. That would be suboptimal because it would mean that I'd need to add a database entry in advance for each internet destination, of course. But it seems feasible. And I can think of a couple of ways to automate it.
But, honestly, I don't know. I'm just spitballing what the best way of handling all of this actually is. Every method I have heard or can think of has some serious downside, and I don't have sufficient expertise (and haven't spent the time) to do actual cost/benefit analyses of the various options yet.
This is all a huge time-consuming hassle, and is why I'm putting the whole thing off until I have no other option.
Though they exist, my understanding is that the need and use cases for them is overall much weaker than NAT64 or NAT44.
Software isn't ready or was written back in the early days before they figured out how IPv6 would actually be deployed. I had an Asus router back in the days, it had primitive and unstable IPv6 support, which just grabbed a single /64 and that was it.
I then ran pfSense for some years and it was unusable for IPv6 due to everything being geared around fixed prefix. Even if my ISP had given me a fixed prefix it would have been a pain, because I got two new cable modems in that period (one due to the previous being too old, other one due to a move), so would have gotten a new prefixes anyway and would have had to rewrite all my firewall rules. Would have been major PITA each time. No such need with IPv4.
I switched to OpenWRT some years ago and it's mostly worked since. Mostly.
Android phones in the homes don't respect my DHCPv6 settings, and it took me some time to figure out it ignored my DNS settings because I had only IPv4 configured as my local DNS server. Other boxes were fine with that, but not Android, which silently used Googles DNS instead.
Running dual stack isn't ideal either, since it can lead to inconsistencies. When some things work and some things don't, and it can be difficult to figure out why because of the non-trivial interaction between IPv4 and IPv6.
It's also painful because just about everything is different. So very little of what I know of how to configure my IPv4 network carries over. It's very much not just IPv4 with more bits. I'm old enough now that I'm not terribly excited about digging around in the IPv6 technical weeds, and I haven't found a good guide or reference I can fall back on.
There's zero proof that an "extended IPv4" would have been adopted on a "more reasonable" timeframe, no matter how you define "reasonable" (faster, I guess is what you are arguing for?).
Exactly where and how do you expect "just add more address bits to IPv4" is an easier transition than IPv6?
The IPv4 header is a fixed size. You can't add more address bits without breaking existing routers. Period. End of technical story. You could embed the additional address bits in the next layer up (TCP/UDP) but you greatly increase the complexity of routing equipment by making it have to understand those layers, to what benefit? In the dual-stack real world we do that all the time with VPNs and STUN tunnels and other gateways and tunnels. We have those exact same tools, already, and those haven't made the transition any more "reasonable", have they?
But it's worse that while routers don't understand the extra bits, the parts of the addresses that get used (the prefixes small enough to fit in IPv4 headers) have to become massive NAT gateways and become massive gatekeepers of huge parts of the IP address space. We know from deep experience that IPv4 address allocation wasn't "equitable" (ARIN got way more space than RIPE and both got more space than AFRINIC and so on; companies like Microsoft and GE got /8 allocations just for asking in the right years).
Does it make that much sense to establish existing IPv4 holders as the forever "landlords" of the internet? That seems to me to only add more incentives to make the transition more unreasonable than IPv6: why support router initiatives that understand the additional address space when the IPv4 address holders can get "extra rent" if they don't, presumably charging all their "downstream" traffic for their gateway usage? We're in a time where IPv4 addresses have noticeable rental costs, I can't imagine what that would be like in a world where large parts of address space have to be on VPNs controlled by IPv4 owners. That doesn't sound to me like a good present or future for the IP protocol, no matter what.
None of these things are IPv6's fault.
Hell, give me remote access to your network and I'll set it up for you -- at least enough to get you started on it if not 100% on every single thing. I don't expect you'll take me up on that offer, but it'd be easier to just do it than tell you about it, since you can't tell people anything: http://habitatchronicles.com/2004/04/you-cant-tell-people-an...
Nah, the problem is ipv6 has been designed by a commitee for a lot of enterprise-ish features so the hobbyists have taken a look and postponed setting it up internally for when they have absolutely no choice.
I've asked for simple ipv6 tutorials in discussions on HN and elsewhere and whatever I got pointed at was always longer than the article we're discussing and incomplete.
Basic set up of ipv4 for a home network can be explained over just one pint. Looks like you need two barrells for ipv6.
And that's only for configuring your router, not your local network...
There really does seem to be a lack of good documentation about all of this. The docs that I've seen appear to be aimed at actual network engineers, or are so incomplete as to not be worthwhile.
I would be much less stressed by all of this if I could find something good that sits between those two extremes.
A part of me, though, suspects that the reason there is no "middle ground" documentation is that it's not possible -- that IPv6 is too complex for that. Not saying that's the actual reality, but it has the whiff of it.
I asked the other guy this, but I'll also ask you. Please provide me the level of documentation you are looking for, but for an IPv4 network. If you have some grand tutorial that explains it as easily as you make it out to be, then I truly would love to see it, I will include it in my onboarding documentation at work.
Because I understand both IPv4 and IPv6, and do not consider IPv6 the more complex protocol by any measure. I suspect your "whiff" is more a bias towards what you are comfortable with, rather than a true reflection of IPv6's complexity.
You mentioned "new hires" while i mentioned hobbyists. You're talking about a business network where people are paid to do it, I'm talking about home networks and home labs.
You're basically confirming my statement that IPv6 was designed for enterprise needs?
Could help to get an idea what you see as "hobbyist level". Setting up IPv6 has been pretty straight forward on plain Ubuntu as a router just with replacing the dhcpd subnet configs with a radvd config, enable ipv6 forwarding in addition to ipv4 forwarding and replace the iptables NAT rule with a only forward RELATED,ESTABLISHED connections (which already is optional).
The path of least resistance with ipv4 was to have static ips for every PC and change the default route where I cared about which connection the PC used. Also was using the dhcp from the home router.
Now ipv6 has addresses that are auto generated by the OS on boot, addresses that are forwarded from the router and are in a subnet assigned to home ISP #1, and ... whatever I'd need to assign manually?
It's still hobbyist level not enterprise level if you ask me. But it's a lot to read for a network that works just fine thank you.
The home connection is probably dynamic. I asked them about paying extra for a fixed IP and they said they're all out. They meant IPv4 of course but that doesn't mean i get fixed IPv6 now.
> A better alternative IMHO would be to use different VLANs for the two networks but then your switch and possibly access point have to support it.
... and it might be fun getting the internal machines to talk to each other? I ssh around all the time.
I recall reading about Facebook's internal IPv6 migration for their data centers and the problems they ran into. The two that stuck out the most to me and which I still remember details for are:
1. The PHP function developers used to convert an IPv4 address into an integer to store into a database or something. It didn't work in IPv6, meaning the code broke badly when it was considering an IPv6 host. They kept asking developers to stop using it, but new code kept getting added which used it. Eventually the decision was made that 'we've warned you enough, so from now on we're just going to go ahead and let your code break'.
2. Their networking hardware wasn't extensively tested for an IPv6 single-stack deployment. Turns out that when presented with a BGP advertisement that contained only IPv6 routes and no IPv4 routes, their switches would immediately crash. How did they discover this? By sending out a BGP advertisement that contained only IPv6 routes and no IPv4 routes, crashing every rack switch in the data center. There's no reason why this should happen; it's not a violation of the BGP spec or anything, it's just a bug for a case that no one tested on the vendor's end because none of their customers tried to do that.
So it's not that IPv6 sucks; it's actually pretty great in a lot of ways. The problem is that everything else sucks; people don't bother to support things, they don't consider it in their code, they don't test for it, they don't bother to roll out support because no one else has done so either so why bother, and so on.
In the end, IPv6 is 'avoided' because of the problems that Facebook ran into, or that OP ran into, and is 'resisted' because it's extra work that they could put into doing other things with their limited time and engineering work.
To be clear, dual-stack deployments work great; my ISP recently did a trial of a dual-stack deployment which I was lucky enough to participate in, and it was almost completely transparent. My Unifi gateway picked up the address and handed addresses out to clients internally, clients used IPv6 where appropriate and fell back to IPv4 when necessary, and everything worked completely transparently.
TL;DR IPv6 isn't the problem, the industry is the problem.
OTOH - for people making real-world decisions, the difference between "$Networking_Technology sucks" and "~all available implementations of $Networking_Technology suck" is pretty meaningless.
Consumer routers still suck regarding IPv6. Last time I tried setting up IPv6 on my wan I got a /128 which is utterly incompetent.
No service wants to cut off access to the ipv4 customers so they've made things just work. We have only recently hit address exhaustion (relative to IPv6 age).
No one wants to jump first and there is no government mandate for support.
I don't know who you think the big boys are, but it's not like Google or Meta have throttled ipv4 services or put banners on their site warning users they are on a legacy protocol.
The important part is the delegated prefix, which you normally get via DHCPv6-PD and should be at least a /56.
If it is built to last a 1000 years or more, ~25% of internet traffic by the end of the first 30 years isn't a terrible rollout curve. That's 0.3% of its expected lifetime. (Assuming the 1000 years clock started 30 years ago. It's even believable the 1000 years clock starts closer to 90% rollout of IPv6 than to IPv6 announcements 30 years ago. IPv4 address scarcity wasn't truly felt until decades into IPv4 usage, though it was an academic concern.)
Many Internet projects have always run on different timescales compared to the vastly faster timescales people associate with software and even hardware generations. (In part because these rollouts happen across software and hardware generations.) The IP protocol is so deeply fundamental to that, it is somewhat "civilization defining". It would probably be a lot scarier if an upgrade to something fundamental like IP was an "overnight success". Civilizations are built on the timescales of decades and lifetimes. It should not be a surprise that IPv6 rollout has happened on such timescales. We mostly can only hope that the IPv6 architects were as smart as they hoped they were in preparing for the deep, unknown future of the internet, because they knew they were working on a civilation-timescale tool. (Which is exactly why IPv6 is not just "IPv4 with bigger addresses".)
I don't think that measuring the rollout curve relative to the expected lifetime of the thing is reasonable (or at least, useful), though. In terms of something like this, measuring it relative to when IPv4 is simply no longer feasible is better. And that time is very, very near.
Like, I said this elsewhere, Cloudflare public DNS is 1.1.1.1. If I switch to ipv6, I get to use 2606:4700:4700::1111. You telling me that's an upgrade?
That's a serious question. Do you have a way for this to work? Because I don't think it's possible (for fundamental reasons: https://en.wikipedia.org/wiki/Pigeonhole_principle), and it would be extremely unfair to criticize the designers of v6 for not fixing an unfixable problem.
The concept of vanity IPv4 addresses was invented in 2009, when Google acquired 8.8.8.0/24 from Level3. This is an emergent feature of a small, densely packed address space. IPv6 had existed for a decade (EDIT: not two decades) by that point, so you can't really blame the designers.
Sprint controls 2600::, probably by accident, but they're not doing anything interesting with it.
Maybe the bigger issue was trying to get rid of NAT. People don't want every local network device to have a public IP and have to trust that the router's v6 firewall will do its job.
I absolutely don't want this. But as I understand it, I can avoid this by assigning my machines the IPv6 nonroutable addresses fe80::/64. They're the equivalent of 192.168.* and 10.*
Meanwhile, if someone sends a v4 packet with TCP port 22 to my router, it can't tell where to forward it even if it wanted to. It takes effort to do that, namely a port forwarding config.
If you use DHCP, then I think you can configure that. What I have in mind is to assign static IPs to all of my fixed machines anyway, and use DHCP to assign IPs to transient machines. Not sure if that's reasonable, but it's my current thinking.
> does every router (even crappy ones) respect the no-forward rule?
There may be broken ones, but it doesn't matter so much because your ISP won't route such addresses regardless.
You probably can, but as it's often said, most security breaches are caused by misconfiguration.
>There may be broken ones, but it doesn't matter so much because your ISP won't route such addresses regardless.
Yeah fair enough, I can trust my ISP to do that at least.
Yes, but that's no different with IPv4. What it really means is that we have to learn the intricacies of IPv6 in order to use it confidently. Right now, I am nowhere near comfortable that I have sufficient understanding. That can be fixed through enough study, but is also part of the friction in adopting IPv6.
Guess the most likely mishap is a bad router supports upnp and has it on by default, and a bad device maps an actually used port. No PC is going to do that, it'd have to be something like a cheap knockoff security DVR.
I don't think that's the case. I think how you set your router/firewall rules with IPv6 is the same as with IPv4 aside from the addresses being longer.
> it's really hard to mess up NAT in a way that causes a breach
You can continue to use NAT with IPv6. I know that when I make the change, I'll still be using NAT, for convenience if nothing else.
They are called Unique Local Addresses (ULA) and are in the range fd00::/8.
Which itself is so much better than RFC 1918 addresses. If you need private, non-Internet routable addresses, then you generate a random one. In the event two private networks need to communicate over VPN, for example, there is no clash.
If a connection comes into your router with the destination IP set to one of your LAN machines, NAT will not stop the connection.
There's no reason to be using NAT to protect yourself from inbound connections, because that's not a thing NAT even does in the first place. It often makes things actively worse even, by making it easier to port scan for your servers and by giving you a false sense of security.
Also, any change to the protocol was going to be a massive shift regarding network hardware. It wasn't ever possible to slap a few more bytes onto the address.
If you're going to make a monumentap shift, why not do it right?
yes, that was the argument from the very beginning, and it's not without merit. I disagree with it, because it's making a monumental shift into one that is even more monumental and increases resistance to making the change at all.
But who knows? Both sides of this argument are just speculating.
There have been other huge migrations pulled off in networking. HTTP->HTTPS is the first that comes to mind. Extra layer of security, no other changes. Browsers slowly made users more and more wary of unsecured sites, and it became easier for site admins to obtain SSL certs. Once plain HTTP was finally made uncommon, versions of SSL/TLS still got upgraded slowly. They also avoided making it too flexible and turning into a fragmented mess like email or XMPP, i.e. browsers strongly avoid self-signed certs and started banning old versions.
I understand that he's building a usable service and just trying to git 'er done, but it's a lot of hacks, so I'm glad they're documented in this here blogpost.
I hope that he can continually probe the edges to find out when real IPv6 support becomes available, and can gradually remove the hacks for a purer experience.
My hope is folks know of workarounds and I’ll do them and update the post.
I feel like excluding IPv4 folks is a large reason why IPv6 continues to fail. I feel like this is a pretty good compromise between pushing IPv6 and not being an IPv6 hermit in the IPv6 desert.
This could be done on the ISP/enterprise level, but it is more counterproductive to tell IPv6 adopters and promoters that we need to bend over backwards and hack in NAT and purchase/rent/lease public IPv4 addresses, when this is not our problem anymore.
I feel like the more juicy services that are IPv6-accessible-only, the more it will drive consumer demand, and will light a fire under people who are responsible to update the support and ensure that IPv6 works, even when IPv4 doesn't.
It's mostly the Enterprise level that has failed to get the message and is failing the IPv6 internet. Even just the examples in this article: It makes zero sense that GitHub still has no AAAA records (and is increasingly slow and lethargic on mobile carriers via NAT64 gateways; it is not just that their mobile app is only so-so, it's also their networking is slow). It makes zero sense that Docker put its AAAA records on weird secondary domains instead of their main domains.
Now that all of the major cloud providers are charging for IPv4 address space on a per-hour scale that might see reflection in bottom lines in IT budgets, maybe there will be a fire finally lit under Enterprises to consider using more and better IPv6.
I.e. it allows you to push IPv4 to your internet edge only in a way that doesn't downgrade anything about actual IPv6 capable connections. For a single server that probably does seem pretty silly but once you have multiple it can make more sense.
What I want it for is so that I can have services exposed through my domain name, but operated on different internal servers.
Some ask "why bother with IPv6 if I'm still going to do that then?" and generally the two key advantages are the fe80:: address co-exists with the unique public address of each box so you don't need outbound masquerade NAT pools and the fe80:: address space is enormous+interface specific so you don't have to worry about unique internal space or conflicts with other networks. Or, if you have a static IPv6 assignment in a more "proper" hosted deployment instead of a dynamic home deployment, you can of course just do stateless NAT to the public addresses without worrying about IP scarcity.
If you want to NAT inbound connections, you can do it without NATing outbound connections. Essentially: you don't need to NAT, you just need to port forward.
Honestly, I think you should just suck it up and use different hostnames for different services, because running all of your services on one IP is really bad for security since it makes it much easier to enumerate every service you're running -- it only takes scanning 65k ports on one IP to find them all, rather than 65k ports on 2^64 IPs. That's the difference between megabytes and yottabytes of port scan traffic.
If you NATed outbound connections to also come from this IP then things get even worse because every outbound connection any of your machines make would immediately inform the server of the IP needed to make an inbound connection to you. That's a completely unnecessary security sacrifice.
But if you're gonna do it, you can do just that, without trying to run the network on some local IP range too.
This is the major flaw. Sites can't go ipv6-only. In reality we will have parallel ipv4/6 networks in place until the wheels fall off
Which hardware and OS are you describing? Clearly the blog post illustrated some examples where switching to v6 was not happening, so it seems to contradict your comment right off the bat. There are many implementations of dual-stack IPv4/v6. In fact, they are more divergent than IPv4 implementations, because the latter often derives from the BSD-RENO codebase, while IPv6 was introduced after Linux became King, so Microsoft, Apple, and Linux (and lots of router/firewall vendors) have ostensibly developed IPv6 stacks separately, some being more open than others. They're not all going to work the same way with fallbacks/failovers.
> Which hardware and OS are you describing?
This is how it's supposed to work on all OSes; on any recent BSD (excepting perhaps Apple?) or Linux setup, it should work this way.
> Clearly the blog post illustrated some examples where switching to v6 was not happening
In those situations it was for connecting to services that do not advertise a AAAA record.
https://www.rfc-editor.org/rfc/rfc5220.txt
I'm not sure why you specified "if a host has no route to the address", because that's a very specific and transient failure. Furthermore, the dual-stack handling necessarily happens in the application, so this is not an OS or kernel-level decision, it will be subject to each individual app's behavior: https://issues.apache.org/jira/browse/SERF-190
As you can see from RFC5220, IPv6 is preferred over IPv4, unless an option is set to swap those around. Of course, certain configurations can confound this preference order, such as ULA IPv6.
"No route to host", as should be obvious, is only one of many errors that could prevent an app from establishing an IPv6 connection. It would seem that they should handle most failures as an occasion to fall back to IPv4, unless configured not to.
The rules are somewhat involved but they roughly boil down to "sort v6 addresses first if the client has a non-ULA v6 address, otherwise sort v4 first". However, note that the very first rule in 6177 is "avoid unusable destinations" so not having a route to an IP may factor in to the sorting.
A machine with no v6 will(/may) still query AAAA records, but it will attempt to connect to any A records first before trying any AAAA records. (This sometimes exhibits as people seeing software like apt-get report connection failures to v6 addresses and then blaming v6 for it, even though the problem is that they only have v4 and the v4 is broken.)