IPv6 Performance – Revisited [video]
ripe73.ripe.net
ripe73.ripe.net
I recently fixed a performance bug in our (FreeBSD based) TCP stack where having just 10% of connections be V6 on a busy CDN server would result in performance issues due to massive lock contention on network related hashtables. The issue was that when an ICMP (like must fragment, for path mtu discovery) was received targeted at a connection, the V4 code would look up the appropriate connection (with a read lock), and do the work for that connection. However, the V6 code would take a write lock on the global hash table (shared with v4) and iterate over the entire table while holding the write lock. So the entire system would come to a screeching halt, with readers spinning on that write lock while V6 bumbled along. This is something most people would not notice, but on a busy CDN server with ~100K connections serving 80-90Gb/s, it was pretty deadly.
The cause of this bug was primarily because people have been optimizing the V4 code for years, but the V6 code "scary", unused, and often forgotten about. So people mostly forget about it when making optimizations.
Looking at some data from last night, one of the biggest sources of lock contention on one of these 100G boxes is still in the v6 code, even at a site that's serving only 15% of traffic as V6. Sigh.
I am (or effectively have) inheriting (-ed) an aging small campus network. No external services, and no core internal services (beyond DHCP). We'll have to get all new core switches (still on 100mbit...). If my isp gives me a /64 ipv6 net, and I firewall off windows filesharing (and take other measures appropriate for placing user devices on the routable Internet) - can I expect most things to just work?
I'm fine with demanding windows 10, recent Linux, iOS or Android for wired/WiFi access - and plan on moving to/requiring 802.1x for access (which means setting up some form of auth/authz back-end, possibly Azure Ad - more likely a Linux directory server). Can i live the dream and just go for ipv6? Am I crazy because I'm considering it?
(granted, nat64 is also nat - but on paper, it would be so nice to just stick to ipv6...)
The thing is that there are still many websites that are only accessible from IPv4, and in order to access it either you or your ISP would have to translate IPv6 address to IPv4 (NAT64) and make connection to it. If you will be the one doing the NAT then you will need to have at least a single IPv4 address.
Many thanks to all the "nay sayers" in this sub-thread. Helps me manage my expectations, if nothing else :-)
But now, this new network is my home network. So if I want ipv6, I have to maintain it myself :-) (This is, assuming that the upstream ISP here that provides the fiber line will be able to provide ipv6, but I'm fairly certain that shouldn't be a problem)
Private addresses are discouraged, please use public addresses (you will have plenty) and filter them correctly. If you must, you can use link local (fe80::) or unique local (fdXX:XXXX:XXXX::/48) addresses.
And I'm not talking about the difference such as absence of default gateway concept, replaced by router advertisement. I'm talking about having to build a new parallel Internet, beside the old one, and having to support both networks indefinitely. IPv4 sunset is an illusion. We're stuck with it to the end, and no incompatible protocol will replace it.
At this point, it would be reasonable for a mobile device focused on the US market to only support IPv6 over the air; although not if you wanted to use it overseas.
There's a clear path for new end-user devices to be IPv6 only, and ISPs to only support a protocol gateway where they interconnect. It's probably a little early for ISPs or devices to start. We're still in the phase where a service can use IPv4 only with little penalty; but there are enough customers with IPv6 enabled, that it may be compelling to use it too.
What other choice do we/did we have? I've responded in the past to a different commenter, who lamented IPv6 not being backwards compatible. You can read my response[1] to that comment; essentially, I don't see a way to make IPv6 happen, aside from the introduction of a new, separate network, mostly due to the pigeonhole principle: IPv4 only has 2 ^ 32 addresses, so how do you fit the 2 ^ 128 IPv6 addresses into that scheme?
You can introduce a legitimately new protocol, like v6 did. You could perhaps add a special flag, or special block of addresses, or a special option — but whatever mutation you perform on IPv4 packets to "upgrade" them, the intermediate routers would still have to understand that flag in order to properly route packets — no? And thus, since they must understand that flag, you're still upgrading the entire infrastructure, and building out a completely separate network, even if it is just IPv4.1.
I do think the IPv4 sunset is far, far out in the future. There will be stragglers for a long time to come, and I'm not sure it's really worth considering when IPv6 isn't deployed to a majority of people yet. But I do think it's possible; at some point, I hope someone realizes that the maintenance of that code is benefiting so few people as to sunset it, perhaps at an ISP-ish level first. No doubt this will break somebody's workflow, though.
Hardware vendors hate it, because it is another protocol they have to deal with, which doubles ASIC testing matrixes. And don't even get me started on how evil extension headers are from a hardware point of view. The only good thing about the new V6 proto from a hardware perspective is the elimination of the header checksum (which was trivial anyway). This lead to hardware vendors delaying support for V6.
V6 is miserable in OSes because everything has to be done twice because you have two stacks (see my other post here regarding the PMTU locking issue in FreeBSD), and changes are applied unequally. Even if you want to say that FreeBSD sucks, pretty much every innovation has some to V6 later than V4 on every OS I can think of. From simple things like TSO and checksum offload to RSS. There are examples of this on Linux, OSX and Windows.
I've always said that if I had a time machine, rather than shooting Hilter or John Wilkes Booth, I'd concentrate on trying to make the ethernet header 16 bytes and trying to argue for an IPv5 that was just an extended v4 without all this added complexity.
On the other hand that flexibility may save us from going through that "we have to upgrade the whole Internet at once" pain again. In the end widespread v6 adoption is driven mostly by v4 address space exhaustion. At the time this got really severe some years ago the implementations in both hard and software were already pretty reasonable and widespread. Would we really be at a higher adoption rate right now with no additional reason to upgrade? (also IPv5 was already used in RFC 1819 for sth. different, that's why they jumped to 6 ;))
> The only good thing about the new V6 proto from a hardware perspective is the elimination of the header checksum (which was trivial anyway).
What about fragmentation which is entirely left to the hosts in v6?
> Even if you want to say that FreeBSD sucks, pretty much every innovation has some to V6 later than V4 on every OS I can think of. From simple things like TSO and checksum offload to RSS. There are examples of this on Linux, OSX and Windows.
I don't think such optimizations coming a bit later had a significant negative impact in v6 adoption (and if v6 adoption would have been higher it would have been implemented more equally). Also bugs get found and sorted out a lot faster if someone is actually using the code.
With html features, we also have the extreme end with websites that support ten or even twenty different OS, Browser and hardware configuration for the same site. That is 20 different test system, 20 different code paths, 20 support "version" of the same live system. So long there is enough users for one version that can pay for it a programmer will be paid to support it.
Maybe the issue with v6 is that it allow dual stack on the client side. If it was v4 or v6 exclusive we would be in the situation that a website could not afford not to support both versions based on the fear that customers would go to a shop where their OS + Browser + hardware are supported.
Being more deliberate and serious about NAT, rather than taking the position that "NAT is evil".
In particular, this would mean throwing away four assumptions about IP addresses:
-- non-ephemerality (an IP address survived for long periods of time, often not even changing across reboot)
-- global uniqueness (your IP address is in use only by you)
-- isotropy (what you think your IP address is, is what all the parties you communicate with think your IP address is)
-- non-overloading (one IP address per host, or at most one address per interface)
In the years since the early 1990s when IPv6 was still being decided, the first assumption was weakened by DHCP, the second by RFC 1918, and the third by NAT. The fourth is somewhat weakened by Dual Stack IPv4 and IPv6.
If instead of dealing with issues like https://www.wikiwand.com/en/Happy_Eyeballs under Dual Stack it were applied in IPv4-only settings where a host would acquire and use different addresses with different prefixes & masks, multihoming and mobility would have gotten easier faster.
However, application layers that embed IPv4 addresses into them were commonplace and even today when a smartphone can swap addresses rapidly (disconnect from wifi, fall-back through 4g or the like, connect to new wifi) causing user-noticeable disruptions. IPv6 does not help in this situation because the same assumptions are in the way: addresses are long-lived, a host uses approximately one globally unique address at a time, and IPv6's "NAT is evil" foundations make using NAT more bug-prone -- and NAT involving IPv6 on at least one side of the NAT is in widespread use.
CATNIP was one IPng proposal wherein there would be canonical translations among various protocols, with extensibility to add yet more protocols, that would face these problems and more like them head-on. It was rejected because the compromise SIP+PIP was "more ready" to be rolled out immediately, and taking the time to get the details of CATNIP right (or do it iteratively) would mean a delay in standardization and thus deployment. That was more than twenty years ago. Standardization was done pretty fast for IPv6...
The lessons of slow and reluctant IPv6 deployment have been mis-learnt as, "we can never introduce a new network layer protocol in less than twenty years".
Because (in part) isotropy and global uniqueness are taken to mean that there's no protocol translation "in the middle" or "at the edges" allowing completely uncoordinated deployment by any size site (from a backbone ISP to an individual household). That is, as with IPv6, the expectation is dual stack with one stack doing the heavy lifting, and with the other stack used only in coordination with some "upstream" or "peer" network.
Finally, I think the wish to sunset IPv4 is backwards. IPv4ever should be embraced, with IPv4-only devices never needing to upgrade to talk to the parts of the Internet that no longer talk IPv4 themselves. Protocol translation and Application Layer Gatwaying are things that should be made to work well, rather than things to be called "evil".
The lessons learned in doing that should be applied to progressing beyond IPv6, too.
http://packetpushers.net/podcast/podcasts/show-275-future-of...
Then tell us how that magical "compatible" protocol would look like. It's compatible as much as both can coexist and also IP stacks expose a backwards compatible API making it relatively easy for applications to support both.
> IPv4 sunset is an illusion. We're stuck with it to the end, and no incompatible protocol will replace it.
The ongoing support of the IPv4 is itself not so much a problem, it's there, you don't have to change much if anything. The problem is more of getting v6 support into each and every network device (mostly done now) and overcome the laziness to also configure v6. The "support both" problem is more the other way around, having to also support v4 makes much of the nice features of v6 (e.g. more flexible or in some sense easier configuration) less beneficial so they can not work as a driver to adopt it. I also don't think v4 sunset is an illusion, there will surely be a tipping point somewhere (hard to predict) where some decide v4 is no longer worth the hassle for each and everything pushing the last non-v6 holdouts forward. Sure v4 will not be completely go away anytime soon I think, more of a slow fade out.
A big problem nowadays may be that the impact of the IPv4 address exhaustion is distributed unevenly making it really painful for some maintaining v4 connectivity compared to simply activate IPv6 while others can still comfortable ignore it. This may also lead to v6 being deprioritized by those who badly need it because you have to put so much hard work into making v4 work somehow, you can't do without anyway.