GitHub is preparing for IPv6 support for Github.com | Hacker News Reader

GitHub is preparing for IPv6 support for Github.com

githubstatus.com

111 pointstonymet67 comments

Funny, around 25 years ago as a university project I added IPv6 support to the Java JDK. Today we're still not there.

Java's not there either: https://bugs.openjdk.org/browse/JDK-8170568

On the scale of migrations, this is worse than Python 2 to 3 (which was really bad), but better than Perl to Raku (the relevancy is entirely gone).

Perhaps a smaller migration should have been considered first.

I think the problem with Py2 to Py3 was that it was too big of a migration to be easy but also too small of a migration to be game-changing. Like, switching for seriously improved packaging, parallelism, or performance would've been more compelling.

And yeah ipv6 could've been a much smaller change.

At least py3 runs on the same hardware.

Most hardware supports ipv6 nowadays (I think), but the problem is something else.

Yeah did anyone make the switch to Raku??? I don't really understand it.

I’m curious how this wasn’t caught in testing. When you set up a dual-stacked listener which is common when you omit the binding IP, you already get IPv4-mapped IPv6 address, and the request logs would have also shown the ::ffff:x.x.x.x addresses. Also curious as to why they are working with text representations of IP addresses instead of binary representations, although the incident GitHub described here could be very well a firewall deciding that IPv6 datagrams were invalid traffic.

However, to make this thread complete, I guess someone will again complain about the address format, without realizing that shoving in extra address bits on an IPv4 datagram is already a new protocol, the networking equivalent of Godwin’s law[1].

[1] https://news.ycombinator.com/item?id=39101523

I'll be that guy. Glad to know I'm not the only one who just wants ipv4 with extra octets.

Edit: An aside, these addresses are different in both text or binary representation, but the article doesn't provide many details, so I can't entirely follow.

The other big problem I see to getting buy-in from non-networking people is the lack of good sources on how to prepare and roll out ipv6. It takes forever scrolling through pages of documentation that go through the history of ipv6 and the ipv6 numbering format. As though the world needs yet another primer on how long groups of 0's can be dropped to make ::1 be a valid address. Just link to Wikipedia's page explaining ipv6 addressing and move on.

Seems like this is a common error, systems that treat IPv4 and IPv6 addresses as entirely distinct things. The mapping is easily missed in new implementations. Wonder how many vulnerabilities there are out there with this same mistake.

It's a common bypass of server side request forgery filtering. Backends will try to validate that a user-submitted url doesn't resolve to an internal IPv4 address, but they'll happily allow an IPv6 mapped version for the same IPv4 address.

Yes, and what other vulnerabilities would be out there? IP6 is something that nearly all machines have, and have enabled, but it is rarely used.

One potential scenario is bypassing rate limits, enabling abuse, scraping etc. because of the large IP blocks each customer is granted, and seemingly the lack of standardization there.

The assumption of there being a finite amount of IPs to cycle for an attack doesn’t entirely hold with IPv6, and seems like not all software is configured to take that into account.

Explained here better: https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/

Unsavory but true, it's nice in a way that ip addrs have a cost. Kinda like phone numbers.

The naive suggestion for ipv6 in that article is what I first thought, block a /64, but since that might still be too cheap, it also quotes a more robust algo of dynamically expanding the blocking scope.

I can just imagine that moving to IPv6 is as messy on Azure than it is on AWS.

Don’t get me wrong, actually using it is simpler, you can just have your range and then be free to do whatever you want. But the fact that it was added later to a stack never made for it shows everywhere, which makes UX tough.

Currently struggling with metadata service weirdness and slowness in IPv6 land on AWS :’)

It's almost certainly worse in Azure than AWS.

Random examples:

Azure hands out contiguous blocks of 16 IPv6 addresses. No, not a /56 or anything useful like that. Sixteen addresses.

If you enable IPv6 in some virtual network, other peered virtual networks will have unrelated services just break. Like Postgres, Azure VPNs, and more.

There are no IPv6 to IPv4 gateways, and you can't even build such a thing yourself without enabling IPv6 in the whole virtual network... which breaks other networks!

Azure NATs IPv6, defeating the entire purpose of the thing. It's basically IPv4 with extra steps.

Azure doesn't support IPv6 for any of their PaaS offerings, especially not in their firewall rules.

Etc...

If you think there are excuses for any of this, consider this: IPv6 has been a standard for two decades and Windows has supported IPv6 since 2000.

I like to swap IPv4 and IPv6 in any sentence to gauge the insanity of it. E.g.: "Enabling IPv4 breaks unrelated services in other networks" would have you running for the hills, would it not?

>you think there are excuses for any of this, consider this: IPv6 has been a standard for two decades and Windows has supported IPv6 since 2000.

That's like complaining that Linux came out in the 1990s yet Photoshop doesn't support Linux. Like how it doesn't make economic sense for Adobe to support Linux, it doesn't make sense for a lot of organizations to additionally support ipv6 when they can just support ipv4.

Microsoft was an early IPv6 adopter, they even experimentally "forced" it onto their customers by making their DirectAccess VPN tech IPv6 only! They control both Azure, and the OS it runs on (Windows). Sure, bits of it runs on Linux, but they have their own distro of that too that they control (Mariner).

This would be more like complaining that Linus Torvalds prefers not to use open source software.

That should be 16 IPv6 /64 networks, right? Which means Azure gives you a /60 prefix, I guess?

No, he’s right. Sixteen singular addresses. And those a quite expensive for what they are.

Very prudent of them to not hand those addresses out like candy. They don’t want to wake up one morning and realize that they’ve run out!

It’s easy to guess what happened: they developed an IPv4-only network stack and baked the limitations and constraints of IPv4 into it: private addresses are mandatory, public addresses are scarce, and NAT is required.

Then they got told to “do the needful” and make IPv6 happen, so they did… by weaving IPv6 support through the tangled briar patch of their codebase. They wove it through the NAT, the tiny public address blocks, and the mandatory private address spaces on virtual networks.

The result is IPv4 with a sticker on it with a hand-written label that says “IPv6”.

“Job done boss!”

With how weirdly complicated Azure was about IPv6 last time I looked into it, I wouldn't be surprised if it'd be more complicated than on AWS.

I'm not convinced it ever improved. Looking up a quick guide brings up https://learn.microsoft.com/en-us/azure/virtual-network/ip-s... which tells you to just... assign a random network from 2404:f800::? What even is this network? Are they using a routable IPv6 address as a substitute for an ULA for their NAT'ing load balancers? Why 2404:f800:8000:122::/64 specifically?

Microsoft owns 2404:f800::/30.

I just did the tutorial and I noticed that in the Azure portal it shows a public IPv6 address and a private IPv6 address. From my machine I connect to the public one and magically end up on the private one.

Curling what is my ip6 from the machine yield the public IPv6 address.

I suppose all of this is needed to ensure LB can be done? And it's easier to do with a range like this than a ULA which by default isn't routable.

They do, but they tell you to use a specific /64, which confuses me. I would've expected them to say something like "the address starting with 2404:f800 in your dashboard".

The "private" IPv6 address can be a ULA without any issues if the network is designed to be fully NAT'ed (i.e. for load balancing, maybe failover I guess). If you're not using the global address on your local machine and translate the public address into a private one, your local network doesn't need to have a routable IP address.

I suppose it works just as well, but it makes using IPv6 more confusing for now reason. It's as if Microsoft decide to use 20.64.0.0/10 for private networking on Azure, which they can do (they own that space after all, they can decide not to use it), but just doesn't make much sense.

I'm curious what issues you're having with the metadata service on IPv6?

> The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality.

Sounds like this bug: "Unable to reliably distinguish IPv4-mapped-IPv6 addresses from regular IPv4 addresses" https://github.com/golang/go/issues/37921 .

Use https://pkg.go.dev/net/netip instead.

Sounds like the opposite of this bug. GitHub's system represents 0.0.0.0/0 and ::ffff:0.0.0.0/96 as distinct values, when they should be merged/normalized.

I think it's best to treat IPv4-mapped addresses as an implementation detail of the socket API, and not leak them into general purpose code.

Though things get interesting when an external user provides "::ffff:1.2.3.4" as text.

Sad that even for experts ipv6 is still a no go filled with landmines.

When I was a network engineer I frequently experienced IPv6 skill rot. I'd go long periods of time not encountering it in the wild. It's hard to be confident with a thing you're not working with on a regular basis. I guarantee you I made mistakes because of it. There was only one maybe 2 year period where I would have considered myself an IPv6 expert.

Any tool can be used incorrectly. IPv4 mapped addresses are a part of the protocol as any "expert" would know.

How many bugs occur due to misconfigured HTTP servers every day? Do you conclude that HTTP is also a "no go"?

Which http are we talking about? HTTP3 that people wanted to use before the security portion of the spec. was even done? HTTP2 that had an entire section dedicated to possible exploits that where later found on major sites? Or HTTP 1 which enabled some of the exploits seen with 2 since implementations are neither consistent nor strict when parsing parameters?

Sometimes it feels as if the Internets security was supported by paper walls with a "please punch here to exploit" written all over the place.

Was gonna say, if this is about ipv4 vs ipv6, better also specify which version of HTTP we're talking about.

Does it matter? Every HTTP version from HTTP/1 to HTTP/3 has each experienced its own set of vulnerabilities.

Never stopped us from using HTTP.

IPv4 has had multiple vulnerabilities in its stack as well, never stopped the internet from growing. (But the exhaustion of IPv4 _is_ hindering further expansion of the internet!)

I mean it's not a good comparison. IPv4->6 is a migration, with a lot of difficulty having to do with the fact that v4 is already in use. HTTP wasn't a replacement for something, but there is HTTP1->2.

Seriously what is going on at GitHub engineering?

How was this not caught in a pre-production environment?

I created a realtime app for us to track https://isgithubipv6.live/

NAT is the main reason slowing down (or preventing) IPV6 adoption.

NAT is a nice feature for home and office. Safe default-deny behavior, clearly separated LAN. I'd even want it with ipv6.

NAT isn't a nice feature, it's a necessary evil to deal with address exhaustion in v4 and causes many problems of its own. Using it in v6 makes no sense.

It doesn't have a "default deny" behavior -- NAT doesn't block connections at all. It doesn't separate your LAN either, that's done by having a router. If you want those things then NAT isn't the thing you want.

You're right that in a technical sense, NAT isn't block connections, more like it allows connections in the first place that would otherwise not be possible. But that's the best part. It's very hard for a misconfigured NAT to result in devices being exposed, unlike a true firewall, which is actively blocking connections to devices that would otherwise be totally reachable.

NAT is sucky in plenty of server-side situations and is thus avoided, but in homes and offices, yeah I wouldn't want it any other way.

It's sucky in homes and offices too.

It's not hard to test if your firewall is working. Using NAT to avoid that is not the sensible approach.

Don't you have any firewalls? My ISPs router blocks inbound connections by default on both address types.

Many internet users don't even know what the word firewall means. And there are a lot of devices with unpatched software. Obviously NAT shouldn't be seen as a firewall, but for many consumers it's pretty much the only line of defense.

I don't know if ISPs without NATs automatically deploy firewalls, but would hope so. IPV6 is harder to scan, but once you know a device's IP address (e.g. from web logs), it'd be easy to exploit older devices.

One example, AT&T cellular gives you an ipv6 address besides v4, and it's still behind a NAT (not just a firewall). Though I'm glad my home ISP lets me decide that for myself.

I have firewalls but don't trust them to have safe defaults. Or more like, I don't trust every random router, including ones that each member of my extended family will inevitably call me about, to have safe firewall defaults until I test it myself.

I reasonably trust, not 100% but pretty well, that my boss's or grandma's PC with ip 192.168.1.3 on NAT (not even routable from the WAN) will not receive inbound TCP/UDP. That would take effort.

Following this logic, don't trust your grandma's router to have UPnP and NAT-PMP disabled by default too. NAT becomes effectively unsecure with them.

This argument is so overused we might as well add it to ipv6-excuses.txt.

Upnp is where the shred of doubt comes from. Only a shred because the PC very likely won't use it, and nowadays few routers have it by default. And the solution is, upnp shouldn't exist.

>nowadays few routers have it by default

And nowadays few routers have IPv6 firewall disabled by default... Do you see how fragile your argument is now?

Enough routers still have ipv6 firewall disabled by default, in which case, every port on every networked device is exposed. Upnp, I'm less worried about being enabled, and even if it is, Windows won't map SMB to itself.

In the long term with our world of cheap crappy routers, it's also much easier for upnp to die off than for ipv6 firewalls to become ubiquitously strong. Both of those are extra features that take effort to do right. So is NAT, but a broken NAT doesn't expose hosts.

My grandparent's crappy ISP-sponsored router from ~2017 (maybe 2016), TP-Link C1200 v1, had the IPv6 firewall enabled by default. (It has UPnP enabled by default too... which I turned off promptly. They don't need it anyway.)

My own Asus router from 2016, RT-AC1200G+, also came with IPv6 firewall enabled.

Both routers are 8 years old at this point. And they are far from being a prosumer product. What router are you using that it doesn't come with IPv6 firewall enabled?

edit: Oh, and my friend's cheapo Netis router (quite an obscure brand IMO) also had IPv6 firewall enabled out of the box.

>ipv6 firewalls to become ubiquitously strong

They don't need to be strong, the firewalls just have to be stateful so that they can drop unsolicited packets. It's that simple. Any other more sophisticated attacks would have made your NAT equally vulnerable as well.

Additionally, stateful firewalls have been available since decades ago. They are not exactly black magic.

I'll have to check the one I have at home when I get home. Airport Extreme is one example I remember not having ipv6 firewalling by default, even though it has it for v4.

It's important to test this. I own an MSI router where the firewall is completely broken for IPV6. They even made the mistake of listening for DNS and HTTP (admin tool) on the external interface.

IPv6 is still confusing for many developers

Exactly, even if my router says "ipv6 firewall enabled," I still can't trust that they did it right. I would rather go full NAT (which on most routers equals disabling IPv6) than have to pen-test my router.

"Full NAT" doesn't even make sense. Do you mean symmetric NAT or something in that vein?

>which on most routers equals disabling IPv6

Turning on NAT on IPv4 certainly does not entail turning off IPv6. Just _what_ exactly are you sprouting? It's not the first time you have made beginner mistakes like this.

IPv6 will never replace IPv4 for good. It is a non-solution that we will have to contend with along with IPv4 until something proper is worked out.

Oh no, is this yet another proposal of IPv4 with extra octets?

No. The notion that it must be IPv6 or extra octets is naive. IPv6 is not worth a complete departure, and it won't happen for that reason.

"This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites."

I am not gonna read this postmortem, but post link to this HN thread as a proof of mass IPv6 adoption.

Sorry, I thought that it would sound like a fine joke. Don't want to offend someone

But but all the people on the interwebs told me that ipv6 is ugly and stupid and it's never going to happen!

> But but all the people on the interwebs told me that ipv6 is ugly and stupid and it's never going to happen!

And that article proves them right.

This article isn't proof, or even a very good example of ipv6's issues.

Terrible reasoning. If the article has truly proven them right, GitHub will have not experimented with enabling IPv6 in the first place.

There was a 42 minute downtime for 0.23% of their user base.

Hardly "never going to happen".

Reply on news.ycombinator.com