AWS and their billions in IPv4 addresses | Hacker News Reader

AWS and their billions in IPv4 addresses

toonk.io

192 pointsbgpdude183 comments

I was working at GE when AWS bought GE's 3.0.0.0/8 block, and it caused a massive headache. GE's network is gigantic, and there are a ridiculous number of assets. They didn't give much notice about the sale, so they shimmed all of the routing internally, but lots of services still pointed to the block.

There supposedly was an agreement that Amazon wouldn't start provisioning them for a certain amount of time, but whatever amount of time was specified, they either didn't honor it, or it wasn't enough time.

We were also moving assets over to AWS, and all of these things going on simultaneously caused what we called the three-pocalypse.

We would occasionally run across issues with external sites or newly provisioned lambdas who were on Amazon's new 3.0.0.0/8 block, but we couldn't reach them because internally that IP address didn't exist.

At the same time, they would open up a small block to allow access to those external sites, and then some internal service would no longer respond. Repeat ad nauseam. It was also compounded by the fact that there are countless teams in GE and not everyone would connect with who made what changes.

Sounds like the perfect time to migrate to IPv6!

IP6 on AWS ECS is a pain, and impossible if you use their recommended and dedicated command line tool.

This has always seemed deeply suspicious to me. It's like AWS is trying to prop up the value of its speculation on IPv4 addresses.

or merely lazy. why spend time/$$$ supporting IPv6 when you have plenty of IPv4 addresses?

Well given that they keep on buying more IPv4, just supporting IPv6 better seems like the far cheaper option. That's what's so weird!

They're buying up taxi medallions and investing in ride sharing.

Truth be told, IPv6 still doesn't work. I've stayed in plenty of hotels and plenty of airport Wi-Fi that only had IPv4 routing. Until it's universal, it can't be considered a solution.

Fortunately public facing systems are capable of being dual stacked so that won't be a problem. There's no reason internal networks can't be v6 only. Microsoft's already done it.

I know they have plans:

https://teamarin.net/2019/04/03/microsoft-works-toward-ipv6-...

But AFAIK that is still aspirational. I would be pleasantly shocked and impressed if they had already achieved that.

Back up about 20 years and you could say the same thing about the (IPv4) Internet as a whole. I stayed in plenty of hotels which had zip for connectivity back then, but here we are now.

Things change...

Can you describe any specific issue, that was not caused by your own misconfiguration or a hardcoded list of address ranges in a major third party service?

The situation was internal to the company. Neither cause you described was at fault, as per the post your responded to. Side effects of a massive bureaucracy are lack of information retention and communication, miscommunication and inaction (in decisioning as well as technical configuration).

I worked for NBCU and used to manage 3.3 3.54, 3.23, and about 5 other 3.x /16s. We had some design policies created by an elderly architect that was a bit lazy, so instead of VLSM for point to point links we would use /24s. Ahhh, the good old days.

This is what I think about when I'm told I should use /64s for point-to-point links in v6.

Some say that he is still doing that today.

Sounds like a very bad IPAM - probably was a good move to sell it, since it showed all the places, where the network was not managed well and brought in a decent sum. At least so it seems from the few sentences you have written.

Also, in year 2018 GE could have been ready for IPv6 but that would require not only existing IPAM but also some proper leadership, which based on your words GE doesn't seem to have.

What's IPAM? Until about five years ago my F100 company barely used more than a network drive excel sheet for our public /16 + RFC1918, and they contacted the network security team to ask what addresses were used in the firewall NAT tables to determine what was free to use.

IP Address Management systems. Similar to solarwinds Orion.

We used that particular product extensively at my previous company. If you stole an ip without putting it into Orion there was a job that would enumerate/update info, if possible.

If the job failed, you and your boss would get an angry email.

This was in a company probably much smaller than F500.

Something like https://www.globalservices.bt.com/en/solutions/products/diam...

Used it at a previous org I worked for. Pretty nice - it can crawl your routers, build up your existing network allocations then help you analyze/optimize them.

Stupid powerful - we used it mostly for decentralized DNS management (!!) but after a while I finally got the network and security folks to realize what the system could do and the spreadsheets started to finally go away.

Speaking of decentralized management it has robust hierarchical role-based security - we had thousands of site admins managing DNS and eventually IP addresses for their individual sites, but also smoothly maintaining overall command-control. A very cool system.

Ouch. That does indeed sounds painful. I hope GE at least got a big pile of cash from the sale :-p

On a similar note, AWS purchased some chunks of the 18.0.0.0/8 block from MIT, much to my dismay. It was so much fun having a class A network back in school, and getting static IPv4 addresses for projects was easy.

On a separate note I had hoped for some time that an AWS 18.x.x.x address would be useful to get access to journal papers but I tried and that sadly didn't work :(

> They either didn't honor it, or it wasn't enough time.

Knowing what they say about both companies, it was probably both reasons at the same time :-))

I worked for HP back in the late '80s/early '90s, and it always seemed insane that they numbered everything internally with their 15.*/8 block, especially as you weren't even allowed out on the public Internet at the time. You had to get a manager to sign a paper to get access to a SOCKS proxy.

This is how the Internet is supposed to work, with unique addresses.

Why? If you have a private network, like HP did, why would you want public IPs on it?

A big reason is mergers (something HP has a lot of experience with); merging two 10/8 networks is a mess but if they have unique IPs it's easier.

Also, I think the concept of a "private network" is inflexible and in some sense a premature optimization. If you use unique IPs you can decide on a subnet or even host basis what is exposed to the Internet and what isn't.

Ok, I'm sold! Where can I get 8 /24s to use in at my (small) work? :-)

https://auctions.ipv4.global/

Same thing happen in my university. When I was graduated 10 years ago, they were using their ip block for their internal network, which can't even connect to the internet without a squid proxy. My wife did her master degree there last year and they were still doing it. I can't help but think how wasteful it is. Is there any benefit of doing that anyway, except due to legacy baggage?

MIT used to have all of 18.0.0.0/8 but they sold half of it to AWS a few years ago. Because MIT had so many IP addresses they didn't need to use a NAT. Everyone connecting over wifi got a real world 18.* IP address.

It also made network architecture easy. Every MIT building got its own B class subnet. Hilariously, even the MIT boat house which couldn't have had more than a dozen computers online at once had 65,534 addresses. It made it very easy to find out which building someone was in based on their IP.

My old uni still has two /16s. Everyone on campus gets their own “public” IP (behind the firewall, of course).

IP blocks were handed out like candy in the mid-90's and earlier. I have a /24 ("class C") personally, and know of several others who do also.

My old university has three /16s, and a bunch of other IP blocks, so gave public IPs to everyone (though they now seem to be using private IPs for some purposes). I never looked at how the public machines were allocated, but it did make picking out traffic from the residences trivial - they reverse DNS resolved to <userid>.<college.>cam.ac.uk

It’s a good future-proofing investment by AWS. IPv4 is not going away anytime soon. We’ll be dependent on IPv4 for a long time.

IPv4 scarcity only hits in certain segments. It doesn’t hit the major players.

The real problem is the hoarding by the major players. AWS will most likely never use that many IP addresses. I know that sounds like a stretch based on what we’re led to believe about running out of IPv4 addresses. I don’t know their IPv4 utilization, so it’s a guess. My guess is based on how we deploy internet facing services these days. It’s less dependent on needing lots of public IPs.

IMO it's good for the internet if few entities were to hoard most IPv4 addresses. Only while IPv4's are sufficiently available will we keep using IPv4-only networks and keep prolonging our miseries.

My ISP only gives CGNAT or is happy to rent IPv4 for a premium, one address at a time. Keeping IPv6 out helps the IPv4 margins and staves off competition.

According to the article AWS usage is at 47% of their IP pool.

You cannot use IPv4 100% efficiently. Even if you approach 90% usage, that would probably be very optimistic. AWS has probably millions of physical servers by now (they had ca. 1,4 million servers by December 2014 it seems) each with probably hundreds of services each needing an IP. Yes, you can stretch 10.0.0.0/8 with NATs all over the place but at some point, you need the public IP. With that kind of appetite you need multiple /8s of IPv4 space just to keep things running. I don't think they are crazy buying that much address space and concurrently improving IPv6 support - they most likely really need it.

> You cannot use IPv4 100% efficiently.

Can you expand on this a bit? I'm not a networking guru but it's my understanding that an IP address is an IP address, how could it be used more or less efficient?

Each time you subnet you lose three or more IP's.

A /24 has:

- .0 - network address - can't be used

- .255 - broadcast

On top of that you generally need at least 1 more IP address that is the gateway for that network.

- .1 - Usually

If you have a network with fail-over gateways generally you need to assign them individual IP's, so you end up with:

- .1 - floating IP

- .2 - router 1

- .3 - router 2

If you end up subnetting down into small subnets to give customers only let's say 16 IP's, (so a /28 (32 bits - 4 bits)) customers can only use 13 of those addresses (network/broadcast/gateway are already taken).

This gets worse as you go smaller, because each time you subnet you end up losing more IP's to the network/broadcast/gateway.

I’ve been assigned a .0 public IPv4 address by AWS before. I was a bit confused when I first saw it, but it worked just fine.

Sure, there is no problem with using a .0. For example in 10.0.0.0/23 you'd have

10.0.1.0/32

As a perfectly valid IP address.

Also, routing an entire block allows you to use all of the IP's in said block. So there are ways to do it efficiently, but with routing a block that block is not considered on-network and thus doesn't have a broadcast address nor can it use one.

That works when the subnet is /23 or larger.

I had this too. I assumed it would be fine - and it mostly was. However, we tracked a weird bug with some of the companies IoT products not being able to open a connection to the .0. Never figured out if this was a dodgy IP stack on the devices, or if it was the particular mobile carrier in that area of the world. My money would be on the latter though.

Yep, you can run into weird issues at times with devices/firewalls that are misconfigured. Saw more than 1 admin that would drop anything from a .0 as "it is a bad address and nothing from that address should cross a router".

All public traffic is NAT’d through AWS’s Blackfoot servers, EC2 instances are not directly assigned public IP address. (you won’t see the public IP in ifconfig, only the private IP) So there is little to no subnetting happening in Amazon’s public IP space, as a small cluster of machines are assigned the entirety of the DC’s public IP block.

I don't disagree with that, I was simply explaining to the asker why IPv4 can be wasteful when it comes to how IP space is allocated usually.

Consider what happens if your machine has another IP address somehow (a so-called "primary"), and then you add all of the "additional IPs" as /32s. Bingo, you can now use all of them.

The rest of the network doesn't care what they look like at certain scopes. Those devices just know to route it to the next hop, and when that next-hop is your box with a bunch of interfaces/aliases/whatever configured, it'll just handle it.

Try it, you'll like it.

(Edit: IPv4 here, hence /32. No such foolery needed with v6.)

Yes, absolutely. Unfortunately there's still some wastage as you end up having to set up a public IP for both endpoints to allow the routing to be public. (Yes, you can use private IP's for routing, but it makes diagnosing issues much more difficult). So then you end up with /30's used for point to point links, which is 4 addresses.

You can use /31's for point to point if both devices support it, but it's still hit or miss whether they do or not.

Even when getting a block of IP's from providers to a customer edge (think ISP's like Comcast or others) they tend to require using the .1 as the gateway, and the others are considered on-link and thus there is a network/broadcast address.

Your suggestion doesn't allow direct L2 communication between machines without sending traffic to the router, even if the two systems are L2 adjacent.

Even with v6 you'd want to assign multiple IP's to the same interface, except that instead of getting /128's routed to a machine you just a get a single /64 and you can use any of the addresses out of that range.

> If you end up subnetting down into small subnets to give customers

Since this is about AWS... they don't do that. They will assign a random IP out of their available public IP for every resource that needs an IP. For many of these you can allocate an IP and move it around if you want (that's why they call it 'Elastic IPs').

Unless you have special needs (and actually own the IP ranges) your VPCs will all be in a private IP range. You can even have multiple VPCs with the same range with zero issues (unless you want to peer them).

To keep things manageable and prevent any broadcasting node from reaching every system you have, you generally slice your address space into smaller networks

Each network needs at least one gateway inside that network to reach the rest of the world, so that IP address is lost. Conventionally you also lose the lowest and highest IP address in a block to the network number and broadcast address (you can free those up, but it might confuse devices or admins)

So for every split or subdivision you make, you lose at least 3 addresses.

Conventionally you also split a network at a specific bit boundary (CIDR) so each network has 2^n IP addresses in it. If you went for n=4, you have 16 addresses and already lost 3 to the above factors, so now you have 13 addresses left.

If you happened to have 14 machines attached to that network, you need a network with 2^5 = 32 IP addresses, and you've got 15 unused IP addresses with no way to give them back to a higher level.

All quite similar to why a phone area code may not be able to make use of all phone numbers inside that area, but have no way real practical way to give them to other networks.

Routing tables get bigger and bigger as the topology of the addresses stops being a good generalization of the topology of the network. As you slice and dice networks into smaller different globally assigned networks, routing tables need new entries for the smaller networks.

Pretty sure that we’ll never see routing tables with 4 billion entries. So it has to end somewhere.

https://tools.ietf.org/html/rfc3194

> AWS has probably millions of physical servers by now (they had ca. 1,4 million servers by December 2014 it seems) each with probably hundreds of services each needing an IP.

Why would any significant fraction of these servers require public IPs? And services for that matter.

I wouldn't be surprised if most of them are in private VPCs with only very few endpoints exposed.

If you are referring specifically to AWS, having servers in public subnets is actually an anti-pattern. You may want to do so with bastion hosts and a handful of more specialized services. For everything else, put them behind a load balancer. A single NLB will take one IP per availability zone and will be able to service hundreds of servers, if not more.

Then you have things like Cloudfront and the like.

Not many IPs are needed, overall, compared to the number of actual servers.

> AWS will most likely never use that many IP addresses.

Are you sure about that? A comment here said that they had 1.4 million servers in 2014, and AWS at the time was probably 1/50th of what it is now. On top of that, I think they have some IoT offerings, and who knows what they're going to offer in the future? They're basically building an internet-sized platform to run half the internet on top of it.

Every tenant in AWS has an independent address space - there could be a million 10.0.0.0/8’s inside and no one would be any the wiser.

It will be quite interesting to see how these assets will be devaluated in the future when we finally roll off IPv4 (if we ever do).

Besides the direct technical benefits of having so many addresses Amazon also potentially holds billions worth of assets that will eventually devalue and devalue quite extremely that they could write off for tax purposes.

IPv4 addresses will continue to appreciate in value until their collapse which means that Amazon might be able to write off tenfold than what they paid to acquire them in the first place.

EDIT: Looks like I was wrong when I thought IPv6 was backwards compatible

IPv6 is backwards compatible so why would we "finally roll off IPv4"? IPv6 has been "here" in production for over a decade, and has seen slow adoption. I suspect it may be another 8-10 years until IPv6 gets close to a majority, but wouldn't IPv4 still be in-use?

What would force a roll off of IPv4?

Either way, I too would like to know more about how IPv4 space has appreciated/depreciated in the past decade

In what way is it backwards compatible? It's not as if all allocations are kept and people just add 96 bits to their existing addresses. They are completely disjoint address spaces and protocols, as compatible as Tor is with the general internet: to communicate between the two, you need a proxy.

Tor operates over the Internet.

Did you mean to compare Tor to the World Wide Web?

Tor is sort of a virtual network. You need a gateway between the virtual network and the Internet to access the Internet.

Tor overlays the Internet. It can’t operate without that underlying network. That is not how IPv6 works relative to IPv4.

Yeah, that is indeed a flaw in the comparison.

I was wrong. I must have rememberd an early draft or the translation layer

So I'm not saying it's backwards compatible, exactly, but what would you call 64:ff9b::?

A reserved gateway address space?

https://en.m.wikipedia.org/wiki/NAT64

By embedding the IPv4 address space inside the IPv6 address space.

They are different protocols but they are not disjoint address spaces.

You add 96 bits to the address and boom it's converted.

IPv6 is not backwards compatible - it is a completely separate networking stack for all devices. And routers that speak only IPv4 have no idea what to do with IPv6 traffic. Similarly, IPv6-only endpoints cannot communicate with IPv4 endpoints without some sort of 6-to-4 gateway between them.

There shouldn't be any IPv4 only routers in the wild anymore. That isn't to say they aren't out there, only that anyone who has one is negligent in not updated/replacing it within the last 15 years or so. IPv6 is not a new thing, it was already live in 1996 (my memory might be a little off, but not much). Maybe you have turned off IPv6, but it should be supported and any competent IT should have a plan to turn it on (it might be a 5 year plan without any budget, but the plan should exist). Likewise you should have a plan to turn off IPv4 (if this plan has a budget it should be paid for entirely by the sale of your existing IPv4 block - this is mostly the business costs of IPv4 only customers being unable to reach you NOT the technical cost of updating all your servers)

The problem isn't supporting IPv6, the problem is often having to support both 4 and 6 simultaneously. There's just a ton of work to get a single IP stack working on infrastructure, and it's more than twice the work to get 2 working in tandem.

Is it really? In 2020? What examples are there? I worked on network switches at a major switch manufacturer and our IPv6 setup mostly just came along for the ride (or at least I thought it did - maybe our customers thought differently!)

(Genuinely interested in hearing war stories from the front on this)

Just look at the amount of engineering effort and time that has gone into projects like kubernetes: https://github.com/kubernetes/enhancements/issues/508

That's just for IPv6 support, which is beta, and took years to get there. Dual Stack is still in alpha, and that didn't appear until a really recent release (1.16).

I think routers and switches are probably the easier pieces of the puzzle, at least as far as managing them goes. The vendors have worked out all the kinks (hopefully?) before the equipment gets to you.

That's just to get the stuff to work. Then you have corporate/government policies and validation. Then you have to solve problems like "My VM resolves things to IPv6 by default, but I have no IPv6 gateway so everything times out". And then make sure that logic makes it up through your entire stack. Multicast? Not allowed on many networks.

Is it the end of the world? No. It's just a lot of extra work for everyone.

Yeah - that’s been my intuition, too. First we got software support in switches. Then OS network stacks started to support it. Then we got hardware switching chips that supported it. But the application and deployment layers just seem to be super hard. Just look at how hard it’s been to get IPv6 on AWS.

Someday, hopefully!

>IPv6 is backwards compatible

It would be really really great if true, unfortunately this is not correct.

IPv6 SHOULD have been this way, but The Powers That Be took it as an opportunity to "correct" other IPv4 issues and now we have what we have.

> IPv6 SHOULD have been this way

How would it have possibly been backwards compatible? Plenty of routers and ip-aware switches have, in hardware, specified that ips are 32-bits, so anything that added more bits would necessarily break existing hardware, and thus not be backwards compatible.

That's not to mention all the software that has similarly hardcoded the number of bytes in an ip.

How could we possibly have made ipv6 backwards compatible?

I worked on the first switch ever sold in 1995 - it didn't support IPv6, but I assure you all the engineers working on it were aware of IPv6 and assumed IPv4 would be long gone by today. For reference back then we were more concerned about how the switch handled IPX (Novell Netware) than IP. Everything else since then was designed in the era where IPv6 is coming soon enough that you ensure you can support it with a software update if you don't support it.

That isn't to say IPv6 couldn't have been done in a backward compatible way. I can think of ways to do that, and dozens of pros and cons - even though I haven't been in networking for 20 years and so I've forgotten a lot.

> ensure you can support it with a software update if you don't support it

For hardware, that doesn't sound that different from not supporting it. Convincing end-users of internet hardware to update their switch's firmware is hard. Ubiquiti has done a pretty good job of making updates actually doable, but for most hardware I doubt it'd happen more quickly than the hardware itself would fail.

> That isn't to say IPv6 couldn't have been done in a backward compatible way. I can think of ways to do that

Can you explain any of those ways?

Easy, just declare that the entire IPv4 address space is ::x.x.x.x in IPv6 and Bob’s your uncle. No idea why they didn’t do this other than sheer solipsism.

> Easy, just declare that the entire IPv4 address space is ::x.x.x.x in IPv6 and Bob’s your uncle. No idea why they didn’t do this other than sheer solipsism.

They did [0,1]:

  The "IPv4-Compatible IPv6 address" was defined to assist in the IPv6
  transition.  The format of the "IPv4-Compatible IPv6 address" is as
  follows:

  |                80 bits               | 16 |      32 bits        |
  +--------------------------------------+--------------------------+
  |0000..............................0000|0000|    IPv4 address     |
  +--------------------------------------+----+---------------------+

  Note: The IPv4 address used in the "IPv4-Compatible IPv6 address"
  must be a globally-unique IPv4 unicast address.

  The "IPv4-Compatible IPv6 address" is now deprecated because the
  current IPv6 transition mechanisms no longer use these addresses.
  New or updated implementations are not required to support this
  address type.

[0]: https://tools.ietf.org/html/rfc4291#section-2.5.5.1

[1]: https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresse...

This sounds like it works but when you get into the details it gets ugly.

http://archive.is/eHvKF

HN discussion: https://news.ycombinator.com/item?id=10854570

6to4 (2002::/16) was basically backwards-compatible IPv6. If the entire Internet had been allocated from that space, IPv6 would've been easier to deploy.

But it's technically quite ugly, and tunnelling/MTU-related problems would be more prevalent.

6to4 provides a standard way to tunnel IPv6 traffic at the perimeter over an IPv4 backbone, essentially assigning a range of IPv6 subnets to every public IPv4 address, but it doesn't provide full backward compatibility, and it still depends on proxies at the border of the IPv4 network. The applications at the endpoints need to support IPv6—so all the existing software still needs to be updated—and you still need new protocols for address assignment (SLAAC or DHCPv6) and domain resolution (AAAA records), along with any other protocols than embed IP addresses. Most of this is an inevitable consequence of extending the address space; no amount of clever tricks can make IPv4-only applications, hosts, or protocols automatically compatible with larger IP addresses.

6rd works on similar principles, but in a way that plays more nicely with ISP routing and traffic management policies. The main user-visible change is that a variable-length, ISP-specific prefix is used instead of 2002::/16.

There are a few other transition mechanisms and embeddings of the IPv4 address space, for example the ::a.b.c.d address format. They all suffer from the fact that two-way communication without a proxy requires each endpoint to be capable of representing the full address of its peer. For a node which does not have a public IPv4 address to communicate with a node which does not understand larger addresses, some third node must exist in between to perform protocol translation.

6to4 could've been released to billions of users, with no cooperation from the ISPs, just by home router manufacturers adding a few lines of code (maybe a few dozen, for health checking.)

Applications would still need to change, but apps change a lot faster than ISPs.

6to4 died because it was never the primary mode of addressing. Performance between 2002::/16 and the rest of IPv6 relied on anycast, which was unpredictable and therefore useless for real traffic.

Placing the 6to4 endpoint at the home router assumes that the home router has a public, routable IPv4 address, which doesn't really do anything to solve the problem of address exhaustion. At best it might have been a slightly superior form of NAT, but not enough of an improvement to offset the reliability and performance issues or the lack of economic incentive for ISPs (or anyone, really) to operate 6to4 anycast gateways and potentially invite non-customer, and thus non-paying, traffic to route through their network. Considered as a transition mechanism, the fact that 6to4 works better for two 6to4 LANs communicating over IPv4 (where packets can be routed directly) than between 6to4 and native IPv6 (which requires anycast proxies) was a real deal-breaker. It encourages a dependency on tunnelling via IPv4 rather than a transition to a native IPv6 internet.

6rd, fortunately, has none of these drawbacks, though it does require support from the ISPs.

The only realistic reason to get rid of IPv4 is its unreliability, if significant.

15-20 years ago IPv6 routing was often tunneled over v4, and it was noticeably less reliable than v4. Nowadays v6 is a touch faster/more reliable than v4, not very much. If that difference widens much and most users have v6 anyway, then v4 might get the axe. Otherwise... I don't think so.

FWIW I already operate some v6-only services. Not for the general audience. An SSH bastion host, a backup service, that kind of thing, not web sites for the general audience. I'm not in the least surprised if some warez topsite is v6-only already.

That is not how accounting works. You can only write off the book value (ie what you paid).

And GAAP would have to let you capitalize them. Not sure if they do.

But if they weren't allowed to capitalize them, the addresses would have already been written off at time of purchase, so the ipv6 switch wouldn't hurt them anyway

For that tax 'advantage' you'd have to buy them when they were supremely expensive before collapse, so that you had the high acquisition cost of a now-worthless asset to write off.

But you'd have paid X for a reduction of X in tax liability, which, assuming you pay a sub-100% marginal tax rate, is a strictly dumb idea.

I would bet NAT and reclaiming of unused address space will keep the status quo going for another 20 years at least. Especially since computing is migrating to a few huge data centers where almost all the communication is over their internal private network.

That’s sort of what I’ve come to believe. Imagine an internet where there are two types of IPv4 addresses assigned. The server addresses belong to cloud data providers and are provided as elastic IPs and are used to host public websites - any kind of website that you’re likely to type into an URL bar. Client addresses are owned by ISPs and are the routable NAT addresses with thousands of customers sharing them. A few client addresses are provided to ISP customers that are willing to pay a monthly fee.

It’s ugly but it’s kind of the sort of system that everyone is incentivized to move to. Which kind of makes me sad.

I bet that /8 went for more than the "average" value of an IP address. Larger contiguous blocks are more useful, and having a /8 has got to have some prestige value as well.

But on the other hand, wholesale items are rarely sold at piece prices. I'm not sure the price increased that much just by having a large block. Are there other big blocks sold for known prices?

the only recent one i'm aware of is this one: http://www.southgatearc.org/news/2020/october/sale-of-amateu...

AWS paid $108 million for this /10. That’s $25.74 per IP address.

I've talked to our resident backbone engineer about that about a year ago, and he said that larger blocks transfer for a lower value per IP, on average. I was surprised by that, but he showed me data that supported that.

Update: I just found http://www.circleid.com/posts/20180731_the_ipv4_market_2018_... which says

> As we head into the last half of 2018, small blocks are selling for just over $18/number, mid-blocks are in the $15-$18/number range, and large blocks have surpassed the $20/number threshold

so it seems there's a bit of an U shape, where mid-sized blocks have the lowest price per address.

I think this makes sense. A lot of networks need a few addresses, so /24 - /22 is probably pretty active, and you have to justify the addresses, so someone who really only needs a /24 isn't able to buy a /21 and make it work; very few people need huge swaths, but those will be able to really use them and may value having them contiguous. In the middle, it doesn't make much difference having a /16 or two /17s or four /18s, so there's a lot more flexibility, and you take the lowest price per IP if there's a block you can use.

The addressing apocalypse that never comes

Presumably there will come a point where the cost of purchasing enough ipv4 addresses outweighs the cost of making a project work with ipv6. It'll be an interesting piece of economics to see where that boundary is.

The economics (for consumer ISPs) are explored at http://www.asgard.org/documents.html

Carrier grade NAT and cell phones being IPv6 only helped stem the tide a lot.

And my cell phone connection in Germany is IPv4 only :-(

Well, Germany, they don't even have cell phone coverage in every village. And where they have it data rates can be abysmal. Yesterday the news hit the mainstream media that Nokia got a NASA deal to build 4G on the moon. Couldn't resist thinking building it in Germany would be more useful.

Disclaimer: I lived in the county over 30 years and still have strong links.

I wouldn't say it never comes. There are ISPs in (at least) Ireland and Germany that have forced customers to share IPv4 addresses recently. You lose control of your ports because of this. So your kid, for example, can't host a Minecraft server because the ISP's router doesn't let you.

Virgin Media? I know they have started issuing IPv6 addresses now and you must specify IPv4 if that’s what you need

NAT-PMP, PCP, and UPnP take care of this.

I've toyed with the idea of buying a /24 as a (probably terrible) investment that I would actually get at least some use out of in the short term (some VPSes like vultr let you bring your own IPs). Anyone have enough experience with this to talk me out of it?

With my investing luck, the world would suddenly switch to 100% IPv6 the day after the sale went through.

If that's all it takes, I'm willing to start the co-op that fronts you the money.

In a similar vein, is there a cost effective way to purchase a single ipv4 address? Just for the novelty I think it'd be cool if I could spend ~$100 to have an ipv4 address that's permanently mine.

I've got plenty I'm not using, I'll give you one.

Here, this one's an easy one to remember, you can have it:

  127.1.

Enjoy!

The minimum IPv4 allocation is a /24 because that is the smallest network that can be globally routed. Smaller networks are filtered out of BGP to stop routing tables getting too big too fast. Anything smaller than /24 has to be rented from a network provider who will advertise an aggregate route covering many customers.

AFAIK you need to demonstrate a real need to actually acquire a block, this is not like buying a domain name.

I've been thinking about this for a couple of years, because of SMTP IP reputation, ISP & colocation mobility, etc. When I first looked into it a /24 was selling for $4-5k. Now it's selling for $6-7k. See, e.g., https://auctions.ipv4.global/

AFAIU, IANA will only transfer blocks to a business entity, so if you're serious about it you might have to first start with that.

It's interesting when you look at it from this perspective. AWS and other big IPv4 owners have a financial incentive to hold back IPv6 deployment in a convenient and scalable fashion, because that would quickly wipe out billions in value of a scarce resource. It's like some people who own NFA firearms: they support keeping the regulations because a lightning link isn't worth $15k without it, it's just a cheap piece of stamped sheet metal.

Edit: it's also an easy way to outcompete others; IPv4 space has become a huge capex for starting a cloud provider.

Until one runs out of IPv4 addresses. I think it is safe to say that Amazon, Facebook, and google each have data centers big enough that IPv4 address management is a headache. I believe Facebook is internally IPv6 (that is server to server communications - their public space obviously has IPv4, and their desktop space can get by with NAT and IPv4 if they want to). Google has done enough promoting of IPv6 that I think they are the same. I'm not sure what AWS is - they sell server time so they have more need for IPv4 to the servers (google cloud has the same concerns)

Facebook only has IPv4 at the edge. Everything internal is IPv6.

suggests that ipv6 is here to replace the rfc1918 space but "the internet" largely remains ipv4.

the high v6 penetration in mobile networks likewise suggests that smartphones are generally not a part of the network as much as dedicated leaf nodes that require no inbound connections.

There's a difference between "IPv4 is only present at the edge" and "only IPv4 is present at the edge".

I think Facebook is doing IPv4+IPv6 at the edge, and IPv6-only internally.

You are correct. Facebook has done multiple presentations where they showed how they have a IPv4 load balancers/translation layer at the edge that converts everything to IPv6... but native IPv6 is well, just that, native.

There's no translation layer needed.

AWS is IPv4 internally. You know how they promote that their regions are independently isolated and to be the most redundant, you should be in 2 regions. Well, a big part of that was they were running out of IPv4 addresses in their multi-region interconnected network and they were too cheap to upgrade so they started creating every new AWS region in its own separate IPv4 network to save address space in the main one.

The artificial resource constraints created by IPv4 are a substantial barrier to entry. Prediction: we’re going to see a court case that forces a mandatory retirement of IPv4 sometime in the next 15 years.

I think that IP4 will see itself out when service degradation becomes generally noticeable. CGNAT is really painful to operate.

Remember the IE6 banners? There will be a time in ten years (haha it's always ten years!) when you'll see "Our site runs with degraded performance over IPv4. Please contact your administrator."

> CGNAT is really painful to operate

It also makes it much more difficult for customers to stress your upload bandwidth though.

You mean to say customers get a degraded experience on CGNAT.

Yup. If you can't host a webserver from your home IP, that's one less thing your ISP needs to worry about taxing their network. It's sad the incentives just aren't aligned for a decentralized internet.

>CGNAT is really painful to operate.

For whom? There are lots of providers (for example Huawei) that offer turnkey solutions for ISPs to painlessly roll out CGNAT.

CGNAT means extra memory and extra processing on network nodes. Less freedom to switch routes. Operating something extra that can only degrade performance, but not improve, is painful.

The current situation is that the ISP can lessen load on their CGNAT solutions by providing IP6.

A court case by whom against whom?

Id love to know similar stats on IPv4 space for Google Cloud and Azure.

If we insist on IPv4, at least treat those IPs as a public resource of humanity, and don't just let anyone horde them in all kinds of strange deals.

Officially, hoarding has never been allowed. You can't buy IPs unless you have a plan to use them. And there's nothing strange about a company with tens of millions of VMs buying tens of millions of addresses.

People hoarded toilet paper when Corona began with full intention to use them. People hoard food in wartime with intention to eat it.

So yes, Amazon hoard IPv4 with intention to maybe use them in the future. But hording it is, and in a non-hoard system they'd rent - not buy - IPv4 with a month notice or so.

Can someone explain why these have such value? What systems cannot handle IPV6 yet?

Like 60-70% of all internet users: https://www.google.com/intl/en/ipv6/statistics.html#tab=ipv6...

This is a little misleading. This just measures users that are using IPv6 not users that would be able to use IPv6 if IPv4 stopped working.

My local network and ISP are perfectly capable of using IPv6 but you have to call them to switch.

Zoom into 2020, and you can clearly see when the lockdowns due to the pandemic start on that chart. The other interesting artifact is that it looks like IPv6 network rollout tends to almost exclusively happen in the April-July timeframe.

Imagine that you have two ways of addressing your system. 1) Allows you to access almost 100% of your potential customers. 2) Allows you to access around 35% of your potential customers.

Which one would you choose any why?

Both.

Especially if that one with 35% of my customers provides me with lower latency, higher throughput and costs me less in CPU time/power to run my traffic across.

The rest of the people I need to eat the cost for...

I believe most home ISP doesn't support IPv6 in many countries like Canada. Thus, resulting in IPv4 still being considered as "default".

Also, with the increasing numbers of devices connected everyday, we're running out of IPv4. Think of the demand vs supply curve (demand high, supply low, result = higher price/ip)

You can check IPv6 adoption in each country here: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...

Most is incorrect. Many big ISPs support IPv6. The IPv6 charts notice more IPv4 when people are working, and IPv6 while people at home (nights and weekends) because so many ISPs do support IPv6 and it just works. The big cable ISPs and the big cell phone (not sure if all, but some at least) support IPv6 to everyone and have been doing it for a long time because it just works (they had to do some effort early to get it to work).

The cellular carriers are now doing CGNAT for IPv4, so supporting IPv6 means less traffic has to run through their CGNAT gear.

I have noticed that when looking at the internal IPv4 address of my phone. What's the technical difference between Carrier Grade NAT and "traditional" NAT?

/Not working in networking

From what I can tell it is just the power. Traditional/home NAT runs on low powered computers. CNAT is the same thing, but with very powerful computers with a lot of memory so they can handle thousands of users (possibly each with gigabit internet connections!) behind one IP address.

Comcast/xfinity uses ipv6 and ipv4 for my connection.

It also seems, that Starlink will not support IPv6 initially. At least the website and SpaceXs website are IPv4 only if that is a hint.

Correct, Starlink does not currently support IPv6.

The graph of IPv6 adoption over time on that page [1] is pretty cool because you can see the average jump up with COVID. I'd assume that means residential internet is more likely to use IPv6 over cell or work internet.

[1] https://www.google.com/intl/en/ipv6/statistics.html#tab=ipv6...

Residential and SOHO gateways usually don’t offer much configuration for ipv6, and it’s generally an afterthought in their interfaces, even if it is increasingly being used.

I would guess around half. We've bought brand new Cisco gear, which for some unholy reason didn't support IPv6. We've worked with vendors who told us that they've been supporting IPv6 for years, a decade even, but try to enable it, and you'll see that no one actually ever used it, and it doesn't work.

Amazon could perhaps do with less IPv4 addresses, if people did misuse them. I work with a client who have a public IPv4 address associated with every single EC2 instance they have, despite only 5% of them have public facing services. They just got in the habit of assigning a public IP I guess.

I do the same thing. Without IPv4 for an EC2 instance, your options are:

- No outbound internet access

- IPv6-only outbound internet access

- NAT, for an addition monthly and per-GB fee

Given you can assign a public IPv4 address at no additional cost and have everything just work, there's little reason not to have one.

So you have everything in a public subnet? That's asking for trouble.

Sure, if you have a tiny deployment you may not care (and the NAT fees may be a significant portion of that).

At some point, the NAT fees are noise - it amounts to ~ a dollar per day in us-west-2. Data processing charge is $0.045

It becomes way more valuable to ensure IT security, regulators and auditors that no, no inbound connections are allowed no matter what anyone does with the security group rules.

Also note that the AWS managed NAT gateways haven't been there forever. The option, before they were available, was to use one or more of your instances to NAT traffic. That's still available and could be an alternative, while reducing your potential footprint.

Why shouldn't someone assign a public IPv4 address to a server? The whole NAT game is just that, a game.

Also, NAT gateways cost money in AWS, so much that it is a running joke:

https://twitter.com/QuinnyPig/status/1294047698560012289

https://twitter.com/QuinnyPig/status/1293366642567651330

NAT doesn't add any additional security, Security Groups are fantastic at allowing you define your ingress/egress between instances and protecting them from harm.

All my instances get an IPv4 address an an IPv6 address by default so that there is parity. The fact that the IPv4 address still goes through some sort of NAT on AWS's side (1:1 but still NAT) kind of bothers me.

> The fact that the IPv4 address still goes through some sort of NAT on AWS's side (1:1 but still NAT) kind of bothers me.

Why does that bother you?

Cause all my services bind to a private IP on the inside. I don't see the real IP that it is receiving traffic on.

Also, if I have multiple IP's with EIP's attached so I can host multiple services (with unique IP's) I have to write automation to make sure I bind the service to the right internal private IP for the appropriate external IP address. It'd be much better if the IP address were routed directly to my EC2 instance.

> I have to write automation to make sure I bind the service to the right internal private IP for the appropriate external IP address.

Isn't that done in a more straightforward fashion by AWS loadbalancers? AWS load balancer IPs and ports on one side, listeners on the other side talking to your instances - if the instances are also in auto-scaling groups, there's zero automation needed after you set this up.

ISP is one of them around the world, some of them don't have the support for IPv6 yet.

Devices, you will be surprised a lot of devices that does not have support for IPv6.

My residential broadband connection here in Australia still doesn’t support ipv6. I email them about it every year or so to keep it on their radar, but “they have no plans at this time to support ipv6”.

So long as the internet keeps working, my isp won’t care. I set up a HE ipv6 bridge, but it adds noticeable latency whenever it’s used, for sites like YouTube and Netflix.

I wonder if we need regulation to force the transition. The move to v6 might never complete otherwise.

I'm a little surprised Netflix worked for you over HE.net's tunnel broker. I got treated as though I was using a VPN when I used it (US.)

Maybe they have one endpoint that is reachable via ipv4 only and see that, while you have a HE.net IPv6 address, you still have a proper US telco giving you an IPv4 address?

In China, all ISPs for individuals are providing a router with IPv6 support on by default. All major APPs are forced to provide IPv6 support by the government push. Let's see how the transition in China will go. https://blog.apnic.net/2019/06/06/100-by-2025-china-getting-...

My residential connection at home. "Switch to another provider" is not an option, since there are no other broadband providers near me. (and no DSL does not count as broadband)

Google Cloud does not support IPv6. There's no way to make connections to IPv6 hosts from a GCE vm.

My home internet (Verizon FiOS) is still IPv4-only.

Plenty of networking gear has trouble with it. IPv4 is just so easy to keep using and IPv6 support is often treated as an after thought. I had troubles with my Ubiquiti gateway using IPv6 and the forums often recommend just disabling it. Some features don't work correctly with IPv6 even now.

Google/Nest wifi did a good job of just making IPv6 enabled by default for all consumers.

Google/Nest Wifi took a LONG time to get IPv6 support and to support it well.

apt-get.

More precisely, the repos; When you activate IPv6 on Debian, then apt-get (the package manager) is extremely slow. This is because it first tries to reach a repo in IPv6, then after 30s falls back to IPv4. If you disable IPv4, it is lightening fast. Many services behave the same way, to the point that computers are generally faster on IPv4.

Maybe it changed recently but it wasn’t the case for the last 10 years and I’ve quit trying, and I’m not knowledgeable enough to configure the Debian system far from the defaults.

Edit: Maybe it is my ISPs who don’t support IPv6, which makes it hard to improve because the problem is invisible for, for example, Debian developers who work on IPv6 support.

Not sure about Debian, but all of the Ubuntu apt / package manager servers work quickly and reliably over IPv6 and have for some time.

You can modify /etc/gai.conf to prefer IPv4 over IPv6; cf. gai.conf(5).

Which is why the first commands to run on a Linux installation are

    sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
    sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

note that this is a runtime config. drop it somewhere in /etc/sysctl.d/ to make it permanent.

A depressingly large number of end-user internet connections come without ipv6, so if you're serving a website you need to be reachable via ipv4

What does the slash mean in the article? Where it says “... in 2018 GE sells 3.0.0.0/8” what does the “/8” mean? I guess it indicates the range of addresses?

Yep. https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing... for more info.

https://www.calculator.net/ip-subnet-calculator.html

Checkout the table on the page (scroll down)

You're right that it indicates a range of addresses, the number after the slash is the number of starting bits that the addresses all share.

Check out CIDR Blocks for a more complete answer: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing...

/8 is the same as a 255.0.0.0 subnet mask. Also known as a class A subnet. In other words, literally any IP address that starts with “3.” In other other words, 16,777,216 IP addresses. In other other other words, about half a percent of the total number of public IPv4 addresses. Crazy.

The sad thing is that's probably why IPv6 is never going to be a thing. Why would all major tech companies be willing to write off such expensive assets?

Because they routinely do? They're tech companies. Everything they own depreciates.

They can try and resist. But they will lose. For example: I've set up a IP6-only backend because it was easier than getting IP4-connectivity. And when I tried Travis and they couldn't connect to that backend, I dropped Travis immediately. That is still uncommon, but I'm sure I'm not the only one who's made a decision in that direction.

Reply on news.ycombinator.com