Hacker News Reader

Comment by NelsonMinar | Hacker News Reader

I sure wish Starlink supported IPv6. They're a brand new ISP planning for millions of customers and decided from the start they couldn't get enough IPv4 addresses for everyone. Fair enough! But we're stuck with Carrier Grade NAT and it is a drag.

On Starlink it's impossible to host a server socket directly, which makes any peer to peer networking a PITA. Geocoding IP addresses doesn't work so I have to bend over backwards to convince, say, Youtube TV that I'm in the Sacramento metro and not LA where the POP is. Also the shared IP addresses seem to trip a lot of DDOS protection; I fill out 10x as many CAPTCHAs on Starlink as I do on my other ISP. And I sometimes get random network stability problems; a few weeks ago Starlink screwed something up so no one could keep a persistent connection up more than a few minutes. Seems to be fixed now, but I bet it was their CGNAT system.

I realize half the world lives with CGNAT. It's not unusable, at least web browsing works more or less. But IPv6 would solve all these problems. A little surprised that a new ISP created in 2021 wouldn't have IPv6 support as one of their launch features. There's hints they are trying to get it working but it's not an official thing now. Some discussion: https://www.reddit.com/r/Starlink/comments/tjr90n/starlink_i...

View on HN

>I fill out 10x as many CAPTCHAs

This a feature, not a bug. You can now identify a traffic light on a subconscious level 3x further away than the average driver that has an assigned IPv4.

Where average drivers hesitate upon seeing a yellow light, wondering if they have enough time to go or not, you just know that a slight uptick in speed will get you through the intersection right as that yellow flashes over to red.

The downside is every time I don't identify one of the squares containing a bicycle, somewhere a self-driving car claims another victim.

shoot! now you tell me

Wait, so that's how Elon is now training his autopilot? :)

Tesla are paying for labellers, aka Data Annotation Specialists:

https://www.tesla.com/careers/search/job/data-annotation-spe...

But, I would be surprised if they don't also use 3rd party labelled data sets.

Wasn't it just in the news that they are firing a ton of these?

Yes! That's how I actually found out they have them.

That’s a non-sequitur argument. Simply because your parent comment can identify the state of lights faster does not make them any faster at deciding whether to cross them because it doesn’t influence them to make different decisions based on the speed of their vehicle and then distance from the light.

it was a joke

This is 2022. Humor is verboten. You're cancelled.

A local ISP told me that they want to get into IPv6 as soon as possible. NATs are getting costly and moving just YouTube traffic to IPv6 would actually help a lot.

The problem is with end user devices that do not use a stable DUID and when the client hits the reset button, it changes. We are probably going to work around this by responding from the closest hop and taking the MAC into account.

Why should an unstable DUID matter? The prefix won't change.

What besides presenting the same DUID would cause you to get the same prefix again?

Could it be based on whatever physical port you’re attached to on the ISP’s network?

The host portion of the address changes. The prefix doesn’t.

The prefix delegated to you does change in many consumer ISP setups, like the upthread poster reported and I've seen in many cases in well, I think repeating your assertion doesn't really advance your argument.

Are you thinking of a scenario where the ISP customers share a single prefix? This would be contrary to all the estabilished best practices and deployment guidance for IPv6 since it wouldn't let you easily subnet. And may get the ISP in trouble with your RIR since it's just extorting customers for access to v6 addresses.

That is intentional. Stable identifiers that get transmitted over the internet lead to unavoidable tracking.

Why are NATs getting costly? Don't quite understand.

To do NAT, you need to map (external) port numbers to (internal) IP addresses. This is done using connection tracking: tracking the state of the connection and the appropriate mapping.

And connection tracking gets expensive at scale.

Yes, I understand how NAT works.

But CPU/memory is going down in cost way faster than bandwidth demand is increasing.

Regardless, its way way cheaper than buying IPv4 blocks clearly, otherwise people wouldn't be doing it.

Edit: ok, the problem isn't hardware, it's comedy license fees. https://itprice.com/juniper-price-list/cgn.html

$470k for a license to do CGNAT at 100gbit/sec. Surely these guys are opening themselves up to be replaced with some cheaper open source based software solution?

> $470k for a license to do CGNAT at 100gbit/sec. Surely these guys are opening themselves up to be replaced with some cheaper open source based software solution?

Good. CGNAT needs to die. Addressing is fundamental and customers deserve not just an address, but their own RANGE, especially now that it's feasible.

i'd love to be able to also ask for the reverse-dns zone to be delegated... if i've got a public subnet, it would be lovely to be able to use it properly.

a man can dream.

Software-defined networking is slowly becoming more popular, but it’s always going to be more resource intensive than these enterprise-grade routers that are typically implemented using FPGA / ASICs.

Having said that, I’m often equally baffled at just how expensive modern networking hardware is, but as it’s pretty much all of these carrier grade networking solutions being this expensive, I’m assuming it’s somewhat justified.

That doesn’t take away the fact that NAT just adds an expensive layer of complexity on top of it, and I can imagine that in the long term, IPv6 is starting to become much more attractive.

> I’m assuming it’s somewhat justified.

in a sense, yes. People claiming software based solutions can match performance of hardware basic ASIC's are simply not thinking about the scale and speeds of modern core routers and switches.

For instance, taken from the blog of ivan pepaljnak[0] > It’s hard to imagine how fast switching ASICs have to work – a modern data center switching ASIC can forward billions of packets per second. For example, the throughput of Broadcom Tomahawk 31 is 12.8 Tbps, and it can switch 8 billion packets per second, or 8 packets every nanosecond.

Another thing which makes routing at large scales with large traffic flows expensive is the separation of the control and data plane. most modern datacenter routers can continue forwarding traffic inside the ASIC while its control plane encounters a failure. (usually for a few 100ms to a second, after that the forwarding table will become stale, and this cannot be refreshed without a control plane).

Having a redundant control plane isn't that expensive, but it becomes harder and harder to keep this failover fast enough if your forwarding plane is pushing more and more individual traffic flows.

Then there are still other items which one can add to a modern router to make it do more but also cost more. (think about accelerated IPsec encryption, MACsec at line rate or DWDM functionality).

[0]: https://blog.ipspace.net/2022/06/data-center-switching-asic-...

Probably a bit of a cartel for "enterprise grade" networking equipment, is my guess. Was similar in the late 90s/early 00s for web/database servers.

My (uneducated) guess would be to look at the way patents last too long. So society ends up suffering, rather than benefiting from IP protection.

I’m not sure the price is justified, however the ISP market is extremely difficult/impossible to break through for startups or any company capable of building their own. It’s a self-fulfilling prophecy, the market is hard to break into (for other reasons besides networking equipment cost) so nobody who can actually do something about it is able to get in.

the cheaper solution is IPv6. if an organization is too resistant to change to implement IPv6, they're going to find themselves subject to exorbitant licencing fees in order to keep using the technology they are stuck on.

CGNAT also needs IP-port-user logging to support disclose request by law enforcement.

Not if you only allow each user 100 connections. That's 1200 bytes of ram per customer paying 100$ a month.

And you can charge them an extra $10 per month for 'pro' internet and let them have 1000 connections for 'all the family'.

That's a laughable low limit. Even the "pro" plan would be marginal for a single person without running into limits from time to time. And nevermind power users that might do something with p2p or have a couple more devices connected to the network.

But that's besides the point. Your home router can easily have millions of connections open (if they didn't skimp on the ram anyway), but if you have CGNAT boxes that do the same for tens of thousands of customers you also have to take into account that they have to move a lot of traffic. This means routing and doing NAT in software won't cut it anymore, but you need dedicated hardware coupled with very fast specialized memory to handle that traffic.

You can still do hardware NAT for the few thousand connections with the most packets and software NAT for everything else.

I bet across even an ISP network of a million users, 80% of the traffic at any point in time is within 10,000 connections.

You do realise that almost all connections are long-lived, and burst up and down in throughput? So the 10,000 “heaviest” connections right now are not the same as in, say, 3 seconds from now ?

So you propose constantly swapping in and out connections from “hardware NAT” to “software NAT”? What heuristic will you use to decide which connections go where?

Such a heuristic will probably look a lot like QoS, which is even more (much more!) resource hungry than NAT.

At which point will the obvious conclusion be, “maybe the carriers who actually deal with these problems have a point, NAT is indeed a significant amount of complexity, and let’s be happy IPv6 starts to make actual economic sense?”

How do you ensure each user is capped at 100 connections without that check incurring additional resources?

You have a per user counter. So instead of 1200 bytes it's 1201 bytes per user.

Memory isn’t the only dimension we care about.

You’ve basically proposed an absolutely horrible solution, for both the end-user and the ISP. Something tells me you haven’t actually done any actual low level network engineering, and just brush all this off as “how hard can it be”.

how are you going to keep this counter? Do you identify the bytes that are processed in individual flows? Which system will keep track of this? the control plane of the router maybe? great... you just added additional complexity instead of just pushing packets through a forwarding plane.

When an unrecognized flow shows up, punt it to software. Handle the counter there, and if it overflows then you drop the packets. No need to add anything to the control plane.

"punting it to software" from a router with seperate control and forwarding planes perspective, is forwarding it to a control plane, instead of relying on the logic programmed inside the ASIC to forward traffic.

Sorry, I meant no need to add anything to the forwarding plane, or interfere with its efficiency at all.

The point is, the really fast part doesn't need to be more complex.

The part that handles new connections needs to be marginally more complicated, but not enough that it should really matter.

Please don’t give Comcast ideas

because maintaining state for GCNAT tables is far more complex then just forwarding packets. routers doing NAT are thus more expensive then those just doing simple forwarding.

Also, in some countries ISP's need to map the use of a specific ip address to a specific subscriber for law enforcment purposes. GCNAT is no exception to this and creates a large amount of overhead because the public IPV4 prefix space is shared between multiple customers.

> because maintaining state for GCNAT tables is far more complex then just forwarding packets.

But it’s a solved problem with mature solutions, decades old. Is it really financially expensive?

compared to rolling out IPv6? definitely, especially on the longer term.

For instance, most Core/Edge routers (my experience is mainly with juniper MX series, but i assume the model is roughly the same for other vendors), you need specific licenses or interface card's to do stateful services like NAT.

Compared to doing IPv6, which is "just forwarding packets" and doesn't require the hardware to track state in nearly the same manner.

Most serious core/edge hardware vendors also do not put IPv6 behind licenses compared to CGNAT and other NAT-like features, because packet based forwarding is the most basic functionality a router should provide.

Routers which are able to do less state, also are frequently far less expensive.

You’re presenting a false dichotomy. The choice for an ISP today is not “v4 or v6”, it’s either “v4 or v4+v6”. A v6 only connection in the US is unusable.

The v4 fallback can operate on slower equipment if needed. The majority of bandwidth-heavy services support IPv6 (and the slowness will encourage outliers to migrate).

The more traffic you can get onto ipv6 the less stress is on the v4 infrastructure. Each v6 connection is one your CGNAT doesn't have to provide an ipv4 port for.

So what? That’s still just a scaling factor at that point and still requires you to have v4 cgnat infrastructure + ipv6.

You don't have to beef up your v4 infra as much though. Think 4 powerful v4 routers instead of 5 or something. If the traffic to the big streaming providers doesn't have to run through these routers, you can save a lot. Same goes for the ipv4 address space you have to rent/buy. The more connections are on ipv6, the less public ipv4 addresses you need to have.

So ipv6 support might be saving you costs already in a dual stack setting.

Take a step back to the wider context of a brand new ISP though. If you’re rushing to market like Starlink appears to be, you either implement just v4 and scale later or implement both v4/v6 up front.

Until there is a bunch of exclusive v6 stuff customers will be up in arms over missing, the answer of which thing to prioritize is obvious.

Yeah I guess it's the same as with the inter satellite communication which is promised for later, but not implemented yet so that they get at least some product out to customers. I don't think dual stack is that hard to do for entirely new networks though.

Also, one of the reasons to do satellite internet is lower latency which is a bit hurt by CGNAT infrastructure.

Last, generally brand new ISPs are in the situation that they have a hard time of getting ipv4 address space. The incumbents, especially the older ones, were around when ipv4 addesses were still plenty so they usually have way less problems with ipv4 address space. Starlink only has 166k ipv4 addresses according to https://ipinfo.io/AS14593 . Compare this to AT&T which has over a hundred million for their AS 7018 https://ipinfo.io/AS7018 alone, and there are other AS numbers they have like AS20057 with 7 million ipv4s. This roughly matches the number of AT&T customers while Starlink has more than double the number of subscribers than its number of public IPs, with growth ahead.

Having your core as v6 only lets you push NAT to limited places (one of the many options for 4x6x4 NAT, including stateless options if you're willing to cut certain corners off v4).

And v6 connections help drop the pressure on NAT resources - and sites that are optimizing for mobile connections are already going to be on IPv6 where possible (due to mobile networks prioritizing v6 traffic for various reasons, including licensing - and NAT resource costs)

CGNAT is just a slightly more fiddly version of DS-Lite (and frankly at this stage your internal network is either v6 or an ad-hoc informally-specified bug-ridden implementation of half of it). You're always going to have to do messy connection tracking stuff with connections going to v4-only sites, the only question is whether you want to do it for connections to v6-enabled sites as well or not.

All apps on iOS support DNS64 on ipv6 only network.

That doesn’t help for servers that are only reachable via ipv4 (see GitHub).

NAT64+DNS64 is specifically for IPv6-only clients to access IPv4-only servers.

A problem being solved doesn't mean the current solution is inexpensive or optimal.

The same could be said about IPv6. I think the point is that IPv6 scales better with traffic increases, to the point where switching from CGNAT to IPv6 becomes financially attractive.

What do you mean, solved problem?

> A little surprised that a new ISP created in 2021 wouldn't have IPv6 support as one of their launch features.

Worse, they had it and turned it off at some point!

They moved from behind Google to their own network, so it wasnt exactly turning it off.

Unfortunately customers tend to be happier with IPv6 turned off. There are lots of ways to misconfigure IPv6 and have it kinda work but slow and unusable. This is especially the case when you let users bring their own router.

Not really: Like was mentioned upthread, with CGNAT you end up fate sharing the reputation of a single v4 address with other customers, you get CAPTCHAs or just outright lack of service (eg instagram aggressively rate limits per IP). Not to mention worse service with apps that can use end-to-end connectivity when availabe, like video calls etc.

Many of our customers are on Starlink, and use our service to bypass the CGNAT allowing them to host web servers, SMTP servers, etc. Our service is called Hoppy Network, it provides a unique and publicly accessible IPv4 and IPv6 range over WireGuard.

https://hoppy.network

Does this work if I'm already using tailscale?

They recently added a bunch of IPv6 addresses to their GeoIP file, and announced more via BGP, I suspect its in the works. FWIW, they arent any more specifically Geo located, so this won't fix your issues.

good, what does geo location offer aside privacy invasion?

Starlink was also initially promoted for gaming, where CGNAT is terrible.

Unfortunately Sony doesn't support IPv6 either.

Without the laser links Starlink also never got the latency advantages it was supposed to have for long range gaming (like US to Europe). Instead it goes down to a basestation and then through traditional means, but that may change with the new bigger satellites if the laser part works this time.

TIL Starlink doesnot have IPv6

Waiiiiiiit, it's NAT causing the bloody captchas.

Bugger this. I've had enough of the captcha storm.

Would them using CGNAT suggest they are tunneling your traffic as it goes between your station and the ground station?

I wonder if they’ll go IPv6 once they are doing inter-satellite routing?

They could do CGNAT at each ground station, with IPs dedicated to each ground station

Starlink dishes support ipv6, I've been using it for awhile. Their stock router box does not support ipv6 though.

What's the point of Starlink?

with 4G/5G home internet, there's no real point anymore.

You likely live in a country with a population smaller than 5% of the worlds.

Many people do not have this option.

For me personally, the pathing of 5G and my broadband are too similar, so Starlink acts as a redundancy for these.

Where I live, I can try to use 4G. I get one bar of signal and when it's working I get anywhere from 0.2 to 20Mbps with 700ms (!) latency. It costs $80 for 150Gb a month.

Or I can use Starlink. I get a solid reliable signal, anywhere from 20-250 Mbps with 60ms latency. It costs $110 for unlimited bandwidth.

The real competition where I am is fixed wireless. That's 12 Mbps, 70ms latency, and $100/mo.

Decent satellite internet for people who can't get anything else?

I don't see how 4G/5G home internet existing removes the point of it.

All your criticism is valid in the long run but right now, SpaceX'es focus is to scale up, focus on usability for majority of its customers and become profitable. Removing any unnecessary feature is a must in order to reduce risk.

> SpaceX'es focus is to scale up

> Removing any unnecessary feature is a must in order to reduce risk.

You can’t scale a space based planetary ISP without IPv6, this isn’t a feature, it’s a requirement.

IPv6 amounts to a firmware update which the current hardware is and any future hardware will be capable of. The major hurdle in scaling Starlink is fast and cheap deployment of 1000s of satellite hardware. At the moment, the sole focus of SpaceX in relation to Starlink is to get their V2 satellites to orbit in order to keep up with bandwidth demands. V2 requires the Starship system which is yet to make it to orbit.

TL;DR bigger fishes to fry ATM - yes you need IPv6 to scale, no you don't need it right now.

It’s really too bad that ipv6 is only… checks notes… 26 years old now. I realize that may be an unreliable metric, so it’s roughly equivalent to 2.88 react.js lifetimes, or 3.25 vue.js lifetimes.

When the ipv6 spec was released, the latest python release did not yet support list comprehensions.

In other words, there is no reason to not support ipv6 out of the box in 2022.

> In other words, there is no reason to not support ipv6 out of the box in 2022.

i'd go even further and say that no ipv6 support means obsolescence.

> there is no reason to not support ipv6 out of the box in 2022

Use the age of a service as the metric, not the absolute year we're in. It's probably reasonable to say that there is little reason not to support IPv6 for an ISP with X years of operation. Starlink is young still.

An analogy is worldwide sales for a new laptop company. You can say that in the age of globalism, there is no reason not to ship to every continent right off the bat. But for a startup with limited cash that has lot of building blocks to lay out, it's a huge risk. They should plan for it, but only branch out when they've got a solid foundation.

A better analogy would be a new laptop company selling laptops with Windows XP.

"Our staff is more familiar with XP. We promise we're still working on the Windows 10 drivers, but in the meantime you can try to run it in VMWare."

A new ISP should implement IPv6 first and then run IPv4 on top of it like T-Mobile. They shouldn't "add" IPv6 because it should have been designed in from the beginning.

They already had IPv6 support while they were still using google cloud for connectivity (not sure why they went with google for their initial phase instead of a more traditional carrier) and when they moved to their own network they disabled IPv6 for some reason.

The funniest bit is that I'm not sure you can get normal v6 on GCP yet...

> no you don't need it right now.

Yes we do. IPv4 exhaustion is a thing.

That's what CGNAT provides a temporary solution for.

I doubt there's much overlap between the people working on IPv6 and those working on getting Starship off the ground.

It's not a function of overlap. If they've determined IPv6 isn't a priority and instead a risk, then it makes no sense to dedicate resources to it right now. It's not as if everything else about the firmware/software is wrapped up and the software team is sitting on their hands doing nothing.

CGNAT itself is a enormous risk. It's WAAAAAAAAAAAAAAY more complex and unstable than v6.

> CGNAT itself is a enormous risk.

How?

IPv6 isn't a substitution for CGNAT, it's an addition to it. You either have to keep CGNAT or replace it with dedicated IPv4 for each customer. Dedicated IPv4 is most likely too costly given the limited availability. SpaceX is also trying to cut cost aggressively.

It’s stateful.

Specifically, if a plain router stops working, BGP will route around it, and all you did was drop packets. If NAT stops working, you don’t just drop packets, you drop whole connections. Applications don’t tend to tolerate dropped connections as well as they tolerate dropped packets.

True. As an ISP you are gonna need an IPv4 stack no matter what. Even if that stack is CGNAT'd up the ass. I can't even ping news.ycombinator.com or amazon.com with IPv6.

That is the biggest problem with IPv6. Who is gonna be the first ISP to shut off their IPv4 stack? There is always gonna be some random website that is IPv4.

When all the big services become IPv6, the number of IPv4 megabits will become small.

You might just direct all the v4 traffic via a tunnel to another ISP which specializes in legacy services like IPv4, running SMTP/news servers, etc. Now you've saved all the cost of maintaining all the IPv4 peerings and config.

>When all the big services become IPv6, the number of IPv4 megabits will become small.

Well I have been hearing about the end of IPv4, and IP exhaustion for about 20 years now, and I fully expect people to still being moaning about it 20+ years from now while the majority of the interment still communicates over ipv4

Amen. I dual stacked my home network 10 years ago. 5 years ago I joined an ISP that gave me CGNATv4 and IPv6 and I opted to disable IPv6 at the router.

Why? If you have an IPv4 address, even a dynamic one, then IPv6 may not offer you a lot of practical benefit, but CGNAT-only sucks if you're at all technical.

It sucks if you're non-technical too, it's just harder for non-technical users to figure out the underlying source of any problems they have.

v6 also has better measured performance on webpage load times. Perhaps "pages load slightly slower than they could do" isn't a show-stopping problem, but faster would still be better, right?

CGNAT specifically means you can't have even temporary peer-to-peer connections, e.g. non-server multiplayer games generally won't work. And forget about trying to host anything, dynamic DNS services can't help you here. That to me is a much bigger problem than IPv4 in general being a bit slower.

It's a partial substitution. What percent of your traffic is youtube, for example?

Where did you get that information?

From Elon Musk in numerous interviews. Latest of which is with Tim Dodd.

Get a raspberry pi, set up two vlans, run wireguard, and send 100% of your uplink traffic to a remote vpn endpoint.

This gets you off their IP, and also has the added benefit of not letting them analyze your traffic.

And adds 20ms+ latency, another complex point of failure, and potential problems depending on the reputation of the IP address block of wherever you're hosting the endpoint. I've used VPNs to smooth over various Starlink problems since I got the service and it helps but it's not a great solution.

My he.net ipv6 tunnel adds about 2ms of latency on average, which I can totally live with.

At least when I was doing this, it also meant that I blocked Netflix and a few other services for my entire network.

Can you use that over a CGNAT?

Most likely you wouldn't with CGNAT, unfortunately, at least I can't see how. They need to ping your IPv4 address to set up.

As an aside, they also want to ping your IPv6 daily (at least in my logs) to keep the tunnel alive; otherwise quite stable.

I'm fortunate to not have to deal with CGNAT. But still waiting for IPv6. A he.net tunnel works for now for what I need: stable IPv6 for SSH tunneling from my IPv6 mobile.

There is no other solution that doesn’t allow SpaceX to snoop on all your traffic, unfortunately.

Having a non-residential IP address is likely to get you blocked from services like Netflix.

It would be nice if there were some way to decouple connectivity from addressing, without becoming a second-class citizen of the internet.

A good way is to refuse to give money to such services, which being unable to access them, dovetails nicely.

Incidentally this also makes torrenting safe, and everything on Netflix can be downloaded via BitTorrent.

That's an interesting point. I wonder what Starlink is doing with any DMCA complaints aimed at IP address traffic.

4K high bitrate HDR / dolby vision + dolby atmos is surprisingly a pain to find in general.

I don't use Netflix but that never happened to me.

I'd go so far as to say that it's actually never happened to anyone that doesn't use Netflix.

I can't tell if this is a 'by definition' comment or if you mean that Netflix is the only major service which blocks VPN IPs.

The latter isn't quite true, sometimes a site is having a bad day and sets up Cloudflare rules which make VPN access impractical or impossible, but it's more true than not: I can usually use Netflix off a VPN, just not consistently.

Oddly the hardest part of this right now is getting your hands on a raspberry pi!

You can get wireguard running on any consumer network appliance capable of running OpenWRT.

Why two VLAN's? Are you worried the traffic is analyzed by the ISP locally?

When I use a VPN I get significantly more CAPTCHAs.

Reply on news.ycombinator.com