IPv6 privacy addresses crashed the MIT CSAIL network

Everything unknown of is untrivial - and I'll leave it at that.

It seems to me that the presence of a "bazillion knobs" is a bug in itself. A stable system can not depend on puny humans for its operation—nor should a design be able to blame puny humans for its failure.

Absolutely that's why all network pros advise against tuning the timings for a v4 network even though in some edge cases it can help unless you really really know what you are doing.

If you are genuinely interested in answer to this question, you will like the below eBook that answers that question fabulously.

http://www.internetsociety.org/deploy360/blog/2013/12/free-e...

It's free, very high quality content, and it is written as if you co-design the new IP protocol together with the author - you start from "why" and arrive to "how".

Don't be so surprised. There can be corporate-political problems to engineers solving the problem. For example, the official marketing line is "we support feature X!", and while the engineer wants to say "that doesn't work right, turn that shit off" he can't really do that.

Or maybe the solution is to not use Juniper for ipv6. I really don't know. It's possible that Juniper's development process and goals prevented solid ipv6 support - another case where even if their engineers were aware, they couldn't do anything to help this guy.

I wouldn't be surprised if this turns out to be a MIT-environment-specific issue.

E.g. Juniper support wasn't aware of it because they assumed no one would try to use stateless configuration in what would generally be a managed environment using dhcpv6.

Normally when the big dogs for a product get called in it is to find out why their product has an issue, they are the code experts for that. This doesn't mean they are inherently experts in every protocol their product happens to be using.

That's not what I said. There are plenty of very real and very painful ways to shoot oneself in the foot with IPv6, especially in a larger kind of network that I can imagine the folks at MIT have.

But based on my experience with running the real-world IPv6 networks of large scale, there has to be a better solution than just "turning off IPv6" - even if it is a great interim band-aid after being up for 40 hours! (and I emphasize with that!)

The blog entry says: "The most serious issue is that there appear to be bridge loops occasionally being created". IS this something related to IPv6 ? Specific to IPv6 ? What's the root cause ? IPv6 with default settings in a larger net has very nonlinear behavior in the case of unreliable connectivity, and can make the situation worse.

The other issue, multicast group overload, should be solvable by making the switches blind to MLD. This will make it as "bad" as IPv4, but in todays' network size it will work fine because IPv4 works, mostly. Meantime they would need to kick their vendor's representative to have it fixed.

Also, a nit about the whole issue at hand: the "worst offender" is iOS, which will create a new privacy address everytime you disconnect and reconnect to WiFi. And if you were to "fix" that, imagine the wrath of the "privacy advocates" about this.

Privacy addresses in IPv6 are very valuable for a reason that due to a fast refresh cycle they help to uncover a lot of corner cases that would otherwise pop up with a kind of "shrug, something weird" type of RCA. And fix them before the IPv6 usage hits 50% (cf my comments on the other thread).

(From the privacy standpoint, privacy addresses are a pretty entertainingly weird construct.)

So to summarize this somewhat long-winded post: no question there's a problem. Whether it's in "IPv6 the protocol" or "IPv6 the vendor implementation" or "IPv6 the configuration" is another story. And that'd be something to understand and fix - but needs more info/context/legwork IMO.

University campus typically have similar, although a bit less dynamic pattern of usage, than conference type scenarios. We actually observed on different (not just conference, but also campus) that wireless networks in general seem to have the same curve of connection duration distribution, though with different parameters. (If you have a large WiFi net, and would like to collaborate on re-confirming or disproving this hypothesis, please let me know).

The privacy addresses refresh depends on 2 factors:

1) preferred lifetime. http://tools.ietf.org/html/rfc4941#section-3.5 for more info.

2) frequency of reconnection. Some platforms (iDevices in particular) decide to make a new the IPv6 address every time they reconnect. Great for privacy (And that's exactly how I ran the abovementioned networks, also take a look at the http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi... for the observed leave/join stats), but if the infra does not have enough leeway to accomodate the extra addresses.

In any case even then not all is lost - clear the "A" bit in the RA to turn off the SLAAC, and use DHCPv6. Poor Android clients will be off the IPv6 grid because they don't bother to implement IPv6 for the reasons left unnamed here, but the rest of the gear will work just fine, with a single, predictable address, that an admin controls.

So, again - it all depends on the local situation and there's no unsolvable problems. There are existing and production-proven solutions.

> I personally see IPv6 adoption going to a full stall if users are told that advertisers found a way to tattoo cookies on their machines.

Heh. Better not tell them about https://panopticlick.eff.org/ .

Seriously, tracking via your v6 address is a minor concern. Browsers already provide more than enough information to an advertiser to do a really good job of tracking a person as he changes his attach point to the Internet.

Let me ask you a question. (Please address this question, and not some hypothetical situation that isn't covered by this question.) If the combination of my User Agent string, screen resolution, and installed browser plugins uniquely identifies my PC, why should I be concerned that I also have a stable, globally unique IP address?

Thanks for the upvote :)

And you're right about the additional mcast addresses required by IPv6 NDP. It's called the 'solicited-node multicast address' that every host must join.

My historical guess about why multicast was chosen over broadcast for NDP was due to NBMA networks. In the 1990's NBMA networks(frame-relay) were much more common than they are today. And NDP just makes more sense than ARP over NBMA networks. This is just a guess.

Someone else suggested DOCSIS was the reason that multicast was chosen instead of broadcast. I doubt this. I'm not that familiar with DOCSIS, but I think it has a broadcast type link-layer. Also, IPv6 predates DOCSIS.

It could well turn out that multicast was the right choice for NDP once we get over the inevitable roll out problems. NBMA networks could return in 20-30 years. You never know.

That depends which part of the stack you're working on.

If you're writing web apps that receive a REMOTE_ADDR field, then the protocols look very similar; you just need to parse and store bigger values, and perhaps account for the fact that users tend to control a prefix instead of a single address.

At the BGP and packet forwarding layers, everything's the same, with bigger numbers.

The linked article relates to IPv6 over Ethernet, which is the area where IPv6 added the most new features and quirks.

Indeed, IPV6 is a possible entry point for pen testers, as plenty of admins will leave ipv6 enabled, but not bother to setup ip6tables. Plenty of software is setup to bind on "all interfaces", which includes your link local address. Fortunately, takes some effort to scan the link local address space.. But if they are based on MAC, and you can just ask for the MAC of the host via ARP...

A very astute observation!

But...

There are already quite large both enterprise and service provider networks already using IPv6. It's more in-between the "pioneers did not document the thorns they hit, so the others would not" and "the future is not evenly distributed yet, so we don't know about it" territory.

I've volunteered myself to understand which of the two and to do whatever is actionable.

A house may have multiple people living there, multiple devices, visitors, etc.

You could track a person connecting through multiple locations by using their MAC address used to assign their IP. i.e. if I am connecting with my laptop from various places, my IP address will be roughly the same, only the network prefix will change.

I use privacy addresses even while I am at home. Yes, a /64 identifies my house, it doesn't identify all of the people in it.

I find the Privacy addresses a pain when it comes to managing traffic on my network. My (mostly Apple) devices will never use their DHCP provided v6 address to connect to a public service and this means that my gateway doesn't know what the hostname is for that connection.

I ended up writing a little script to scan the "ip -6 neigh" and correlate hosts together based on MAC, then write that to a hosts file for DNSMasq to use. Hack, but it works for me.

Set your reachable and valid lifetimes to the max. Though you will still end up cycling through some of the privacy address, it's not going to be bad.

They're enabled on quite some of the Linux distros. And if someone did not know the consequences of what they were doing when they decided to play with the reachable/valid lifetimes on the prefix, they could end up with a lot of addresses.

I'd love to know more on the origin of the story to understand how we can fix the defaults in IPv6 so they do not blow people's network - but I have a vague feeling someone played with them anyway.

This was my take on it as well. Multicast was a very under deployed system, one way to fix that is to make it mandatory :-)

So pushing something no one uses, and has significantly narrow uses, and still isn't deployed publicly, into the core... How is that anything but self-serving protocol designers? How do they get away with that?

I believe their thinking process was more like:

-> Broadcast is just a special case of multicast, where every host listens to an "all hosts" multicast group;

-> Since this is a new protocol, there is no need to keep broadcast; just mandate that every host join the "all hosts" multicast group (notice that the numerically first two IPv6 multicast groups are "all hosts" and "all routers", implying they were the first ones thought of);

-> Since we will always have multicast, instead of flooding ARP-like requests to everyone, let's create many "buckets" so each ARP-like request will go only to the hosts to which it's relevant, saving traffic and processing power.

IPv6 is old. It's possible that when it was designed, global multicast was just around the corner. Designing the protocol to use multicast would have seemed natural back then.

The subnet space is 64 bits long. Collisions between pseudo-random values only become likely when you have billions of nodes, so it's not worth the effort to preemptively avoid them.

This would also require all nodes to implement the same algorithm, which means it's not incrementally deployable.

Edit: The actual RFC7217 algorithm contains a DAD_Counter field in the hash input. In the event of a collision, the counter increments, generating a new address.

I have 14.04s and OP talks about 14.04s.

Thus, a typical machine — say, an Ubuntu 14.04 workstation with the default configuration

It may be interesting, but my point was that there is nothing in this article to learn from with the exception of problematic code from a network vendor.

Yes - I am. I deployed a 4 state IPv6 overlay servicing 250k subscribers on one of the very first (fully rolled out) DOCSIS 3 HFC networks in the US. I was responsible for the security and performance of the architecture. This rollout started in 2010. The TCAM exhaustion was self inflicted based on the hints of design throughout the article and the original authors understand of MLD is, just generally, incorrect unfortunately.

I think that is a very telling point in and of itself. I've sat on the boundary between the devops / security and networking world my entire career. I know many people who are great at *nix and tooling of backend infrastructure but choose to ignore the plumbing, and complexity contained within, the planning and architecture that can make the pieces above it that much more successful. When I read an article like this there are blatant points that showcase how green the OP is with regard to networking. I happen to have an advantage in this space and the point being is that I don't claim any expertise in compiler design, never will. However - it pays to bring people in to help. While you only see the story from the OPs perspective there's an entirely different side from the vendor most likely (probably around fundamental design if I had to throw a dart).

Sure, I knew I'd get downvoted for the post - but sometimes that's a calculated position to show a point, and my point is - sometimes you need help, this article was very one-sided and rife with flaws. Not something I generally expect to read on HN - but I know there is likely a lot of that out there that I "just miss" because I know that I don't know.