Apple iOS 7 surprises as first with new multipath TCP connections | Hacker News Reader

Apple iOS 7 surprises as first with new multipath TCP connections

networkworld.com

222 pointskoenigdavidmj86 comments

That's really awesome. I'd like to know, though, whether the MPTCP is integrated into the mainstream API, or a special case for Siri. It's normal that only Siri generates MPTCP traffic since it requires both endpoint to support and enable multi-path. However, multi-path TCP is compatible with original TCP stack so I don't see a reason it can't be built into the general purpose API.

Could somebody who has developer account that can deploy to real device verify it by making another MPTCP-compliant[0] endpoint?

[0]: http://multipath-tcp.org/pmwiki.php/Users/HowToInstallMPTCP

I suspect that this is done primarily to enhance Apple services like Siri, iMessage, and the like (anything that touches the iCloud), since Apple controls both end points.

For now it's used only for Siri which I guess makes sense: other services can easily retry the connection after switching data source, it's much more of a pain with Siri as one would expect greater responsiveness.

Maybe this is why I always find my iMessages to be working, even when the office wifi is on the fritz?

I've noticed this as well, pre iOS 7.

Awesome to have. I know Glass can't even understand my speech when it loses internet connectivity so devices are becoming really sensitive to connection loss. Meanwhile on my phone my LTE is often better than my WiFi so it would be nice to have this and not have to turn off my WiFi to use it.

ProTip: when whiting out your wireshark packet captures, always remember your hexdump:

dst=17.130.16.4

    $ whois 17.130.16.4

    ...

    NetRange:       17.0.0.0 - 17.255.255.255
    CIDR:           17.0.0.0/8
    OriginAS:
    NetName:        APPLE-WWNET
    NetHandle:      NET-17-0-0-0-1
    Parent:
    NetType:        Direct Assignment
    RegDate:        1990-04-16
    Updated:        2012-04-02
    Ref:            http://whois.arin.net/rest/net/NET-17-0-0-0-1

    OrgName:        Apple Inc.
    OrgId:          APPLEC-1-Z
    Address:        20400 Stevens Creek Blvd., City Center Bldg 3
    City:           Cupertino
    StateProv:      CA
    PostalCode:     95014
    Country:        US
    RegDate:        2009-12-14
    Updated:        2011-03-08
    Ref:            http://whois.arin.net/rest/org/APPLEC-1-Z

    OrgTechHandle: ZA42-ARIN
    OrgTechName:   Apple Computer Inc
    OrgTechPhone:  +1-408-974-7777
    OrgTechEmail:  droot@apple.com
    OrgTechRef:    http://whois.arin.net/rest/poc/ZA42-ARIN

    OrgAbuseHandle: APPLE11-ARIN
    OrgAbuseName:   Apple Abuse
    OrgAbusePhone:  +1-408-974-7777
    OrgAbuseEmail:  abuse@apple.com
    OrgAbuseRef:    http://whois.arin.net/rest/poc/APPLE11-ARIN

    RTechHandle: ZA42-ARIN
    RTechName:   Apple Computer Inc
    RTechPhone:  +1-408-974-7777
    RTechEmail:  droot@apple.com
    RTechRef:    http://whois.arin.net/rest/poc/ZA42-ARIN


    ...

EDIT:

also

    subject=/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=California/businessCategory=Private Organization/serialNumber=C0806592/C=US/postalCode=95014/ST=California/L=Cupertino/street=1 Infinite Loop/O=Apple Inc./OU=Siri/CN=guzzoni.apple.com
    issuer=/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA

I don't understand. What does dst being apple's IP imply? It's connecting to Siri server which is supposed to be an Apple's server. Am I missing something?

Yea, i'm not sure why they bothered to try to obfuscate the dst address at all, the article mentions siri...

Haha.. good one.

(They redacted IPs at the top of the screenshot, but they forgot to redact the hexdump at the bottom that just happens to contain same IPs).

I'm not sure if MPTCP is great or if it's an over-engineering that stuffs brand new abstraction into the socket layer.

See, previously when open a connection with an unbound socket, it would always bind it to the local IP that points at the default router. If you had multiple interfaces and you wanted a specific local IP, you'd bind to it explicitly before connecting. But it has always been that a connected socket is bound to exactly one IP. If you want to use two IPs for connecting out, you will have two sockets.

With MPTCP the socket can now end up being bound to two or more local IPs. So effectively it changes the abstraction of the socket model to be more of a collection of (conventional) sockets. Doesn't it sound like something that could be done in the userspace? Sure, it does :)

Though on the other hand, networking code in the kernel has access to a lot more information about the state of the network, so it can make better decisions and quicker. For example, if there's an interface with no default route going through it, the userspace solution may still try and use its IP to try and connect out. The userspace will also need to monitor the routing table for changes and re-try connecting out in response ... so MPTCP makes more sense if you think about it, but a userspace solution is much easier to implement and deploy.

SCTP has multihoming and is exposed via standard socket API. So, it's nothing new.

They have a handle on this stuff:

https://datatracker.ietf.org/doc/rfc6897/?include_text=1

Basically if you bind an address, MP is disabled. If not, then MP will happen by default. If you had code that was doing MP with multiple connections before, it was binding interfaces on multiple sockets and will still work. The idea is that MPTCP is desirable and should be used if possible. Also you get new sockopts for handling the MPTCP stuff like TCP_MULTIPATH_ENABLE and TCP_MULTIPATH_ADD if you want to control the MP stuff in MPTCP.

I'm not a networking expert, but does this require system changes/support on the other end of the connection?

No, I don't think so. Everything seems to imply that no modification is needed at the application layer.

Additionally, one of the researchers goals behind the technology is to support unmodified applications. Here's a presentation from the team's website http://multipath-tcp.org/data/MultipathTCP-netsys.pdf

It will, however, require changes at the kernel level.

apllications use just the socket api for their interactions with the n/w stack. there is no change there, and hence the "unmodified applications" part.

from the mptcp paper, we have the following :

In brief, here is how MPTCP works. MPTCP is nego- tiated via new TCP options in SYN packets, and the end- points exchange connection identifiers; these are used later to add new paths—subflows—to an existing con- nection. Subflows resemble TCP flows on the wire, but they all share a single send and receive buffer at the end- points. MPTCP uses per subflow sequence numbers to detect losses and drive retransmissions, and connection- level sequence numbers to allow reordering at the re- ceiver. Connection-level acknowledgements are used to implement proper flow control.

so yes, both endpoints need to support mptcp for this to work.

Here is a video where they show it running, droping wifi and network connections.

http://www.youtube.com/watch?v=VWN0ctPi5cw

If iOS is exposing the API, I believe it should at least provide a parameter whether it should be a MPTCP or a classic TCP. Sometimes you don't need that reliable trasmission; using MPTCP would cause unnecessary cellular data usage.

Multipath TCP could be used for smoothly handing over from the slower, more expensive, less-reliable cellular system, to the faster, cheaper, more reliable WiFi.

Without multipath TCP, you have to drop and restart each connection, which in some applications can cause hiccups, latency, or even complete loss of state.

Of course, all of this depends on server-side MPTCP support as well, which most systems don't ship with out of the box.

Right. What I'm saying is that, by keeping alive both path, you need to use some traffic. Although when there's WiFi available, you are primarily using WiFi, there are traffic going through cellular network in order to keep the path alive.

would be really cool if slower / more expensive connections could ultimately be drained, and the whole data transfer just switches over to a single more faster connection.

edit-1: note that this is not the same as subflow 'deletion' that would happen if an access mechanism was not available at all.

edit-2: section-3.4 of the mptcp paper indicates that FIN has a more limited 'no more data on this subflow' semantics. so, in theory it should be possible to do just that. would be so much better to not go the ANDSF route at all ;)

Hey! I didn't go into that deep! So if FIN is used in a subflow, does it terminate that path? Is it possible to wake this path up again?

the shorter answer first: i don't think there is any possibility of resurrecting a subflow once it is gone.

longer one: with mptcp, there needs to be some way to distinguish connection-teardown vs subflow-teardown. with, RST the subflow is terminated. FIN is more subtle (since it occupies the sequence space in normal tcp) however. FIN is handled via an explicit DATA-FIN within TCP option to indicate the end of data-sequence-space (which maps subflow sequence numbers to data sequence numbers in TCP options). gracefully terminating a connection, thus involves sending DATA-FIN on all subflows together with a FIN.

Cellular mobile units are likely to have a short dwell time on any particular WiFi if they are in a moving vehicle. Can the underlying connections be initialized quickly enough for this case?

> to the faster, cheaper, more reliable WiFi

This is far from universally true and a wonderful illustration of why multipath is great: I frequently see wifi where the back link is down or behind a payment gateway, requiring me to turn off wifi to get a reliable connection. This is sadly common at conferences where the venue skimped on AP placement and left entire rooms at max range or with inadequate backhaul.

The true beauty of MPTCP is that you don't need to make assumptions and hope — instead we can let the kernel sort things out based on the current environment.

Yes, both endpoints will need to support it. Support for MPTCP is advertised through a TCP option during the setup phase of the connection.

One of three things can happen:

- Both endpoints support MPTCP and announce it through the MPTCP option : MPTCP will be used. Subflows will be set-up if possible (i.e. if additional addresses are present on at least one of the endpoint)

- One of the endpoints (say an ios7 device) supports it, but the other (say a webserver) doesn't : the server will ignore the MPTCP option (which it doesn't understand) and a normal, single path TCP connection will be used.

- None of the endpoints support it, thus no MPTCP option will be sent, normal TCP will be used.

Perhaps this is why (according to the article) the usage in iOS is limited to Siri and navigation.

From what I read one of the main design decisions was to make sure the nodes along the route do not drop the packets which means it will be a TCP packet for all intents and purposes. So why should the other end change in this case?

It's less about routing and more about connections. Typically, your client makes a single TCP connection for HTTP communication, all data requested on that connection is communicated on that connection. With a supporting server, multiple TCP connections can be used for the same HTTP connection. This doesn't help with transfer speed, but it improves TCP's ability to route around failure. One of your multiple TCP connections goes out, and another one (presumably on a different route) continues handling data transfer. With regular old TCP, the connection tends to fail and has to be re-established.

Pointless Pontification: since TCP was designed to route around failure, multi-connection TCP sounds to me like an adaptation of technology to the "demands of business." What I mean is network operators seem to want to centralize their networks to reduce costs instead of distributing them to make them robust, thus the technologists are attempting to work around this "failure."

It doesn't make sure the nodes along the route do not drop the packets. Nodes along the route would see those packets as normal TCP packets (which is by design because in this way routers in between don't have to change). But both end are aware of MPTCP and they may use multiple path.

In this case, iOS is connecting to Siri server using both 4G and WiFi. They apparently go through different routers. This helps improve reliability when the device is at border of a WiFi coverage and is switching between 4G and WiFi back-and-forth. Using two path simultaneously would allow Siri to switch to the other route when one of them is showing significant packet loss.

When MPTCP is established, which IP address would be reported to the server software?

The first one.

So does this mean if I'm connected an adhoc eyefi wifi network and want to share whatever pic I took over the internet, I don't have to turn off wifi to do it?

If so, this is very awesome. It makes things like the Sony QX camera probably x10 more seamless.

That's basically how AirDrop works on recent Macs and recent iOS devices and why they require more recent wifi hardware to make it happen than the OS supports generally.

Cool. So it's like http://mosh.mit.edu/ mosh for TCP

Does the phone actually send the TCP packets multiple times? Then it would certainly put more stress on the network. Does anybody know how fairness compared to non-multipath TCP stacks is achieved?

They discuss the fairness attributes in their NSDI '11 paper: http://static.usenix.org/event/nsdi11/tech/full_papers/Wisch...

It's an interesting read, and it will explain it better than I can.

Multipath TCP sends packets over multiple paths; multiple different connections, such as cellular and WiFi. It doesn't repeat the packets, unless one of them is dropped. It can be used for several purposes; aggregating bandwidth between the paths, taking advantage of the bandwidth of both connections. It can be used for reliability. And it can be used for transparent handover of connections, where you may walk out of your house and switch over from WiFi to cellular seamlessly.

LWN has a good article on it: https://lwn.net/Articles/544399/

Am I correct in assuming that MPTCP could potentially make snooping on in-transit data somewhat more troublesome? All potential pathways would need to be monitored simultaneously in order to reconstruct the application-layer stream.

On the other hand, all cross-Atlantic traffic is likely to pass through the same few undersea cables, regardless of the "first 50 miles" are covered (3G/WiFi), so the net effect on snoop-ability is probably null.

> On the other hand, all cross-Atlantic traffic is likely to pass through the same few undersea cables, regardless of the "first 50 miles" are covered (3G/WiFi), so the net effect on snoop-ability is probably null.

Wouldn't the traffic be split over multiple IPs (not all of which might be known), and wouldn't latency sometimes cause them to be out of sequence? It's definitely not as simple as "monitor this singular connection", though of course it's still feasible.

Indeed, on the surface, the multiple IPs would increase the amount of obscurity. Can anyone familiar with the protocol clarify if there are any other header elements which reveal that the packets belong to the same abstract TCP connection?

What if you do not want to use your cellular connection when wifi is enabled. Will you have to start disabling cellular?

That is in theory exactly what this allows you to do. e.g., if you're watching some streaming video while outside, then come home to wifi, the connection will switch to wifi.

It gives you the potential to switch but it isn't explained in the article how iOS7 uses this potential.

Does it automatically kill the cellular data connection and only use the Wifi if wifi is available? Or does it keep both open and use them both?

As far as I understand, it uses the least congested path available. In theory, this would mean it would prefer the wifi connection.

Ah, that is my misunderstanding. I thought it was some sort of "split traffic between each path".

But I guess for it to work it would still need to check the cellular connection every now and then. I wonder how much traffic that would generate? (hopefully a negligable amount.)

> As far as I understand, it uses the least congested path available.

IOW, it uses at least some cellular data even when connected to WiFi, because otherwise it wouldn't be able to determine which path is the least congested.

Does this mean that if all end points support this, connections could be split all over the place?

A 10 IP round robin setup, if I'm correct, could show stats for 10 different locations. Or does it work differently?

All 10 IPs need to be bind on your local NIC(s). If you can get those 10 IP(s), they are mostly registered to your location already I guess?

"First" apart from the three implementations which were released before it - including one for linux. So that'd be fourth then...

First implementation not to be used for testing purposes?

Developers running a modified Linux kernel does not a deployment of a technology make.

an enabler for ad-hoc, peer-to-peer wireless networks by providing the additional resiliency of tcp apps simultaneously leveraging (n) network interfaces and distributing traffic?

Does this mean that I won't drop connections when I leave Wifi range and switch to 4g, which happens literally EVERY TIME I LEAVE WORK OR HOME? :)

Would be great if FaceTime Audio can take advantage of MPTCP for seamless handoffs.

Another packet to chip away at our Tiered Data Packages.

Android doesn't even support adhoc wifi.

It supports acting as a wifi hotspot. I've never tried it, but does that not work for local area networking with data off?

Also: http://multipath-tcp.org/pmwiki.php/Users/Android

The nice thing about Android is you get the source. If you want, you can add MultiPath TCP to Android.

I'd still love ad-hoc better. HotSpot allows you to share 4G through WiFi, but it doesn't scale. The hotspot because AP, and that's the center of the network. On contrast, ad-hoc can scale to thousands of nodes across miles by using multiple hops. Also, ad-hoc is a more distributed model.

I never understood why ad-hoc mode was disabled by driver. AP mode (hotspot) is still available; users can still share their 4G connection without being approved by their carrier.

Ad-hoc networking on wifi generally only supports point-to-point links, not routing, unless you use another routing protocol on top of it like BATMAN Advanced.

I'm not sure why connecting to an ad-hoc network is disabled in Android, but it's certainly never been anything that's caused me problems.

Right. But I supposed by hotspot/ad-hoc, we were talking about the link layer possibilities only. hotspot is also about MAC layer only as well. IP runs on top of that. However, the difference is that, in AP/client mode, all nodes need to associate with and be able to talk to an AP; while in ad-hoc mode, every node is equivalent and they can be as distributed as they want. For ad-hoc routing, there're olsrd, batman, etc. They are really cool :-)

We used to use Android devices to test different scenarios of ad-hoc applications. But none of current mainstream devices support ad-hoc mode any more. On contrast, iOS devices can still join (although not create) an Ad-Hoc network, which is ... ironic.

And what of it? Here's another unrelated whinge: no Apple stuff supports WPS even though it's highly useful for getting your device on the network quickly.

Way to find something negative to say when this MPTCP deployment is overwhelmingly positive no matter what you think of Apple vs Android.

From the wikipedia page:

> WPS has been shown to easily fall to brute-force attacks.

(along with citation).

Sounds like it's a good thing Apple doesn't support it.

Actually, the AP fails, and most of the vendors fixed the brute-force attack by throttling the amount of tries.

Not implementing it on the client side has nothing to do with this security issue, but some other reason.

So this is an application level approach to the same things as Mobile IP, MPPP, etc?

Does this means it will allow me to use both Comcast and Uverse at the same time?

First? OpenBSD and the Linux kernel have had this integrated for years.

The article says "first large-scale use."

Integrated? I don't know about OpenBSD, but in Linux it is not in mainline. There is an experimental fork of the kernel supporting it, but it hasn't been proposed for merge upstream yet.

Sure, there has been development going on for years, but there are no major distros you can download and use it out of the box.

The point is, this is the first shipping product using it in the wild, as opposed to experimental implementations used in research and development.

Note that this is not multipath routing (which takes place at the IP layer, and requires involvement in routing protocols that end devices don't usually have access to), nor is it channel bonding (which occurs at the ethernet layer).

I stand corrected, is anyone trying to merge it?

EDIT: No I don't.

Not sure, there's no indication that I could find of a roadmap to merge on the Linux MultiPath TCP web site: http://multipath-tcp.org/

“The current implementation of Multipath TCP in the Linux kernel supports RFC6824 and includes this fallback mechanism.” - [About Measures][0]

0: http://multipath-tcp.org/pmwiki.php/Users/AboutMeasures

Please don't say things so scary without looking into it.

When they say that, they mean in their fork of the Linux kernel: https://github.com/multipath-tcp/mptcp not in the official upstream kernel from Linus https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.... nor any of the mainstream distro kernels.

You can check this for yourself. Just take a look for the MPTCP patches in Linus's kernel. You won't find them. Here's my kernel git repo; origin is Linus's repo, mptcp is the MPTCP development repo:

  $ git log --pretty=oneline origin/master..mptcp/mptcp_v0.87 | wc -l
       2841

As you can see, there are 2841 patches in the mptcp_v0.87 branch that are not in the kernel repo, all the way back to the original shim6 patch that was used to start building MPTCP on top of shim6, which was a proposal for multi-homed IPv6 (MPTCP now works on top of IPv4 as well).

Note that when you look at the "installing on Linux" instructions, they mention running it in a virtual machine.

This is not terribly uncommon for an ambitious research effort; work on an out of tree fork for a while, and then when it's ready, propose it for inclusion into the upstream kernel. For a recent example, see Bcache http://bcache.evilpiepirate.org/, which was developed out of tree for several years before being merged a few months ago.

> Please don't say things so scary without looking into it.

Why do you find this scary?

And why do you think that I didn't look into this?

Ahh, the way you said it made it sound like the first bit was you talking out your ass, assuming that they hadn't merged any of their work by a few pages on their site.

I am currently on a very restricted link, so I was not able to check the commit logs.

I think the "first" means being used in a consumer device. 802.11ac was there for a while. But none of big companies was making 802.11ac enabled laptop until Apple. Same thing here. It's not about who at when implemented it first; the point is this time it's being applied to a popular consumer device, even though it's just got approved by IETF (Jan.). This is how new technology gets involved into people's daily lives.

I don't think it's in OpenBSD at all, there is a monster of a patch (experimental) for FreeBSD though:

http://caia.swin.edu.au/urp/newtcp/mptcp/tools.html

Yep, the title is inaccurate. On other sites I've seen "First commercial MPTCP implementation" :)

I wonder what Apple is using on the server side that supports MPTCP?

They seem to be using Citrix Netscaler, which has supported MPTCP for a little while:

http://blogs.citrix.com/2013/05/28/maximize-mobile-user-expe...

Disclaimer: I'm one of the authors of MPTCP, but I don't have any inside information from Apple, beyond what nmap tells us.

is this a good way to minimize the likelihood of MITM attacks?

Probably not. A dedicated MITM attacker could simply strip off the MultiPath TCP options from the initial connection, so it would act merely as a regular TCP connection in that case.

Good crypto is a much better solution to minimizing the likelihood of a MITM attack. MultiPath TCP is design to provide speed, reliability, and mobility of TCP connections.

Except it already existed in the Linux kernel for years, or am I missing something? The article even says so itself.

> am I missing something?

The first phrase of the article maybe?

> Apple's iOS 7 is the first large-scale use of a newly-minted Internet protocol, called multipath TCP.

(emphasis mine)

The linux implementation is a third-party module with, as far as I can tell, little penetration so far. And the article linked here seems to have been partially cribbed from said module's implementors (the wireshark screenshot was lifted straight): http://perso.uclouvain.be/olivier.bonaventure/blog/html/2013...

Android runs Linux. Thanks for the unnecessary downvote. Want me to lookup the charts for you? Also there are millions of people running Linux (about 1.3% of all computer users if I remember correctly).

> Android runs Linux.

For the second time, since you apparently missed it the first time around, MPTCP is a third-party module, it is not part of the mainline kernel, it requires using the developers's kernel fork. Do you have any evidence that Android is doing so?[0] Then Android running Linux is irrelevant.

> Thanks for the unnecessary downvote.

I didn't downvote you.

> Want me to lookup the charts for you?

No, I'd like you to read what people write.

> Also there are millions of people running Linux (about 1.3% of all computer users if I remember correctly).

That's completely irrelevant to the thread because they are not running an MPTCP-patched kernel.

[0] I'll help you, the MPTCP project has a page pointing out to Android ports of the MPTCP module: http://multipath-tcp.org/pmwiki.php/Users/Android which include a rebuilt kernel to include MPTCP as well as patched connection manager, network daemon and iproute2. Doesn't sound like base Android supports MPTCP to me.

> MPTCP is a third-party module, it is not part of the mainline kernel, it requires using the developers's kernel fork.

Oh I get it now, that explains. I stand corrected. Sorry for being so bitchy about it, I just thought they were saying Linux was not widely used and hyping iOS7 instead.

Reply on news.ycombinator.com