Beyond TCP: The evolution of Internet transport protocols | Hacker News Reader

Beyond TCP: The evolution of Internet transport protocols

slideshare.net

71 pointsjsnell23 comments

One of the major problems with TCP often not mentioned is bufferbloat. TCP's common congestion avoidance algorithms usually have no precautions against bufferbloat and frequently induce it, i.e. when probing the congestion window. And those TCP congestion avoidance algorithms which do manage bufferbloat are often not deployed in public for fear of losing out to greedy congestion algorithms.

QUIC uses fundamentally the same basic congestion avoidance algorithm as TCP (QUIC's algorithm is a work in progress AFAIK) so even QUIC is still in the same bloat.

The problem eventually bubbles up the stack and affects most web applications, so that a single user, just using a web application on one machine, can break the Internet for all other users on the same LAN.

Try this demo for yourself:

1. Run "ping google.com" from another computer on your LAN.

2. Upload a 10-20MB file via Gmail or Dropbox from your computer.

3. Watch the ping times on the other computer skyrocket from around 100ms to upwards of 5-10 seconds.

4. Try a Google search from any other computer on your LAN while this is happening.

Web applications which use protocols such as WebSockets have no way to reduce the bufferbloat footprint of their application, other than re-implement their own delay sensitive congestion avoidance algorithm on top. And actually, if you want to build a robust application which does any uploading (or plenty of downloading), this is what you need to do.

For example, to prevent inducing bufferbloat, Apple's software update actually uses a variant of LEDBAT (the delay sensitive congestion avoidance algorithm from BitTorrent's protocol) when downloading software updates.

Shouldn't RED type algorithms in routers prevent buffer bloat?

It may help for other users on the LAN but it's going to be years before it's fully deployed. It's not the solution for today.

In any event, I don't think it's going to make any inroads towards fixing bufferbloat for a single user on the same machine, i.e. where multiple applications are creating several seconds of bufferbloat each, the latency sensitive applications (Skype, Google searches, typical web browsing) are still all going to suffer.

I think it's actually not a router issue but an application issue, in the same sense that CPU overusage by one application is an application issue. Applications just need to be better designed to not bloat a user's network.

Hopefully, the next major Apple OS update will include bufferbloat measurements in Activity Monitor, so offending apps can be named and shamed. But even this won't prevent web apps from having massive bufferbloat footprints (for that the browsers will need to share data with Activity Monitor, or highlight this to the user).

After this, I think pressure from users on application developers to reduce bufferbloat footprint and build network efficient apps is the way forward.

Then we will be able to switch to some of the better higher-bandwidth TCP congestion avoidance algorithms without having to settle for algorithms which ignore latency as a congestion signal (packet loss is a poor signal and causes throughput to saw-tooth).

Maybe I misunderstood your first message about 'buffer bloat'. I was thinking about buffer bloat in routers/switches when many TCP connection start to send packet, router starts to buffer them, once packets are dropped, congestion avoidance in TCP connections is activated and synchronises - bandwidth goes down and once again increases in all TCP connections, and this cycle (more packets are sent/packet drops/congestion avoidance triggered) repeats again.

Yes that's right. I meant the same. Each application on a user's machine contributes to bufferbloat generated by that machine, and bufferbloat as a whole encountered at the router (which affects everyone on the LAN).

Note that bufferbloat is a problem regardless of there being many or few TCP connections.

The Codel paper explains why not (Codel creator is the same as RED). RED needs fine tuning and people couldn't do it right. Codel has only one knob with a sensible default. This should simplify implementation and allow widespread adoption.

"QUIC uses fundamentally the same basic congestion avoidance algorithm as TCP (QUIC's algorithm is a work in progress AFAIK) so even QUIC is still in the same bloat."

This is one of the many disappointing aspects of QUIC apart from the whole focus on just being a replacement option for HTTP(/2). Which means being constrained by many of the problems that have plagued the web (browser) model for so long.

And in the case of congestion avoidance, they've decided to just follow the TCP model despite many better techniques available in the literature. E.g. the protocol that I'm working on (see my profile) tries to figure out the buffer bloat case by tracking rtt evolution during a session. This sort of technique is not easy to do in the TCP's lock-step ack based model. But it gets easier to do if you replace lock-step acks with periodic updates from the receiver side combined with special ack packets to measure rtt more accurately.

There's an opportunity available today to build a whole new networking stack from the ground up for the new (mostly mobile) use cases. The design space opens up amazingly once you look beyond the web browser as the ultimate client. I'm about to board a very long flight but I'll be happy to engage in further discussion to whoever connects directly.

Linux now uses fqcodel as default queue manager. It solves bufferbloat.

It won't help. To serve its purpose, any AQM (active queue management) like fqcodel must be done at the congestion point.

The end-user computer has a faster link than its internet connection upload speed, typically a gig ethernet whereas a xDSL upload speed is a few megabits/s.

If you transfer data to a random website (upload) from the computer, packets will accumulate inside router/modem tx queue, because this is the slowest link between the two hops. This is where it's important to have an AQM running to reorder/drop packets in that queue.

If the end-user's router has firmware with AQM, would that solve the buffer bloat?

No, not really, see this comment: https://news.ycombinator.com/item?id=10546875

To clarify, I meant that fq_codel solves bufferbloat. Its wide adoption will solve the problem globally and being present at Linux will help that, as Linux is the kernel in several home network appliances.

Other AQMs could solve it, if properly implemented, but fq_codel needs no tweaking.

CORRECTION:

fq_codel is not default in Linux, but is default in some distributions, like Fedora.

Is there a recording? Slides only supply maybe 50% of the information needed to form the picture. If the accompanying spoken words are left out then it is difficult to receive the message as intended.

There are no recordings, I asked.

There's this presentation at the IETF: https://www.youtube.com/watch?v=Wp0Kr3B64tA

These are not the same slides, but it should tell you all you want to know about Multipath TCP.

It really annoys me that some routing systems drop packets that use any protocol other than TCP/UDP. It makes innovation of new protocols basically impossible.

Do you have any more info on this? What protocols are being dropped? And which routing systems are dropping them Just curious

Many firewalls filter out non-TCP/non-UDP traffic as a matter of policy. Heck, network professionals are having to be reminded NOT to block all ICMPv6 on their border (like they do for v4).

This generally occurs far enough down the stream that it's only a matter of bopping a clueless senior netop on the head. Nothing upstream is going to block arbitrary packet types. Could you imagine if backbone peers were dropping packets like that? The internet "routes around the damage" and this would be an application of that. At the social level, it would involve people within an org saying "fix this".

Pretty much every router/modem people have at home does NAT, and does so for UDP, TCP, ICMP and some do GRE and that's it. And they're not getting updated to NAT any new protocol directly over IP that people come up with.

See slide 22.

I wonder why SCTP over UDP was not mentioned. This solves the problem of routers dropping unrecognized protocols.

I wish for a HN URL replacement algorithm that replaces decks with talk videos.

Reply on news.ycombinator.com