Whose house is of glasse, must not throw stones at another | Hacker News Reader

Whose house is of glasse, must not throw stones at another

gettys.wordpress.com

47 pointssmokinn15 comments

This article is part of a series, which examine performance problems in high-speed internet connections. Unfortunately, in addition to being split over several pages, it is written in one of the most rambling and verbose styles I've ever seen. The best single explanation is found not in the submitted link, but in < http://gettys.wordpress.com/2010/12/03/introducing-the-crimi... >

My background in networking isn't strong enough to fully understand the problem, but here's an attempt:

When using a fast Internet connection, in the 20+ Mb/s range, throughput and latency for common residential use cases are both far worse than expected[1,3,4]. The problem is masked from most users because Windows XP is missing a feature required for maximum network performance, and because most American residential Internet connections are slow. As more users upgrade their operating systems and connection, significant performance will be lost.

The problem occurs due to excessive buffering, combined with the TCP window scaling feature. When a network stack with support for TCP window scaling is allowed to connect to a fast link, the link is saturated. Normally the dropped packets should signal the stack to back off, but buffering in the stack and local switch combine to cause massive latency (on the order of 10-20 seconds) before packets begin dropping. This latency, which the author terms "bufferbloat", is responsible for the performance degradation.

The author goes on to diagnose and correct excessive buffering in various nodes; his described fixes are beyond my understanding, but indicate that altering his Linux desktop's TCP queue parameters is sufficient to correct many cases. Similar fixes to OS X and Windows can likely be applied by their respective developers, once they become aware of the problem.

The articles, in order:

[1] http://gettys.wordpress.com/2010/10/02/first-puzzle-piece/

[2] http://gettys.wordpress.com/2010/10/13/browsers-and-tcp-revi...

[3] http://gettys.wordpress.com/2010/11/29/home-router-puzzle-pi...

[4] http://gettys.wordpress.com/2010/12/02/home-router-puzzle-pi...

[5] http://gettys.wordpress.com/2010/12/03/introducing-the-crimi...

[6] http://gettys.wordpress.com/2010/12/06/whose-house-is-of-gla...

[7] http://gettys.wordpress.com/2010/12/07/bufferbloat-and-netwo...

[8] http://gettys.wordpress.com/2010/12/08/bufferbloat-mitigatio...

[9] http://gettys.wordpress.com/2010/12/09/bufferbloat-and-conge...

[10] http://gettys.wordpress.com/2010/12/13/mitigations-and-solut...

Thank you for that. Can someone post a fix here, if any, for whatever OS they know? I couldn't detect one in that monster of a post...

I've been using such a thing since about October of 2002 in linux. Its based (well it was at first anyway) on a quick little hack called "wonder-shaper".

http://lartc.org/wondershaper/

I put it on the linux box that was routing my ISDN (don't laugh) and magic happened.

I was then compelled to study the deepness that is LARTC and found much enlightenment.

The path lies at http://lartc.org

Thanks for that, it's even included in Ubuntu. I'm just not sure if it will work if I install it on all my computers individually, because I don't have a router running Linux (pity me).

Indeed. It would be nice if dd-wrt or a variant supported a wonder-shaper sort of fix..

A quick read shows that this is hardly novel. If I missed something, please point it out.

The issue is simple: broadband providers/modem manufacturers/... want to advertise the largest download speeds they possibly can. To do this, they add large buffers (such that they can always fill the pipe with bits).

However, they overdo this: by the time you've buffered a second worth of data, new data can't get through the router in a timely fashion. This means that VoIP sucks (one second latency), TCP sucks (it sends a limited amount of data before waiting for ACKs - note the recent "Google floods clients with 12 TCP packets at once" post on HN), and you may get random drops as well.

The solution is to rate-limit your outgoing internet connection to just a little less than the actual rate, and do queue management on a competent device (e.g. any unixish system should do). Prioritize VoIP, Quake, TCP ACKs, etc, and deprioritize BitTorrent/FTP/anything with the appropriate QoS bits set. Google for a guide relevant to your favourite implementation (netfilter/pf/...)

I believe the author is discussing excessive buffering in the local hardware and OS, not the ISP's. He specifically calls out Intel and some Linux drivers as known causes. Additionally, he demonstrates that changing OS buffering parameters corrects the problem in many cases (going from 1-2s latency to 30ms).

The conclusion summs that article up much better than the title: "All broadband technologies are suffering badly from bufferbloat, as are many other parts of the Internet."

It's a little over my head -- any TCP/Networking experts want to chime in?

Normally, TCP waits for the buffer to exceed the maximum segment size before sending any data. This creates serious delays when the two sides of the connection are exchanging short messages and need to receive the response before continuing. For example, the login sequence at the beginning of a telnet session begins with the short message "Login", and the session cannot make any progress until these five characters have been transmitted and the response has been received. This process can be seriously delayed by TCP's normal behavior when the message is provided to TCP in several send calls.

http://en.wikipedia.org/wiki/Transmission_Control_Protocol

Basically, there's a bunch of large TCP buffers between you and the rest of the internet, which can fill up when you're trying to transfer large amounts of data. When these buffers are full, important packets get queued up for long periods of time, causing latency. This is why running something like bittorrent causes your ping time to go up.

It's especially a problem when uploading, because upload speeds are often capped at a tiny fraction of your download speed, so it takes much longer to drain the buffers and get to the important packets.