Google fixes nearly decade-old Linux kernel TCP bug | Hacker News Reader

Google fixes nearly decade-old Linux kernel TCP bug

bitsup.blogspot.com

355 pointstim_sw46 comments

I would love to see a characterization of this bug's impact. Reading the description, it doesn't seem to be nearly as serious as the author of this post is making it out to be. But there's no information about how fast the client will step up the transmit rate after a period of quiescence and no data from traffic analysis to determine how frequently real-world servers encounter this condition and the overall effect in terms of dropped packets, increased queue depths, and latency spikes.

(Google are actually in a good position to provide these data; a number of Google's own services will doubtless benefit. I suspect the scale of Google's operations means that even a barely-measurable decrease in latency or queue depth compounds to a big gain pretty quickly.)

Strictly speaking, it can't be too bad, since it would immediately stand out if Linux boxes have decreased network performance in a mixed shop. Maybe a few percent in the average case, with maybe some specialized applications more affected.

I'm confused by the last sentence:

> The whole web, including Firefox users, will benefit.

Why is Firefox thrown in here? And the post is even tagged with "firefox". But the contents of the post seems to have absolutely nothing to do with Firefox, except inasmuch as Firefox is an application that interacts with the web.

The author works for mozilla, and firefox is a common theme on his blog. I think it's for regular readers of his blog to understand the practical impact (readers he assumes care more than average about firefox).

I'd guess that because it was a bugfix by google - people might assume it was something to do with chrome or android, when it should help any webuser that talks to a linux based server, including firefox users.

The tag is probably so that the post will show up in Planet Mozilla (http://planet.mozilla.org).

I often tag non-Mozilla posts as "mozilla" if they are still relevant to Planet's scope on my blog too. I've submitted a category feed instead of the full RSS blog feed so that I can exclude things which are physicsy or otherwise irrelevant.

The commit also says "This particularly shows up when slow_start_after_idle is disabled as a dangerous cwnd inflation (1.5 x RTT) after few seconds of idle time." And the slow_start_after_idle sysctl is enabled by default.

The bug fix is in the CUBIC congestion control algorithm, which has been the default in Linux TCP since kernel 2.6.19 (released in November 2007). So impact in distros has been ~ 2008-2015.

Remember that TCP is robust enough that it seems to work anyhow - even at the cost of reduced network throughput in this case.

That to me sounds like this is not really a bug, but a performance enhancement; a bug would be something like corrupted/missing/extra data at the application level. Tuning the TCP algorithms beyond the basics is really more of an art.

It's a bug if you consider performance to be a feature.

Presumably it is considered to be a bug because it does not accomplish what the code is meant to accomplish - namely, the code does not properly characterize a congested state and thus does not operate as if it were congested.

Given your definition, and viewed from a high enough level, every feature or enhancement is a bug fix. I think nitpicking the wording (while an interesting philosophical discussion) detracts from the point of the article: an unnamed Google engineer waded into the Linux kernel code and made the web slightly better for many people.

I think it was actually multiple people.

You've hit on a fine point I often make about how the term "security bug" is used.

I would call this a bug fix since someone has found what is probably an unintended behavior in the cubic algorithm. The cubic algorithm itself though would be a feature improving on the original tcp algorithm (nagle, if I remember right)

No a feature or enhancement mostly introduces something new, like a new algorithm for cubic congestion control, which this patch of course doesn't to, it just fixes the old one. ;)

I thought that too, but then

> that an endpoint that is not moving any traffic cannot use the lack of errors as information in its feedback loop.

I think the bug is that it was counting something when it should have not been, which would have violated both the design and intent even at the time it was written.

But I agree that code is only a bug if it does not do what was originally intended, if the original design was bad, and the code correctly implemented that bad design, there is no bug.

> But I agree that code is only a bug if it does not do what was originally intended, if the original design was bad, and the code correctly implemented that bad design, there is no bug.

The frame of reference for determining if something is a bug or not is the requirement specification, not design. If the design incorrectly or inadequately addresses the requirement, that is a bug as well. There isn't much point in rejoicing over the code that correctly implements an incorrect design!

I think the frame of reference is the programmer intent. If he/she intended to check for equality but wrote:

    if (x = 42) {

Then that is most certainly a bug.

If a bit of text was supposed to be blue but came out green, maybe because the person writing the requirements doc got it wrong, it is hardly a bug in the code. The code is doing what the programmer intended, even if that is not what the user wanted. That's why the world has moved on from "bug trackers" to "issue trackers".

> There isn't much point in rejoicing over the code that correctly implements an incorrect design!

And there isn't much point blaming a coder for correctly implementing an incorrect design, especially if the design document is all they have to go on.

> I think the frame of reference is the programmer intent.

Sure. However, that is secondary. The sort of mistakes that you mention can be caught during code review or unit testing, completely independently of what the actual user requirement is. This post, evidently, does not involve such a case.

> The code is doing what the programmer intended, even if that is not what the user wanted.

Your perspective appears to be inside-outward. It may be helpful during a performance appraisal, but does the business no good!

Edit: inside-out --> inside-outward.

Could it be that "bug" is defined from an "inside->outward" perspective regardless of whether that is the most useful perspective for the situation?

Programming is 100% design, there really is no separate "implementation" phase.

If the design has a flaw, whatever the reason: it is a bug.

A big part of development is "discovering the requirements." It is typically the hardest part for real world projects, and when that fails, it is perhaps not a bug in the code, but it still a development error that needs to be fixed.

If a developer got a piece of paper that said "blue" but the real world requirement was "green", and it was not possible for the developer to find out together with any stakeholder, then there is a development process error, that needs to be fixed.

In any event, it needs to be fixed, so pointing to someone else wont really help.

>not really a bug

Imagine a car turning off traction control in a pouring rain every time you stop at the lights because 'Hey,no tire spin in the last 5 seconds!!1'

I don't see why this would be a problem. Who on earth would jackrabbit from a stop in the pouring rain?

I don't think "bug" is a very well-defined term. It seemingly encompasses most software imperfections. I guess people will generally characterize things based on their overall (negative) impact.

Honest question: How is it that Google is the company that's always doing all of this involved technical work? Or is it just that their work gets publicity?

They have tens of thousands of engineers and work with vast amounts of open source software. Only a few other companies are of comparable scale in terms of sheer software engineering. I think it's inevitable that they'd have a very large footprint across the industry.

At least in the case of the Linux kernel, Google is actually around the 9th or 10th largest corporate contributor. [1] I'm not sure if they deserve to be particularly specially exalted, given the vast body of kernel work unrelated to them.

[1] http://www.linuxfoundation.org/publications/linux-foundation...

Looking at the list here: [1] Most of the top employers are hardware companies, and therefore likely to contribute mostly driver patches. For example, the rankings by LoC for this release list AMD as over a third of the contributions, almost certainly because of their new AMDGPU driver. Drivers are important, of course, but hardly news-worthy. Red Hat is always near the top of the list, and they do a lot of both the drudge work and the interesting stuff, but I feel as though everyone in the Linux community already knows that.

[1] https://lwn.net/Articles/654633/

I really do wonder what kind of numbers we would get if we did the same on a number of other large projects. I keep having the feeling that RH have a massive amount of say in the Linux ecosystem, for better or worse...

Red Hat is probably the most important company in Linux history.

I think that adding the "probably" is probably being overly careful.

Red Hat is an amazing company.

I think that adding the "probably" is being overly careful.

Adding the "think" is being overly careful.

Clearly for better. Most of the Google projects I've come across over the years are close to the throw-it-over-the-wall end of open source development, as opposed to Red Hat which seems to me to mostly let individuals participate.

Their CEO has written a management book (or perhaps anti-management) which explains their take on these things in some detail.

I'm pretty sure that if Red Hat had been running Android, it wouldn't be a big fork.

Oh, they /have/ their fork!

cough systemd cough

Éric Dumazet (author of the patch) is this highly productive guy working on network latency issues. He helped get CoDel into the kernel, and worked on TCP small queues (https://lwn.net/Articles/507065/) and the fair queue scheduler (https://lwn.net/Articles/564978/). Looking at the email addresses he used, I think he started working on the network stack on his spare time, while employed at SFR, a French telco. He implemented an in-kernel JIT for BPF (https://lwn.net/Articles/437981/), speeding up packet filtering; now that BPF is fast it is also used for performance profiling (filtering and aggregating in kernel) and syscall filtering.

There are of course others working in these areas, but I think Google has been going after them: Van Jacobson is a high-profile example. Dave Taht is still subsiding on ramen though (https://www.patreon.com/dtaht).

For the last five years I've worked myself to the bone, mostly unpaid, to solve the "bufferbloat" problem - first by organizing and leading the team that solved it, and giving away, for free, the ideas and code for anyone to use! And I've spent tons of time later convincing standards organizations like the IETF to make the ideas standard in new equipment - and also open source makers like openwrt to retrofit the fq_codel and cake fixes into millions of older yet upgradable home routers, and helping ISPs, vendors and chipmakers understand the both the need for the fq_codel fixes, and how to implement them.

wow - thanks for posting about David Taht. someone like Netflix or Google should get him on their payroll.

They have the money the hire those that can fix the issues. The issues also greatly affects their business, so they allocate resources to look into them. Less latency in the network stack means faster response time, means more ads delivered in less time. And a smoother user experience.

There is a talk https://www.youtube.com/watch?v=gQsOo_skjzk Where they show there are plenty of similar bugs left in TCP implementations. It's a mess.

But less of a mess today than it was 17 days ago.

Speaking of Linux kernel network stack enhancements, there's one I'm interested in but not sure where to start.

I was trying to setup ECMP on Ubuntu 14 and found I was constantly getting TCP RST. Turns out after digging into the code I found the path selection algorithm is not a consistent hash of a packet header tuple but rather pseudo-randomly chosen.

In older kernels there was a route cache so a path would be chosen and cached and so you would get a stable ECMP route per source. The route cache was removed so now the random selection runs on every packet leading to unstable routing.

It seems a simple fix of choosing the right hash function and maybe adding some configuration flags to determine which fields are hashes on (typically 5-tuple SrcIP, DstIP, Ip Protocol, TCP Src Port, TCP Dst Port).

But what would be incredibly cool is if my patch could one day possibly even merge to mainline. Is it reasonable to just blast a 'here's what I'm trying' out to netdev mailing list to get some feedback? I must admit I'm a bit intimidated to post...

Absolutely go for it! Post it even if you feel uncertain about it, just be clear about that, and about what your patch is trying to achieve, when the unexpected behaviour happens, and try to include a way for others to reproduce/investigate (as simple as possible, perhaps a program that demonstrates the bug). Also be clear about how you feel about whether or not the patch is the right approach and about what you would like list members to do with it (are you looking to confirm that what you're seeing is a real kernel bug, or a misunderstanding, or an application bug, and/or are you looking for help solving the bug/issue you're seeing, and/or are you looking to get it merged, etc). Can you demonstrate/quantify how your patch improves/fixes things, etc.

Open source is generally quite kind to newbies, provided you provide that important context. Try to explain not just what you're doing, but also be very clear about why. Explain the problem as well as you can, and then discuss what solutions you've considered thus far.

It sounds like you have a good understanding of the problem. If anything, I'd look into why the route cache was removed; perhaps there was a good reason for this? If nothing else, you can ask and learn something.

The reason this is good is because older wifi cards supported by iwlwifi get hardware buffer underruns and hangs when said bug occurs together with WPA2 enabled.

I'd say that is a reason for why this is bad. If I've got bad hardware I want to know about it and bad hardware should be flushed from the market. Not function "okaish" enough to not annoy users to the point of boycott.

Allow me to clarify. "This" means: "The fact that Google fixed a decade old bug", meaning the topic itself. The "this" you use in your comment is entirely different. The above misuse of "this" causes me to be confused, because it allows your comment to be interpreted as if you're promoting the inference of bugs, where older hardware is affected, such that people lose faith in that hardware faster due to frustration, consequently affecting the market causing it relieve itself of older hardware faster. Which is why I'm asking you to clarify your argument.

I only commented on the aspect of this bug that benefits crappy hardware. That is bad. Because your comment seemed to imply that the benefits for crappy hardware is the only reason for why this is good.

Which, depending on your viewpoint, can be seen as: the only reason this bug fix is good is because some company can still sell crappy hardware and get away with it.

Then we have the user perspective of course, which is "hey, my device doesn't crash as often" - which is awesome. But given the context the hardware is still buggy and it will still crash, albeit not as often. Which really isn't much comfort in the long run. Rare hiccups are worse than daily hiccups. Because daily hiccups you learn how to handle, a rare hiccup can really bite you. So you really haven't gained that much anyway, and in the long run the rare hiccup is rare enough for you not knowing who to attribute it to so you might buy the exact same thing next time which of course is very, very, unfortunate.

Reply on news.ycombinator.com