Critical crypto bug leaves Linux, hundreds of apps open to eavesdropping

But this is anything but true. The reality is that code review is far less common than code use, with many defects impacting many users as the logical consequence.

If it is open source, as in a Debian release, an end user's recourse is to fix it themselves.

If it is commercial, be it closed source or not, an end user's recourse is to sue the supplier (or something similar).

The commercial supplier has a financial incentive to get it right, the open source developer has an intellectual and street cred incentive to get it right. I'm not sure which one actually works better, I know that the popular opinion is the ESR eyeballs claim but it's not clear to me which gets it more correct. Seems like they both fail at times.

  great_success = sum([i.debug_skill * log(i.audit_time) for i in code_reviewers])

I don't personally know Theo De Raadt but if he spends his time bringing down other people, he's probably not a very happy person himself.

I mean shit, I don't need to be an aeronautical engineer or work in the FAA to say that several people investigating an airliner crash will be more effective than one single dude sifting through a literal field of debris.

ESR's quote isn't wrong (perhaps hyperbolic, but in spirit it isn't wrong). It's just an uninteresting observation. What do you do when something has you stumped at work? You ask the guy across the hall to look over your shoulder for a second... It's just common sense.

People simultaneously revere and hate the quote only because ESR said it. If Joe Shmoe had said it all those years ago, it would have just been met with "no shit" and never given second thought.

The problem with ESR's view here is that it occurs in a more general essay on this matter come up with the idea that open source has the advantage because of eyeballs, distributed design and so forth. But you can only distribute design so far, and almost every successful open source project has a small design and engineering team.

This being said there are plenty of cases where I do in fact rely on many eyes. It's not that it makes the bugs shallower but that repetitive review and discussion helps to shake out both design flaws and software bugs. I tend to push a lot of security stuff to the underlying platform for this reason. But part of it is also trusting the design team.

Code auditing is tough work and I generally assume it doesn't get done. What is good however is that a lot of other people are depending on software that is professionally developed in a relatively transparent manner and so chances are somewhat better that people will audit the code at some point.

ESR is not completely mistaken here, but the problem is that he's not exhaustively right with "Linus's Law".

It's more a description of why beta testing with access to source code is good, than a description of why open source is inherently good (and before you flip out, I've contributed to open source longer than I've done my day job).

Some types of code flaws will simply never show up to the kind of beta testing that introducing software to a wide population provides. E.g. proving that a X.509 certificate which is faulty in a certain way is actually caught by a software library; few people run into that in practice.

What these types of bugs require is code auditing (after the fact) or before-the-fact code review that prevents their entry into the source in the first place. But neither auditing nor pre-commit review are inherent to open-source, and in fact it could be argued that closed-source software companies are better able to ensure sure things happen.

The saving grace for open source is that these companies optimize for market success and not code quality (except to the bare extent needed for market success). Additionally you can pay to audit open source code much more easily than you can closed source (e.g. Google's audit teams that do exactly this).

But "bugs are shallow in the presence of sufficient eyeballs" is not unique to open source, either in theory or in practice.

For all we know, bugs are found in closed sourced systems all of the time and are fixed frequently; the only difference is that they're not publicised.

And your tax dollars are paying for all of that work. And you can be sure those elves have known about those bugs for years.

And those bugs have caused many unfortunate consequences. So it's just not wise to go around giving people a false sense of security in order to promote your brand.

In this case it looks like the GnuTLS bug was introduced and fixed by the same person, but I didn't go look to see how many others there were.

Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this:

8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Or, less formally, ``Given enough eyeballs, all bugs are shallow.'' I dub this: ``Linus's Law''.

My original formulation was that every problem ``will be transparent to somebody''. Linus demurred that the person who understands and fixes the problem is not necessarily or even usually the person who first characterizes it. ``Somebody finds the problem,'' he says, ``and somebody else understands it. And I'll go on record as saying that finding it is the bigger challenge.'' That correction is important; we'll see how in the next section, when we examine the practice of debugging in more detail. But the key point is that both parts of the process (finding and fixing) tend to happen rapidly.

In Linus's Law, I think, lies the core difference underlying the cathedral-builder and bazaar styles. In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out. Thus the long release intervals, and the inevitable disappointment when long-awaited releases are not perfect.

In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.

And that's it. That's enough. If ``Linus's Law'' is false, then any system as complex as the Linux kernel, being hacked over by as many hands as the that kernel was, should at some point have collapsed under the weight of unforseen bad interactions and undiscovered ``deep'' bugs. If it's true, on the other hand, it is sufficient to explain Linux's relative lack of bugginess and its continuous uptimes spanning months or even years.

Actually people seem to need frequent reminding that we don't live in the best of all possible worlds, and that some of the difficulties in working with the best-available current tools aren't inescapable but rather shortcomings that can and should get fixed sometime. It's great if everyone now knows how suboptimal C-everywhere-forever is on security grounds, at least—but tbh 'security-conscious developer' sounds a little like 'true Scotsman' here. A hae me doots.

There is nothing wrong with developing security-critical code in C if you're an experienced C programmer and know what you're doing. The GnuTLS programmers were neither experienced C programmers nor knowledgeable about security; they should not have been writing security-critical code in the first place. And no one should have ever believed that their code was trustworthy.

    if(somefunc()) goto fail;

Nonzero: error code, zero: success

Nonzero: one of many possible success values, zero: error (e.g. malloc())

Negative: error code, positive: one of many possible success values

And each has its tradeoffs.

Orders of magnitude different degrees of harm.

That does sound like a good idea as long as the meaning of `1` and `0` do not get confused, and as long as we're not talking about some niche context where extra overhead for the sake of code maintainability would be a problem.

The fallacy that you could have easily written a test case for this bug appears neatly encapsulates the weakness of "TDD" as a mitigation for programming flaws.

Define "known-bad" in a general enough way that a specific test can be created to cover the entire range of "bad" certs. That's quite difficult, and, probably isn't realistically possible to go through all the "known-bad" if you want your tests to run quickly.

Realistically, all you can do is have regression tests to make sure that the found bugs aren't repeated in future releases.

I agree that it's helpful, but the problem is coverage. Your test code, most likely, doesn't cover EVERY single possible condition that can happen with a simple TCP/IP connection. Especially once you get out of localhost land, where you're dealing not only with your code, but all the hardware and software between the two systems

The fundamental problem is that even with a rigorous test suite, you're probably going to run into things you didn't even think were possible once the code is out in the wild. For example, we just ran into a scenario where we were seeing corruption through a TCP connection. Knowing the wire, it was impossible for the packets to be appearing in the way that they were(there was some packet level corruption). After going through multiple wireshark logs, we found that the culprit was a hardware firewall in between the server and client. Thankfully, our code didn't crash, but it's also something that we never tested for, because(in theory, at least), it should never be possible for that specific corruption to be sent in the first place.

    - Here's a library that claims to validate certificates.
    - Here's a TLS httpd with a forged commonName.
    - for ciphersuite in $supported; do

In Arch Linux very few packages depends on GnuTlS.

And in other words, nobody should be listening to thrownaway2424. Ever.

Both cannot be wrong can they? It is a logical impossibility.

It's probably tempting to think that unit testing might save us from security failures. People have written many words on both sides. But how many of us have actually tried it? No one who hasn't tried (and tried hard) to see whether unit testing works has any business arguing for (or against) unit testing. We simply don't have enough data to know whether it's valuable. And the fact that unit testing works in domains other than security isn't evidence that unit testing would produce any kind of benefit in computer security.

What's getting lost in the noise here, I think, is that it's important to zoom out and think about a broad overview of how difficult compsec is. Think about all the different situations that require security. Even something as seemingly simple and straightforward as an OS-level copy-paste mechanism requires thinking about security.

Could you write unit tests to cover all security situations? How about all possible situations? If not all possible situations, then why? What's the difference between a situation that you can unit test and one that you can't? Is it possible to write a unit test that checks whether there's a sequence of steps a user could take to exploit an OS-level copy-paste mechanism? Can you unit test the efficacy of a TCP/IP stack? Is it possible to unit test whether the Debian OpenSSL PRNG is being fed with sufficient bits of entropy? Can you unit test whether the colors of your website are pleasing?

The most interesting answer to those questions isn't "yes" or "no." The most interesting answer is "maybe." Because "maybe" means nobody knows, and that's fundamentally interesting. It's possible that it's possible to write unit tests for domains that nobody has thought of yet. (Yes, even tests that automatically infer whether your website's colors are pleasing to humans.)

Could unit testing have saved us from this one particular case? Maybe. But is it possible to write unit tests in the general case? Maybe not. But you, I, and everyone else doesn't know until we actually try. It could be impossible, or if not impossible then infeasible, or if not infeasible in a dynamic language then infeasible in C libraries. On the flipside, if you investigate, and you come up with a way of writing unit tests in the general case for security libraries, then you may have just discovered a breakthrough in how cryptographers can write security libraries. Because if it's possible to write useful unit tests in the general case for security libraries, then you will have demonstrated something that most cryptographers generally disbelieve, the same way that people disbelieve that it's possible to unit test whether colors are pleasing.

So get to work! There's nobody stopping anyone from right now going and trying to write those unit tests. Try it, write a blog post about your experience and the hardships, write about what you've learned. I guarantee it will be interesting.

All I'm saying is that we should be experienced before we advocate or dismiss an idea, and the only way to get experience is to try it out.

Sorry for picking on your comment in particular. I could have replied to any number of comments in this thread.

Incidentally, would this bug have been flagged by a compiler warning?

[1] http://en.wikipedia.org/wiki/C99

A lot of the rest is just sugar (e.g., C++-style comments). Moreover, many code bases already evolved patterns for dealing with the problems that many C99 features are supposed to address. Booleans are a good example: many environments already have a fine boolean_t definition, and it's not really worth it to convert existing code or introduce a second pattern for representing boolean values.

Variable-length arrays are another good example. They're fine, but alloca() is a perfectly reasonable solution to the same problem, and it's already widely used. IMO, the safety issues usually brought up around alloca() are neither more likely nor more serious than problems like allocating a gigabyte-sized object on the stack or returning a pointer to stack memory. Since competent C programmers don't generally make those mistakes (IME, of course -- I think I've never debugged a problem that turned out to be a misuse of alloca()), I find the added complexity of a new language primitive isn't worth it. (Incidentally, that's the same reason I dislike nearly all of C++.)