OpenBSD has started a massive strip-down and cleanup of OpenSSL | Hacker News Reader

OpenBSD has started a massive strip-down and cleanup of OpenSSL

lobste.rs

304 pointsevntdrvn101 comments

In the past I was part of an attempt to make asymmetric crypto easier to use for a commercial product. My role was to help design the product (simple, collaborative file sharing; you've heard it before) and to let the experts ensure the crypto was used properly. What drove me absolutely nuts was the level of paranoia and uncertainty around the safety of the libraries and algorithms used. The available experts were confident in the concepts, but feared every implementation. "Par for the course" you might say, but the status quo around OpenSSL and other libraries is unacceptable.

From an application developer's perspective crypto libraries look like a lightning rod even for the experts. No one wants to get too involved or make recommendations lest they back the wrong horse (admittedly an easy thing to do). So kudos to the OpenBSD team for rolling up their sleeves and attempting to build a solid foundation for the future.

I'd be happy if OpenSSL is simply fixed. I'd be overjoyed if there was a solid crypto system underneath an OpenSSL compatible API that gives us a path towards an open source, reusable crypto platform.

It's easy to back the wrong horse when the whole race track is full of horses with broken legs.

This might sound totally irrelevant, but it kind of reminds me of the situation with NASA. Once a certain technology has been proven to work, people do NOT like to deviate from that change, since hey, it might fail and no one wants to take the blame.

The end result is that NASA is still using computers from the 60s, and it took someone like Elon Musk to change the status quo which resulted in reduced costs (which is why NASA is currently contracting SpaceX, which can do things cheaper).

Bullshit. SpaceX dropped the launch cost, which is far away from the overall mission costs that NASA has to deal with. Dropping the launch cost means decreasing the cost of red tape and infrastructure (such as using fuel stored at higher temperatures), and increasing modularity and benefits of scale but it doesn't help decrease the cost of R&D and manufacturing of satellites and mars rovers.

NASA uses the RAD750 [1] as well as a few other still in production CPUs. Remember Voyager? The satellite is still flying and sending back priceless data after many decades? To launch missions (and critical infrastructure like GPS) you need years, decades even, for usage & reliability data to accumulate and industry to perfect the manufacturing process of parts that can handle environments that are, by any definition, at the extremes. But, please, show me a mission launched in the last 5 years with a 50 year old CPU design.

Most CPUs aren't even designed to last 50 years of operation, let alone rad-hardened.

The other side of the coin is NASA is rarely rewarded for success, but is punished for failure and it's a problem rife in the space industry - you can't have a mission fail.

One of the things the Deep Space Industries people were feeling confident about was they were signing all their contracts to have a 1/3 success criteria - they could launch 3 spacecraft, but would have fulfilled the contract if only 1 succeeds. That little bit of leeway, they think, will allow them to try new things and definitely lower costs.

Obviously some of this you can't do with manned missions, but its still a big cost saver when you change how you balance the equations. One of the things driving the CubeSat movement at the moment is the ability to fail - the low cost is encouraging people to take non-spacerated components, launch them, and see what happens, which then turns them into space-rated components (for a given mission envelope).

Thanks for bringing this up! In my general aerospace engineering class, taught by several active engineers and project managers from JPL, one of the professors did a rough estimate and figured that NASA could double the number of launches without effecting budget or success rate.

He thinks that, save for a few critical infrastructure missions, the diminishing returns get to such an extreme that they add no benefit to most missions. However, because NASA has to play politics for an ever shrinking pie, they can't afford to take no-brainer risks like launching ten missions instead of five because four failures looks worse than two, even if they both represent 40% failure rates.

Does NASA manufacture CPUs? I was under the impression they just buy radiation-hardened CPUs from large corporations.

You're correct. If I'm not mistaken the RAD750 is a chip made by a BAE systems.

or in enterprise IT "No one ever got fired for buying IBM"

> (which is why NASA is currently contracting SpaceX, which can do things cheaper).

And they're also contracting Orbital Sciences and Sierra Nevada and Boeing and probably a bunch of others of whom I have no knowledge.

SpaceX is following in the footsteps of many other private companies that have developed spaceflight hardware on their own initiative. They might be 'younger' and more agile but it's not a new innovation.

I'd be surprised if NASA is using, on a wide-spread basis for missions, computer hardware older than 20 or 25 years at this point. I'm extremely skeptical they are still using much (any?) hardware from the 1960s or 1970s. Your exaggeration goes way too far.

It was an exaggeration, but it seems that the Shuttle was using computers which were a '70s, then a '90s version of a '60s design: https://en.wikipedia.org/wiki/IBM_AP-101 At least the '90s upgrade removed the ferrous core memory.

Do you know about NaCl (by DJB)? It doesn't implement SSL, but it is a very easy to use library for asymmetric crypto.

http://nacl.cr.yp.to/box.html

Can any crypto experts comment on whether it is feasible / how much work it is to implement SSL on NaCl? Maybe the issue is that NaCl doesn't support all the ciphers you need.

I do, and I hope to give it a try in the latest product I'm building. Like everyone else I'd prefer to see some expert validation of NaCl before I put a ton of trust into it. That said DJB's track record is pretty good in my eyes (I liked the design of qmail, a lot).

I've examined it, and you can fuzz comp against tweetnacl. It does the right thing generally and tests against test vectors in the build process.

I have been wondering the same thing. This link suggests that there are problems with NaCl preventing adoption, and puts forth a repackaged alternative called Sodium:

http://labs.opendns.com/2013/03/06/announcing-sodium-a-new-c...

The SSL/TLS protocol unfortunately uses some known-bad constructions, which lead to intractable issues (see: BEAST, Lucky13 for examples)

NaCl's goals are vastly different to those of SSL/TLS. SSL/TLS aims to provide a simple, clean interface with sane defaults for the majority of simple use-cases, whereas SSL/TLS aims to provide an interface with near-infinite flexibility for the case of providing an encrypted, authenticated tunnel.

NaCl also deliberately does not support lots of ciphers, as that makes it easy for developers to choose poorly, for example, (Alleged) RC4, as is supported in OpenSSL.

Did you mean

> NaCl aims to provide a simple, clean interface ...

NaCl supports only one cipher for each purpose by design, it could never be used to implement TLS. It's only useful for making your own protocol.

Part of the problem here is that it is very hard to audit something OpenSSL's size. It has 450,000 lines of code. Are you going to skim, let alone audit, a mathematics-heavy codebase of that size? No.

Some of that is OpenSSL's fault, but a lot of it is inherent in implementing a 200-page (before extensions!) standard. GnuTLS is 170 KLOC, CyaSSL is 205 KLOC, and even PolarSSL, commonly thought of as the gold standard in trimming the fat, is 55 KLOC.

If you want to reduce the level of paranoia, you would have to pair down the size and scope of TLS by an order of magnitude. For the record, I support that kind of thing, but you would have to cut a lot of things people actually use, not to mention breaking backwards compatibility.

While I understand where you've coming from I think your attitude is dangerous. Implementing the SSL/TLS spec(s) is always going to be difficult while they are in widespread use. That said refactoring the existing libraries (to make them more readable and understandable) and then auditing them (to gain confidence in their efficacy) is especially important given the nature of their applications.

Doing the hard work of getting this right, or accepting the status quo and getting it wrong, will have a very real impact on the future of many, many people. Should the time come that changes have to be made to the standards themselves then we can debate the merits of breaking backwards compatibility. I value pragmatism more than most (I think) but there comes a point where the cost of not changing is higher than the cost of changing. I think we're there.

Unfortunately it's going to be difficult to have much confidence in a huge edit of OpenSSL since OpenSSL is almost devoid of meaningful tests. It would be very easy for one of these "resolve merge conflicts" changelists to contain an error of the goto fail; goto fail; variety.

Obviously there's always a risk that something slips through. However it's not just a huge edit, it's also a huge review. And that is made so much easier when a ton of junk is removed from the code base, and the style is changed towards readable.

When I do code reviews I read the tests first. How can you review code with no tests?

All the bugs in OpenSSL to this date have been reviewed by someone. Unfortunately those people haven't been veteran industry programmers but rather academics and "hero" types who think the kinds of things in OpenSSL are good code. We can only hope that OpenBSD folks can clean it up a bit but I'm not hopeful, because OpenBSD is just a different tribe of "hero" programmers.

When I do code reviews I read the tests first. How can you review code with no tests?

By reading it, running it in your mind, and seeing if it runs the way its author expects.

Tests are ideal, but they aren't the only way to specify the correct behavior of code.

If there are no tests, you can always start by writing some..

Judging by what little I've read of libssl, having tests wouldn't be that much help, as any given test would be spread across three different .c-files and two .h-files and depend on ten different defines, and a few redundant macros. /s

The problem with tests is that they reflect what the test's author was thinking about. When auditing something you especially look for things the code's author did not consider. If you do your review only based on the tests instead of really thinking the code for yourself, you might very well miss the kind of critical mistakes you should be trying to find.

> When I do code reviews I read the tests first. How can you review code with no tests?

Please grow out of this religion if you care about robust, secure and well written software.

You can look and see what are the types of things that get fixed in OpenBSD. Tests are not going to tell you that code works around safety measures built in to the OS. Tests are not going to tell you that the code uses time as entropy to seed RNGs. Tests are unlikely to tell you about the subtle integer overflows or out-of-bounds accesses that attackers can exploit. Tests are unlikely to tell your daemon will ungracefully fail due to fd exhaustion while serving a larger amount of requests on a loaded system. Tests are not going to tell you there are deceivingly familiar looking functions with surprising return values that will confuse people. Tests are not going to tell about bad style and code smell that makes life harder for anyone working on or looking at the code.

There are so many things that are hard or impossible to test, but which are vital to handle right if you're doing anything robust and secure. In fact you could construct a test for some of these things, in theory, but in practice you never will. And you certainly never will unless you have actually read the code, because all these things are issues within implementation details inside the black box; you must read the code (and understand the underlying platform) to even become aware of these issues.

Sure, tests are really useful for catching some specific issues. But there is so much more. Tests are somewhat like compiler warnings: often extremely helpful, but even totally insecure crap can compile cleanly. And it is possible to write good code even if you're not using the smartest compiler to compile it.

If you wish to learn how to review code with no tests, I recommend you read the C language chapter from The Art of Software Security Assesment[1]. Actually read the entire book. Then go through the commit logs in some project -- such as OpenBSD, who've been doing it for almost 20 years -- and see what kind of issues they find and fix.

[1] http://ptgmedia.pearsoncmg.com/images/0321444426/samplechapt...

On the other hand, there's more to testing than traditional unit testing in which a programmer enumerates a series of cases and verifies that what they think should happen (which may or may not even be correct) happens.

There's fuzz testing. There's QuickTest-like testing, which is tricky in this sort of situation, but powerful if you take the time. There's static analysis, which are basically a form of automated testing. In fact, static analysis can in fact test some of the things you claim can't be tested. Some of the other things are also more testable than you are saying, such as resource exhaustion, which can be both simulated via injection or via simply exhausting resources in the test case.

Further, even the relatively-weak guarantees that traditional unit testing can provide are a foundation for further analysis. What's the point of analyzing a bit of code for whether or not it uses entropy correctly if you miss the fact that the code is straight-up buggy? Given the demonstrated difficulty of writing code even before it's security code without solid testing, this is hardly the time to be claiming that testing isn't necessary.

This strikes me as another iteration of the Real Man argument ("Real Men write in assembler" being one of the originals). The truth is we need all the help we can get because we humans aren't very good at programming in the end... and of all the places I'm not inclined to accept the Real Man argument, it's in security code. This is the sort of code that ought to be getting covered by two or three separate static analysis tools, and getting reviewed by skilled developers, and getting fuzz tested, and getting basic unit tests.

No Real Men arguments here. I did not claim testing isn't necessary. I just wanted to challenge the blatantly religious attitude of pretending or believing that a particular form of testing (one with clearly written tests the poster above me would actually read) is all there is and that you cannot review code otherwise. You definitely can, and must.

As far as I am concerned, all techniques that help improve code are welcome and even necessary. This includes testing, manual review, static analysis, etc.

"Removal of all heartbeat functionality which resulted in Heartbleed"

Instead of simply fixing the bug and moving on, we're decided to assume the whole thing is tainted and burn it alive like a depressed teenager at the salem witch trials?

or is the heartbeat and IETF RFC considered suspicious and the heartbeat extension offers us nothing?

It probably isn't needed by 99.999% of users. It should not have been added.

I think it can be added back if needed after the code is properly reviewed.

In addition to kansface's point, I'd observe it's a very recent addition. We made it a long way without it.

There's a quote referenced so often it has become a cliche: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." by Antoine de Saint-Exupery. I think it's sometimes over-applied, but in the case of crypto software, it really fits. In a sense, the heartbeat code got unlucky and hit the mega-anti-jackpot, disproportional to its "actual" risk... but there was risk in adding it, and even before we know the outcome it would have, somebody really should have been asking hard questions about whether the reward justified the risk. I imagine there's a lot of other bits of the code where the risk/reward tradeoff suggests removal. You can look at much of what has already been trashed from that point of view. For instance, if you have no real need to support a particular defunct architecture, why carry around its code? It has no benefit, but non-zero risk; toast it.

An good engineer's time should be spent more on taking away than on adding.

I think Colin Percival put it best in his blog post about Heartbleed:

"The most secure code in the world is code which is never written; the least secure code is code which is written in order to comply with a requirement on page 70 of a 100-page standard, but never gets tested because nobody enables that option."

It's the wrong abstraction layer for a heartbeat signal. That can easily be done by the application if it's needed. TLS and DTLS should provide encryption wrappers for TCP and UDP. That's it.

Sure, it's convenient to have it already available for you to use as an application developer, but critical security software needs to be kept as small and simple as possible.

> or is the heartbeat and IETF RFC considered suspicious and the heartbeat extension offers us nothing?

The heartbeat extension is classic protocol bloat. In theory it can be useful for DTLS as a keep-alive for NAT port mappings or to do PMTU discovery, but you can do either one without it fairly easily. And then they went and added it to traditional non-datagram TLS where it serves the traditional screen door:submarine relationship.

The beauty of open source is that you dont need to worry about silly things like productivity and return on time invested if you dont want to.

The code is a mess from what I hear. As long as they keep the functionality intact I dont really think anyone can complain.

I was quietly just thinking... Man I've gone to some lengths to force a rewrite in the past but this just takes it to a whole new level.

Is this fork a vote of no confidence in the OpenSSL team? Is it guaranteed that this fork will be used in OpenBSD? Could this fork be used in Linux? Will they be cooperating with OpenSSL by sending them bug reports etc?

Traditionally, OpenBSD has made "Portable" versions of their code available for integration on other systems - for example:

http://www.openssh.com/portable.html

Basically, that's the version that gets those sorts of patches.

I'd imagine if this reaches the point where people want to put the OpenBSD fork of OpenSSL on other platforms, a similar approach would be used.

It's been pretty successful for OpenSSH:

http://www.openssh.com/usage/graphs.html

Nearly every operating system distribution maintains it's own fork of most system utilities and libraries with varying levels of divergence from upstream. A piece of recent news around here was Python 2.7 being maintained by RedHat for several years beyond upstream support. Some of the differences make it upstream, some of them don't. Doubtful OpenBSD is trying to usurp the current OpenSSL team, but if they do good quality work, it might be very influential a few years down the road.

>Is it guaranteed that this fork will be used in OpenBSD?

It already is used in OpenBSD. The changes are being made in the OpenBSD source tree.

As a Linux fan who has been hearing good things about FreeBSD, will this work out of the box on FreeBSD?

I think it will require a bit of tweaking, but I would expect the other BSDs to adopt it once it is reasonably complete.

"Removal of MacOS, Netware, OS/2, VMS and Windows build junk"

There's surely code in this project older than some of the developers working on it.

Or maybe only gray-haired curmudgeons are working on this... CVS is quite a high entry barrier for young trendy Githubbers!

Is this an attempt at sarcasm, or just good ole ageism?

I sincerely hope it's both actually. Considering I fall into the Github-generation (age-wise) of developers this is one of the things I constantly find annoying when discussing stuff with - Generation GitHub - collegues; if it's not on Github/Not in the language we usually work with/Not agile/Boring code... according to them it's not worth discussing.

I'm actually really impressed what so many of the OpenBSD developers can delete/reformat/edit the OpenSSL code at the speed they seem to be working right now and not run into issues with CVS. One awesome thing about CVS is cvsweb ( which is a separate thing, I know), I much prefer browsing around, diffing and viewing changes in cvsweb compared to say GitHub or Bitbucket.

Sort a side note: OpenBSD really seems to have picked up speed after they fundraising. It wonderful to see the money has been so well spent.

> One awesome thing about CVS is cvsweb ( which is a separate thing, I know), I much prefer browsing around, diffing and viewing changes in cvsweb compared to say GitHub or Bitbucket.

Me too. It would be really nice though if it gave the option to show diffs for all files touched by a single commit.

I'd also love to get the diffs along with commit messages straight into my mailbox...

Sorry, there's nothing awesome about CVS, especially not cvsweb.

If you want a git client that's graphical, there's dozens to choose from, many of which do a lot of what cvsweb does and more.

You can also create your own Gitorius (https://gitorious.org/) server and view changes there before pushing to the upstream. It's like your own personal GitHub.

It is wonderful to see improvements. It's just that a refined CVS workflow is downright awful compared to the default git one.

Being able to git clone, monkey around with your own branches, and never need commit rights is a huge deal for those looking to work with and improve your software.

This will be interesting to watch; I hope OpenBSD can do for OpenSSL what they did for OpenSSH—clean it, refine it, repackage it, and make it even better, ready for a new decade of use. I published a blog post which mentions how OpenBSD transformed SSH into the de-facto remote administration standard it is today just this morning[1].

I think this will be good news!

[1] https://servercheck.in/blog/brief-history-ssh-and-remote-acc...

Even though passwords are sent in the clear, ...

Ssh has set back the computing landscape 20 years. Why oh why the tty?

http://harmful.cat-v.org/software/ssh

Linking a lobster thread on HN, instead of the original link? Why would you want to do that?

I know it's unusual, but in this case I thought it was justified because the original BSD CVS link offers little context of what's going on unless you parse the changelog, whereas the lobster thread has a good summary by jcs.

OpenBSD Journal[1] points out "To clarify, not all of the cryptographic engines were removed; the padlock and aesni engines are still in place."

1) http://undeadly.org/cgi?action=article&sid=20140415093252

Finding it easier to read through the commit log here than in cvsweb:

http://freshbsd.org/search?q=libssl

That is a pretty nice way to track the changes. I see some of the comments are getting ranty http://freshbsd.org/commit/openbsd/58777eed1cff7c5b34cbc0262...

"OpenSSL must die, for it will never get any better." - PHK http://queue.acm.org/detail.cfm?id=2602816 A pretty good, recent look at OpenSSL code and how it can be improved.

Good. Another look at any code after a long period often yields various degrees of improvements due to a different perspective at the later date.

This isn't "another look"; OpenSSL is not affiliated with OpenBSD/OpenSSH, despite the similar name. This is OpenBSD forking OpenSSL.

Someone at the linked URL humorously commented "I hope the new version is called OpenOpenSSL."

? Did you think I assumed "similarity" due to "Open" in the title? It is "another look". Another look can come from original writers, yes, but often a second pair of eyes brings a different perspective that is almost always beneficial.

It came across that way. You mentioned "after a long period", and whatnot.

Or just call it "WideOpenSSL" until they are done fixing the code and writing tests.

I like how the OpenBSD team is picking up the ball here instead of throwing up their hands in despair, even if it means forking the code.

Excellent PR move on the part of OpenBSD. This should definitely improve their donations and work.

Understandable reaction, when it can be seen how people that wanted to have all the newest features in their product (newest TSL features in this case) shoot themselves into their foot while others that stayed longer with the old (proven) version where save ...

That is one reason for the popularity of Debian: (at least in the past) they did not jump on any new bandwagon immediately. Other distributions have much newer software, but also the newest bugs.

OpenBSD has a great security record, definitely excited to see what they can do if they take over OpenSSL maintenance.

Who knows, maybe one day we'll be able to install OpenSSL without having to first install Perl?

IMO, this a symbol of OpenSSL's unreasonable complexity.

OpenSSL is a default system library yet the user cannot compile it with only default system libraries.

(OpenBSD has added perl to their userland but this is not universal across other UNIX-like OS.)

Where are the compile-time options to turn things off?

The compile process has apparently become so "challenging" that the developers could not figure it how to do it easily without using perl.

I don't know much about OpenSSL but what I know for sure is that refactoring the code itself might not be enough. There must be some process involved to keep track of what was refactored, what was not, enforce rigorous testing, systematically review the code, and make sure all of that is publicly available (when I look at the website, that information is nowhere to be found)

That list misses a an important bullet point:

* Added dozens of tests.

Removal of MacOS/Windows junk?

Is this going to be an openbsd only fork (I guess linux will get a build of some sort as well).

I'm taking this to mean MacOS as in MacOS 9 and prior, not OS X which is unrelated. Building for MacOS was always an ugly ordeal.

Most likely, yes.

I wouldn't be surprised if a lot of the "Windows" support code is of similar age, and thus was actually only needed to support pre-NT Windows.

It's almost certainly referring to pre-OSX versions. However, if it ends up being handled like OpenSSH, the core will be OpenBSD only, with porting handled by another team.

If the cleanup turns out well I assume it'll get ported back to other platforms. It's very hard to change code with a bunch of stuff ifdeffed out (since you're basically guaranteed to break that stuff regardless of how hard you try not to), so even if they cared about other platforms this would probably be a reasonable approach if they're going to make major changes all over the place.

I was looking at this recently. Most of the Windows ifdefs in OpenSSL are junk. Even on Windows.

A lot of OpenSSL is portable C and IMO they should just let portable C be portable C.

It says "build junk", not only "junk". I haven't dug through, but maybe it was all the junk that was used to compile OpenSSL on said platforms.

If the code can be compiled without this, then they might as well remove it.

We take for granted the "modern" security tools and libraries.

But at the same time, one would typically assume that something as large as OpenSSL has been reviewed and tested...

Appending to the PHK on the OP link, it would be nice to be able to randomly choose amongst several crypto implementations.

...still waiting for an article about a team of super-heroes starting to rewrite OpenSSL in Haskell

You mean http://hackage.haskell.org/package/tls ?

Yay! Now OpenBSD can introduce a massive security bug. /s

They are long overdue for one =)

Wow, this is what the Internet runs on?

TLDR; but in my opinion the bug is causes by using C, use well test container structures/functions, like those who test the bound of the container during runtime. Those brave enough could disable the bound-checking at compilation for performance gains.

How about adding "completion of our manpages" or "making our website less shitty" to that list? See https://www.openssl.org/docs/

Hmm, no I'd rather them make a quality product than work on their website.

The last thing OpenBSD or OpenSSL needs is a whack of totally-useless js on their website.

Documentation would be a HUGE help to the code quality of OpenSSL because someone would have to actually sit down and describe its ghastly interface. Whoever it was would get about half way through it and then stop and say "Wait, what? Seriously?"

The only other codebase I've seen that's remotely as hairy as OpenSSL is GCC's gnarly mess. (I've heard it's improved since I looked a few years ago.) And that's notoriously engineered to be difficult as possible to contribute without giving back; I can't imagine what OpenSSL's excuse is. Any sane programmer would have cold sweats even if they understood the implementation because it's so difficult to figure out and verify what the fuck is going on internally with any speed.

Documentation would help, but a good cleanup would make provided documentation much less necessary. Crypto may be difficult to understand, but with clean coding practices and formal verification (even on an audit workflow level) would be a much better investment, IMHO.

I didn't mention javascript. Try running their site through a validator: http://validator.w3.org/check?uri=https%3A%2F%2Fwww.openssl....

Duplicate IDs and font tags? Come on...

But it works, right?

For what it's worth, OpenSSL and OpenBSD/OpenSSH are not maintained by the same team, even though the names are similar. This was what I was given to understand last week after answers to a slew of questions asking about this.

Ah, now all my downvotes make sense ;) The article gave me the impression that OpenBSD was the maintainer of OpenSSL.

>How about adding "completion of our manpages"

That is pretty much a given. OpenBSD considers deficiencies in documentation the same as deficiencies in the software.

>or "making our website less shitty"

OpenBSD developers have nothing to do with openssl's website.

If it's a given, then why do all 3 manpages on https://www.openssl.org/docs/ say "[STILL INCOMPLETE]" and have for years?

Edit: I just read parennoob's comment about how OpenBSD isn't the maintainer of OpenSSL. I didn't know this was the case.

Reply on news.ycombinator.com