Google Chrome 51 disables HTTP/2 on most Linux distros due to old OpenSSL
ma.ttias.be
ma.ttias.be
Dynamic linking does not make it easier to update things. There, I said it.
Updating things confidently and safely requires -- as the article wisely references -- testing those things, and the entire matrix of other things they interact with.
Static vs dynamic linking simply doesn't register. In either case, the newer versions need to be built, and then tested together.
The only thing dynamic linking does is make it possible for distros to hamstring themselves[^] by tossing everything into one big cauldron, so nothing can be upgraded because it requires testing too much at once.
And here's the endgame. "Is OpenSSL getting updated on $distro?" "Nope."
Sigh.
---
[^] And this is a red herring, in turn: dynamic linking isn't the real enemy either; it's the global sloppile approach to sharing dynamic linked libraries that really causes the mess. NixOS and others like that still use dynlinking, and yet have successfully immunized themselves from these problems by using content-addressable library versioning. So let's stop with the "dynlinking is better for security updates" claims. It's not. It's either unrelated at best, or actively a ball-and-chain making things harder and slower, and I submit as evidence OpenSSL versions across these distros.
However for plugins when performance is critical, is a good use case.
For anything else, as the last decade as shown us, it is probably better to use OS IPC instead of exposing the application internals to third parties.
What am I missing here?
In making this transition, Chrome experienced a massive practical regression for many users, precisely because the dynlinked library is not sufficiently up to date for many people.
I'd say dynamic linking is related to the issue at hand, yeah.
No, but if Debian packages had statically compiled OpenSSL you could just update say Nginx and get it working with the new protocol Chrome requires, while 100+ packages could still be left as is and use their own, older, version of the lib.
So yes, two different versions but is that automatically bad? It could be, but remember that exploits, even serious ones, are usually on very discrete vectors. It can be hard to exploit something abstracted away through two application layers.
Google could even take this on and put up apt/yum repos supporting various versions of various distros; with their resources and infrastructure it wouldn't even be that much work.
Also your argument has nothing to do with security updates, and I'd maintain that dynamic linking does indeed make security updates easier, since you still only have one package to update rather than some potentially large N.[0] Yes, there is still the burden of testing dependent packages, but at least they don't need to be rebuilt.
I would argue it's safer from a security perspective, as it's always possible to forget that a particular statically-linked package needs to be rebuilt. At my org we used to have a few special-flower services that needed a newer version of OpenSSL and were statically linked. This just meant more work after every security disclosure. (Arguably we "did it wrong": instead of static linking, we should have just built and installed the shared lib and linked to it, making it easier to identify and track what needed to be updated.)
[0] At least for the case where a patched update can be released, one that doesn't have ABI changes. If you're running a distro that is using an EOLed version of a library, I'm sympathetic, but ultimately that is a problem larger than the one at hand that you need to solve.
1 - http://blog.chromium.org/2016/02/transitioning-from-spdy-to-...
CNN is not even in the top-50 websites, and ESPN is not even in the top-100 (e.g. in the Alexa index).
Heck, pornsites like xhamster have far better placements than either.
Google, YouTube, and Facebook.
(edit: more positive wording)
Based on my experience, replacing nginx (in my case acting as a fairly simple reverse proxy for rails) with Caddy wasn't significantly harder than compiling a custom nginx. If you have a complex configuration, YMMV. I'm definitely not suggesting using Caddy as a drop-in replacement for your existing production web server without further testing. That being said, getting rid of a OpenSSL dependency isn't a bad thing.
That is impressive, will definitely try it. Is there a list of companies using Caddy somewhere?
What is important is that the library is not hardware accelerated and it thus uses a software accelerated AES cipher. So, I think it is a bit slower, but you would really have to measure it yourself.
Upgrading your Load balancer should not be that hard, though.
To be fair, it's not. It simply wraps Go's HTTP/2 server which is mostly developed by Brad Fitzpatrick, which in turn relies on Go's crypto libraries (written in Go and assembly).
As a Go user, I personally don't see any point in using Caddy, as I prefer writing a few hundred lines of Go to serve static pages with all the features I need, with the additional ability of serving more than static pages (you can do fastcgi with Go's stdlib if you need it, I personally prefer writing server code in Go). This is especially true considering that they called their non-paying users dishonest, for not "honestly paying back the value it provides you" while at the same time not sharing a single cent of their revenues with Google who develops the actual HTTP server they wrap (or with any other third party libraries that "provide value" to them). (old discussion: https://news.ycombinator.com/item?id=11264041)
I don't think that's a fair description of Caddy. Of course they're going to use the existing packages in Go, why would they reinvent the wheel? Caddy provides useful things like a config syntax, middleware, automatic certificate deployment, etc. which you'd have to reimplement or take from somewhere else.
> As a Go user, I personally don't see any point in using Caddy, as I prefer writing a few hundred lines of Go to serve static pages with all the features I need, with the additional ability of serving more than static pages (you can do fastcgi with Go's stdlib if you need it, I personally prefer writing server code in Go).
Caddy allows you to write your own extensions/middleware, which should cover all that. In turn, you get to re-use all the existing extensions and other things Caddy takes care of (which, in my experience, you'd inadvertently end up reinventing at some point anyway).
> This is especially true considering that they called their non-paying users dishonest, for not "honestly paying back the value it provides you" while at the same time not sharing a single cent of their revenues with Google who develops the actual HTTP server they wrap (or with any other third party libraries that "provide value" to them).
I don't parse that sentence as "you're dishonest if you don't pay for Caddy", but rather "be honest about the value it provided." It's still a free project, and I don't see the problem here at all.
Caddy is promoted as an HTTP server. If you focus on that feature rather than the cloud of extra stuff that can make life easier for some (which are orthogonal to what HTTP server does), this is exactly what Caddy does.
Sure, I'm not saying they should have written their HTTP server from scratch. But the way they're selling is apparently confusing enough that some people think Caddy developers are actually writing an HTTP server. net/http (and other go libraries) does the real work here, and they sure deserve credit (which they or you don't give in a visible way) if we're going to talk about fairness or paying for honest work.
Let's see about those features. You don't need any configuration script syntax when you use net/http directly, you can simply marshall/unmarshall your config struct as a json file. I'm using Let's Encrypt and renewing the certificate is as trivial as running a single bash line.
So I still don't see anything I need in actual Caddy code that I need and can't implement easily on my own.
> Caddy allows you to write your own extensions/middleware, which should cover all that. In turn, you get to re-use all the existing extensions and other things Caddy takes care of (which, in my experience, you'd inadvertently end up reinventing at some point anyway).
Why on earth would I do that when I can write Go directly? This has been discussed many many times, and the community has been clear and loud about web-frameworks in Go. For most, net/http and standard library, maybe with a few libs from gorilla, is the best way to write a server.
> I don't parse that sentence as "you're dishonest if you don't pay for Caddy", but rather "be honest about the value it provided." It's still a free project, and I don't see the problem here at all.
No, it's not a free project. Not according to them: https://caddyserver.com/blog/is-caddy-free
No need to weasel around words now, the words have been said. This is not something taken out of context. Go read that whole announcement. Regardless of how and why you want to sugar coat it, that is what they said, that is what they wanted to say. And after criticism, they since removed the word "honestly". To add to the insult, they're not "honest about the value if provided" them in the first place, since they're giving any portion of their income back to Google and other developers (they're using blackfriday etc.).
This was their pitch when they called their free users dishonest:
> There's no such thing as free software. The question is, "Who pays the price?"
Being a FOSS project is something. Telling people is morally wrong using their FOSS project without paying and speaking all high and mighty, while making profits from others' FOSS code and not paying them back a single cent is more than questionable. This is in particular true when it is the 3rd party code (go's stdlib) which does the real work.
It's amazing how you claim you're not seeing the gigantic problem here.
To summarize, the problem is two-fold:
1) They're morally trying coerce people into paying a software which they chose to publish under Apache License. While this seems contradictory, they're trying to justifying this with: There's no such thing as free software. The question is, "Who pays the price?".
2) Somehow, the moral code they want people to obey doesn't apply to them. This is in particular disturbing if you add the fact that the code which does the heavy-lifting (the actual HTTP server that Caddy uses) is written by a 3rd party, with whom they don't share any of their revenue. Of course, the "Who pays the price?" argument also doesn't apply to other 3rd part code they use to make profits.
Both has parallels in the brief and colorful history of Elementary OS: they called their free users cheaters while not paying a single cent to the upstream or to Debian. They tried to weasel their way out by saying "cheating the system" doesn't mean "cheaters"` etc. (akin to your attempt above). But the words had been said, and everyone who read the post knew what they said. People were rightfully upset. They edited their blog post to remove the wording "cheating the system".
Does every Rails app owe the core project, gems used, database developers, os developers, etc? How far down the OSS stack do you go? Where would your career be if you never leveraged a single bit of OSS code to generate profit/value for yourself or your employer?
https://twitter.com/mperham/status/708036799285735425
The strategy to try and earn a living wage offering 20% of the value as Mike Perham suggests is a sound one. People have used this strategy for as long as the internet has been around. If the Caddy team is wrapping Go to try and offer that 20%, more power to them.
As far as I can tell it is more likely the package maintainers will backport fixes to their old version in those cases.
Only if they need to do a perfect job. But as history tells us they are just as content of making a ho-hum job.
1: https://www.clay.fail/posts/ubuntu-http2-in-mere-hours 2: https://www.clay.fail/posts/hip-http2-using-docker/
Was there a bug or technical limitation that made you switch to a Docker image or just preferences/ease of updating?
For me, the first HTTP/2 POST request to a server running the affected version of nginx would never leave the browser and Safari left a less than descriptive "Failed to load resource: Could not connect to the server." console message.
I'm having luck with 1.9.12 + OpenSSL 1.0.2h
I may be wrong, but this is not a problem with dynamic linking but with OpenSSL as a project.
Because we don't – look at how many updates Debian, Red Hat, etc. have shipped for OpenSSL vs. the number of times you've had to recompile dependent packages.
OpenSSL is also something an outlier because it's both critically exposed for security and has a somewhat unusual development model, so it's also important to remember that even if it _was_ true that we have to rebuild OpenSSL callers regularly, that's manifestly not true of the hundreds of other libraries where dynamic linking saves time and memory.
For 1.0.1x->1.0.1y upgrades, dynamic linking means you just update the lib and restart running services, no recompilation needed. No random little-used binary hanging around with an old embedded static openssl, hopefully, as they will all pick up the patches as long as they dynamically link to the updated .so
[1] just kidding; I know distros don't have nearly enough resources to accomplish this
The "solution" here is just to wait it out for new major releases of the various distros, and not expect everyone to change core libraries every six months :)
If you only have one single app to host, I can see it's annoying to be stuck with last year's software. When you have dozens or hundreds of small websites and app backends, you really start to appreciate stable versions. :)
This hints at part of the problem - simply not enough resources. It's not that OpenSSL folks are slow or lazy (absolutely far from it) - but rather they've got tons on their plate already.
One day (hopefully) corporations will recognize that everyone can benefit by helping open source (either providing resources or $funding).
This only affects the nginx we use on the frontend for SSL termination and the rest of the OS remains unaffected, linking against the OS OpenSSL.
Been delving into hairy problems at work and yup, it's openssl's fault. More specifically it's the misguided way it handles multithreaded, and memory management. Yay.
At one point, a team I was working with had to code their own RNG replacement for OpenSSL's; the one in the codebase was behaving badly in a multithreaded environment. I don't know if the change was ever accepted back upstream; if I recall correctly, the initial response was something to the effect of "you're using it wrong" (as if multithreaded applications are some new and fascinating beast, and not increasingly just the way you do it to leverage all the performance a modern computer offers in hardware).
I recently released a RubyGem that does just that. It's unfortunate.
Not necessarily. You could simply statically link the newer OpenSSL library with your web server and everybody is happy.
Fortunately, it was pretty easy to redeploy my HAproxy config to Xenial and enable ALPN.
or
Poor support for OpenSSL 1.0.2 forces Chrome team to disable HTTP/2