Optionsbleed – HTTP OPTIONS method can leak Apache's server memory
blog.fuzzing-project.org
blog.fuzzing-project.org
The flaw only occurs if you have a "Limit" configuration that specifies an invalid HTTP method. For some unknown reason, this causes the leak. A "Limit" with a valid method, or no "Limit" at all, will not provoke the bug. I have to imagine that most Apache setups either don't configure this or configure it only with valid methods.
The article points out that shared hosting setups which allow individual users to customize settings through .htaccess would be vulnerable, since anyone on the machine could set an invalid configuration and then use the vulnerability to potentially get data from other users.
Obviously you should patch it when you can, but it is probably not time for most of us to panic.
<LIMIT GET>
Allow from x.y.z.v
Deny from all
</LIMIT>
for I in `seq 1 100`; do curl -sI -X OPTIONS http://myhost/ | grep ^Allow; done | uniq
Allow: ,GET,HEAD,POST,OPTIONS
Allow: GET,HEAD,POST,OPTIONS
Allow: ,GET,HEAD,POST,OPTIONS
Allow: GET,HEAD,POST,OPTIONS Allow: �E2�~,HEAD,,ex.html/,OPTIONS,POST,,HEAD,,HEAD,Pek�~,HEAD,,,ex.html/,GET,HEAD l-lbiegaj ~ -> curl -sI -X OPTIONS http://mysite/ | grep ^Allow
Allow: allow,HEAD,GET,HEAD,POST,OPTIONS
> CVE-2017-9798. -> https://nvd.nist.gov/vuln/detail/CVE-2017-9798
The CVE is still in RESERVED state. I'm frequently concerned about "responsible disclosure" in such cases.
Not sure how that makes you concerned about responsible disclosure. As you can read in the bug report the apache security team was informed (actually it was one of their devs figuring out the details of the bug, I just saw it "in the wild", but at first was unable to reproduce it locally). I had also informed the distros mailing list a week in advance.
I'm aware responsible disclosure is a weakly defined term, but if you have any concerns that I did something irresponsible I'd like to know.
That's just my opinion. Whether or not that's the "definition" of responsibility is of course up for discussion.
Either way of looking at it, delaying release of vuln info to wait for the above to happen seems like it would let vulns keep in-the-wild-and-unknown for much longer.
That being said, if you're able to assist with or fix the not-enough-time-to-get-things-done problem, it wouldn't hurt. :)
Yes, there are a few things apache is remarkable about - the npm-itk engine could run as user processes, which was nice for shared hosting - but all the problems with it - slowloris, anyone? - simply outweights the benefits.
Shared hosting should be containerised or vm-ised instead and run on nginx/lighttpd/cherookee/etc + php-fpm 7.0 which will save you a lot of headaches and CPU cycles.
.htaccess can be just as much a benefit as a curse.
(Note: WordPress is completely nginx + php-fpm friendly.)
Apache is far from perfect, but among its ilk it is probably the one of the more robust and well tested. It's not as if you don't have to care about connection hogging attacks such as slowloris with other http servers.
The cycles saved on the nginx http parser, which is completely irrelevant to the performance of any real world web application, comes at the price of being hand optimized and opaque. You can't really argue for that type of performance engineering at the same time as you argue for solid security. The world generally needs more secure software, not more performant protocol parsers.
Do also note that nginx had a very similar data leak bug not even two months ago in the range header.
I never understand these arguments that basically pit Apache2 on its own/with modules against a completely modular arrangement of tls/cache/web/app as different layers.
If you use apache as the "web" in that setup, it needs relatively few modules loaded and when configured correctly can be blazing fast.
Nginx is surely the first choice if you're rolling your own shared hosting infrastructure, but most legacy commercial shared hosting automation frameworks still use Apache and customers don't care what the underlying tech is serving their bits over HTTP.
I'm really iterested on the whys (in addition to the obvious legacy reasons).
Besides Apache and nginx, there aren't really any servers I'd trust for production use. Conceptually, I really like what Caddy is doing, but it's simply too young, not to mention the concerns that came up when Let's Encrypt had an outage back in May: https://github.com/mholt/caddy/issues/1680
1) Caddy has the most robust OCSP implementation of any web server. It caches staples locally and refreshes them halfway through the validity period so it can endure days-long OCSP responder outages.
2) Even if a certificate needs to be renewed while the ACME server is down, Caddy can endure 2-to-3-week-long outages of the ACME server because it renews 30 days out from expiration and tries twice per day until it succeeds, logging its actions along the way.
And because Caddy is written in Go, memory leaks like what Apache suffers are much less likely, if not impossible.
(Source: I implemented it.)
Two calls to OpenSSL and a few lines of shell script will fetch staples from LE whenever I want (ie more often than your defaults that have caused problems before) for HAProxy to serve. How's that for robust.
>2)...
great so you're as reliable as.. any other LE client that tries to renew before expiry, but without the ability to choose when to renew certs?
2) Ours is the only LE client built directly into the web server, so "reliability" isn't really a comparable factor. By doing renewals automatically, Caddy's HTTPS implementation is more reliable and robust than doing it manually.
Ok fine, you want to pretend HAProxy isn't a thing that exists, we can do that.
> 2)
That isn't necessarily a good thing. Have you ever heard of separation of concerns?
You specifically mentioned how Caddy can manage 2-3 week outages, as if it's something unique to your software.
Every LE client that can be run from cron/a systemd timer renews certificates autagically.
So where's the manual part that you keep claiming is not "robust"?
Failure to renew a certificate (for any number of reasons) and missing intermediate certificates are the most common issues, and both of these are essentially solved with a web server that supports ACME natively. The same applies to OCSP stapling - we're not going to achieve ubiquitous stapling support on the server side if every server admin needs to deploy a bash script calling to openssl because the native OCSP implementation in their web server is broken.
Yes you need to fetch them regularly. I use the aforementioned shell script to do this.
Why do you people keep up this weird narrative that only caddy renews certificates automatically?
The native ocsp handling in HAProxy isn't broken: it just expects to have TLS material provided to it.
If someone setting up HAProxy on a server can't install a shell script calling two OpenSSL commands and setup cron to call it regularly, I don't think your big problem is "HAProxy doesn't handle ocsp". It's PEBKAC.
> Yes you need to fetch them regularly. I use the aforementioned shell script to do this.
It does sound like there's some configuration involved after all, namely the configuration of a cronjob, and perhaps adjustments to a bash script. This is something that the majority of server operators will not bother with, though these numbers might look slightly better for HAProxy since that's generally something you see used by more experienced sysadmins. Even with nginx, we're talking about one line of configuration for the actual stapling file - that's not the blocker, it should just work, by default. Fun fact: IIS does this.
> Why do you people keep up this weird narrative that only caddy renews certificates automatically?
I'm not sure who "you people" are, and I never made any such claim. What I'm saying is that web servers that support ACME natively leave a lot less room for the types of misconfigurations that I commonly see. I'm aware of an ACME module for apache and lua scripts for nginx which provide similar features (regarding renewal and certificates, not for the OCSP aspect), but they're not part of what you run by default when you use those web servers.
> The native ocsp handling in HAProxy isn't broken: it just expects to have TLS material provided to it.
I have yet to see you explain why this should be necessary. The web server has all the information it needs to do this. Other web servers can do this.
> If someone setting up HAProxy on a server can't install a shell script calling two OpenSSL commands and setup cron to call it regularly, I don't think your big problem is "HAProxy doesn't handle ocsp". It's PEBKAC.
This is like arguing MongoDB should continue to bind to all interfaces by default because everyone should have firewalls in place. Yes, they should, but it's never going to happen, so why can't we just have a safe default?
How is "schedule a script to run @daily" or whatever schedule you want, at all match up to what you suggested is required:
>> You should not need the amount of specialized knowledge you currently need to correctly configure OCSP on HAProxy or nginx, for example.
If you're configuring a public facing server, a cronjob should not be considered "specialised knowledge".
> it should just work, by default
HAProxy and nginx do a fucking shit ton more than Caddy does. HAProxy doesn't really have a "default" you have to tell it what you want to do, specifically. This is a good thing.
> I'm not sure who "you people" are, and I never made any such claim.
>> By doing renewals automatically, Caddy's HTTPS implementation is more reliable and robust than doing it manually.
>> Failure to renew a certificate (for any number of reasons) and missing intermediate certificates are the most common issues,
You're both giving the impression that Caddy is able to automatically renew certificates, while no other software does this.
> I have yet to see you explain why this should be necessary.
Off the top of my head:
- Separation of concerns. I don't ask my mechanic to build me a car, or to drive me to the shops in it.
- Not every organisation uses LetsEncrypt certificates. Some will even use self-signed certificates.
- TCP or DNS load-balanced environments will want the exact same certificate & key on multiple servers.
- Caddy has already shown that trusting some "automagic" solution can bite you in the ass. Remember that time Caddy wouldn't even start, even though it had perfectly valid certificate material in it's cache, because LE was down?
> The web server has all the information it needs to do this.
HAProxy doesn't know the email address I'd want LetsEncrypt to use for notifications. It also shouldn't be agreeing to LE's terms on my behalf.
> This is like arguing MongoDB should continue to bind to all interfaces by default because everyone should have firewalls in place.
No, it isn't.
I'm sorry but you simply don't get the choice of arguing anything about how creating a fucking cronjob is "too hard" when Caddy doesn't even ship with a configuration to start as a service.
"I know I just had to google the shit out of how to create a systemd unit, but fuck it, it's better than writing a damned cron job for oscp staples!"
Knowing how HAProxy's OCSP implementation works (i.e., by looking for a $cert.ocsp file) is the specialized knowledge. Odds are, you don't know about it and just won't use OCSP stapling. This is especially true since every web server does this differently.
> HAProxy and nginx do a fucking shit ton more than Caddy does. HAProxy doesn't really have a "default" you have to tell it what you want to do, specifically. This is a good thing.
I'm not arguing that it doesn't do a lot of things Caddy doesn't. I'm arguing for sane defaults. If I tell HAProxy to enable TLS and provide a certificate that contains an OCSP AIA extension, a sane default would be to use that information to manage OCSP stapling.
> You're both giving the impression that Caddy is able to automatically renew certificates, while no other software does this.
You're leaving out the part where I say these issues "[...] are essentially solved with a web server that supports ACME natively". I'm not referring to Caddy specifically. The web server doing this natively is far less error-prone. I have also mentioned alternatives.
> - Separation of concerns. I don't ask my mechanic to build me a car, or to drive me to the shops in it.
Where do you draw the line? Most deployments let their web servers handle TLS. OCSP stapling should be part of a modern TLS deployment. You could similarly argue that using HAProxy for TLS is wrong (and that you should rather put pound or stunnel in front of it).
> - Not every organisation uses LetsEncrypt certificates. Some will even use self-signed certificates.
That's fine; you can continue to manually configure certificates, even with Caddy, and I'm definitely not arguing that this option should be removed. ACME is also an open standard, so switching to other CAs can be as simple as configuring a different ACME server url.
> - TCP or DNS load-balanced environments will want the exact same certificate & key on multiple servers.
That's why you can replace the certificate and key storage in more complex environments[1].
> - Caddy has already shown that trusting some "automagic" solution can bite you in the ass. Remember that time Caddy wouldn't even start, even though it had perfectly valid certificate material in it's cache, because LE was down?
Software can break, what a surprise. You could similarly have a broken OCSP updater script that puts empty or expired OCSP responses in your stapling file because the check is missing or incomplete and break the site for your visitors. Or your ACME client could fail on renewal because of a backend change on Let's Encrypt's end that exposes a bug in it[2].
> HAProxy doesn't know the email address I'd want LetsEncrypt to use for notifications. It also shouldn't be agreeing to LE's terms on my behalf.
You can ... provide your email address? You keep arguing that this stuff is easy, surely adding a single line with your email address won't break the camel's back. Accepting the TOS can be solved as part of the installation or configuration process, much like ACME clients do it.
Anyway, you seem to be having some kind of problem with Caddy. I'm not interested in that discussion (which is to say I'm not interested in discussing this further, I think I've made my point) - my goal is to improve the user experience of server admins when they enable HTTPS, and I have no doubt that native ACME support and automated OCSP management, both with sane defaults, would both help HTTPS adoption and improve the quality of the TLS configuration of your average web server.
[1]: https://godoc.org/github.com/mholt/caddy/caddytls#Storage
[2]: https://community.letsencrypt.org/t/acme-sh-standalone-fails...
FYI, the issue a few months back wasn't "software breaks".
It worked exactly as they designed and implemented it. The original issue was closed, before (as seems to be a pattern lately) they changed direction when they received so much negative feedback about their highly opinionated choices for the project.
To this day the same logic is still in caddy, but with a longer grace period. Try starting caddy offline with a cert that expires in less than 7 days.
I don't trust someone to develop a server if they think failing to start at all, is better than starting with close to expired or even expired certificates.
https://github.com/icing/mod_md
this is being folded into the official Apache httpd repo.
https://news.netcraft.com/archives/2017/07/20/july-2017-web-...
There are lots of legit reasons to run Apache:
- it works like a champ
- it's widely known
- it's got a big ecosystem (plugins, etc.)
- it has a fairly complete WebDAV implementation
- it's widely supported
TL;DR "legacy" sometimes means "good" and not just "old."
In this case legacy is either "I don't know anything else", "well it worked so far" or "meh".
WordPress, for example. It can run on other servers, but their own docs call out the issue..."Since Nginx does not have .htaccess-type capability and WordPress cannot automatically modify the server configuration for you, it cannot generate the rewrite rules for you"
"With Nginx there is no directory-level configuration file like Apache's .htaccess or IIS's web.config files. All configuration has to be done at the server level by an administrator, and WordPress cannot modify the configuration, like it can with Apache or IIS"
What you are saying is, on it's own, true. It's also a solved problem for WP.
Its "killer feature" are htaccess files. It provides a possibility to give users a limited way to configure their virtual hosts, without giving them too much power. (Unfortunately there are some unfixed systemic security issues with that, namely symlink attacks...)
Apache httpd in many ways is a much more complete web server solution compared to, say, nginx (which I like as well). I'm prefering Apache httpd for the depth of available documentation alone, because everything is developed openly, C source is accessible and not hard to develop with, and because of the breadth of available options (certbot/letsencrypt/acme reference implementation on Apache, WebDAV, clustering etc. etc.).
As far as stats go, according to netcraft MS's IIS is leading with Apache second and nginx a distant third. Of course, this data isn't terribly reliable as netcraft can only see front ends, which may hide whats behind the load balancer, but IIS/Apache are still kings.
(And I'm writing this as a dev who worked on a competing web server which IM-biased-O was vastly better :)
Mostly I use it because, after probably 20 years or so, I'm quite comfortable with and accustomed to it.
That said, I have started putting a few of my own web sites on nginx just to try it out a bit and get some familiarity with it.
What your vendor supports and what you are most comfortable with configuring should decide which one you run.
PHP, as you say, is something else entirely. It's so much slower than your http server so the latter doesn't matter at all.
It's kept up to date and it works for everything I want it to work with. This idea that because it's old and not hip therefore it's use is questionable is kinda dumb. I was disappointed when OBSD wrote their own httpd that was initially filled with stupid bugs.
If it ain't broke don't fix it.