Do you check HTTPS certificates in your API clients? | Hacker News Reader

Do you check HTTPS certificates in your API clients?

unfoldthat.com

114 pointsva1en0k72 comments

When you validate server certificates from HTTPS clients, please be sure to use the right set of root certs. Mozilla maintain a decent list of these [1], but it's not in the PEM format that most HTTPS client libraries expect, e.g. Python's ssl.wrap_socket(sock, ca_certs="certs.pem").

Mozilla's list also includes distrusted certificates, so you need to be careful to leave them out when generating the PEM-encoded format. In fact, I'd strongly recommend using Adam Langley's excellent extract-nss-root-certs tool [2] which takes care of the subtle details for you.

And, if you are willing to trust me, you can download my pre-generated PEM-encoded cacerts file from a month or so ago [3].

[1] https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw...

[2] https://github.com/agl/extract-nss-root-certs

[3] https://github.com/downloads/tav/ampify/distfile.cacerts-201...

I'd actually recommend just get the exact certificates you're going to work with. For example, if I'm using GitHub's API, it makes sense not to check all the chain, but just keep the GitHub's original cert.

Disclaimer: I don't particularly agree with the Certificate Authority mechanism that we currently use with TLS.

However, given that it's what we currently have, I'd strongly advice taking advantage of the security that it provides. Requiring API client library authors to ship certs will make for poor security. Not only do certificates expire, they also get compromised.

It would be easy to conduct MITM attacks using revoked certs and API client library users would be none-the-wiser. Instead, it should be the responsibility of HTTPS client libraries to use the latest cacerts data and support features like OCSP [1] for validating certificate revocations, etc.

[1] http://en.wikipedia.org/wiki/Online_Certificate_Status_Proto...

Taking a validated known-good cert and, from that point on, simply verifying that the cert remains bit-for-bit identical is called "certificate pinning", and while it does create problems with revocations, your service is going to break as soon as Github revokes their certificate anyways, so it's not like you won't notice.

The problem is that the CA mechanism (in particular, blindly trusting whatever list of root certs your vendor ships) does not provide security. I cannot go and validate the practices of those CAs, and all it takes is one of them to get compromised (which they regularly do, have you taken out the compromised CA's root certs that ship with OS X, for example?).

The only certificates I actually trust are the self signed ones from my organization which I can actually go validate in person. While I have ZERO trust in any of the certificates that my vendor ships.

the problem here is rather than worrying about one particular cert getting compromised, you now have to worry about every CA in the world getting compromised, a much more likely possibility.

It seems the best course of action would be to trust only an individual cert, and check for revocation.

Also OCSP is basicly a joke, it works every single time, except when it matters (an attacker controlling your view of the world)

Yeah. I'll edit my post to add your points, but I can't do it right now

The problem here is that if your app is deployed in a corporate environment, it's possible (likely) that the corporate firewall is intercepting your HTTPS traffic and returning a different certificate, issued by the IT department.

So if you try to validate that the certificate is the specific one that your API server is using, it's going to fail in that scenario.

Depending on your app, you could just ignore that possibility of course.

How does your scenario differ from a man in the middle attack?

The word "attack". In theory, being in a corporate environment, this is a desirable man-in-the-middle rather than a hostile one. It certainly is man-in-the-middle.

It's exactly what SSL/TLS is designed to defend against, and neutering your apps & applying newspeak doesn't make it preserve the security provided by SSL/TLS.

Conceptually, in a work environment you aren't accessing the website, your company is. If your company chooses to add an SSL proxy for its own purposes, there's nothing invalid, wrong, or unethical about that. Conceptually, you're all functioning as one entity.

You may note I'm using words like "theoretically" and "conceptually" in these replies, and that's basically because ctz's point is accurate. It isn't hard for someone on HN to be more competent at SSL usage than the administrator of the SSL inspector. But, well, welcome to the corporate world. Can't live with 'em, can't live without 'em. But I don't think it's wrong on any moral or technical level, it's just potentially wrong based on more mundane considerations, like competence.

Nobody is saying it is. However, this is the reality in a surprisingly large number of corporate environments. I have to support a lot of enterprise customers in my day job, and working around corporate firewalls is a large part of the issues that come up for us.

So you still need to provide some mechanism to prevent malisous attacks. The point of validating certificates is to prevent this class of attack.

Yes of course. In a corporate environment, you would usually install the proxy server's CA certificate in your certificate store and validate that all certificates were issued by the proxy server.

My original comment was just pointing out that validating that the certificate you get when you connect to https://www.github.com in a corporate network may not be the same as the one you get on the open internet.

It's up to you to decide whether that's something you care about though.

Yes, but you're then reliant on the proxy to do the validation for you.

And for the proxy's private key (which you are henceforth relying on for all transport security) to be kept secure. Given my recent experience with products like websense, this is a very poor bet to make.

If GitHub's cert is revoked or expired, you'll have to manually go grab the new one. You want to trust the issuer of the cert.

That's great until it changes.

If you trust the cURL folks, there are regular conversions of the Mozilla root certificates available[1].

There's a Perl-language converter tool available there, for those that don't have Go handy for the extract-nss-root-certs tool.

And there's also a shell script there that uses the certutil tool [2] that can be used to extract the Firefox certificate store.

[1] http://curl.haxx.se/docs/caextract.html

[2] http://www.mozilla.org/projects/security/pki/nss/release_not...

Why working so hard, when M2Crypto [1] will do everything for you [2] ?

If you're doing anything serious with Python and SSL, you're going to use M2Crypto - period. Because when it comes to security, the less you "roll your own", the better.

[1] http://chandlerproject.org/Projects/MeTooCrypto

[2] http://svn.osafoundation.org/m2crypto/trunk/demo/x509/certda...

Sadly, that M2Crypto script doesn't check for certificates which are not trusted for issuing SSL server certs. So whilst it happens to skip a few, it will include over a dozen inappropriate certs in the final output!!

This is exactly the problem that https://github.com/agl/extract-nss-root-certs was written to solve. I'd strongly recommend using it.

In this age of high-level languages, why do I still have to worry about this? I don't mean 'security' I mean 'managing certificates.' My local framework/API should complain if I don't have a trusted root and should then make it dead simple to provide that root.

Because during most of the development cycle when these libraries are being used, the certificates aren't validating (they're dev/test/UAT systems) --- and so during actual development, certificate verification seems like just another annoying obstacle to clear as quickly as possible.

It's easy enough to generate a snakeoil cert and use its public part as your cert bundle. On Debian/Ubuntu, just install the ssl-cert package, and point to /etc/ssl/certs/ssl-cert-snakeoil.pem as your certificate bundle; the private key is at /etc/ssl/private/ssl-cert-snakeoil.key .

Or just serve your app/API/etc from different urls, plain HTTP for development and HTTPS for preprod and production.

That's exactly how it works in C# (.NET in general), and I would take a fairly large bet that's exactly the same in Java. It seems like a regression (when thinking on terms of lower level --> higher level language movement).

Exactly.

Security is something that should just work - arguably, the first thing with such requirement. Even though it simply can't just work in many cases, it should be as close to just-working as it can.

Python does have some certificate-checking machinery:

http://docs.python.org/library/ssl.html#functions-constants-...

http://docs.python.org/library/ssl.html#ssl-certificates

However, this isn't exposed in the higher-level httplib.HTTPSConnection class for some reason. I'd bet it's not too hard to write your own subclass to handle it though.

If you're using Python for HTTP requests, you should be using Requests, which checks SSL certificates by default (as far as I can tell).

http://docs.python-requests.org/en/latest/

Yeah, I posted this below. It's mentioned in the advanced section. Why checking your SSL certs is an advanced topic I don't know.

This doesn't do any CRL or OCSP checks at all so you are still vulnerable to attacks using revoked certificates.

In Python, a sufficiently recent httplib2 will require valid certificates by default; same with the requests library. Both let you use your own cert bundle, as a quick-and-dirty way to do certificate pinning. That's reason enough to discourage the use of Python's standard library for http requests (another reason is that it's quite low-level).

Java uses a keystore (actually - being Java - you can set a property to tell it which keystore to use).

Importing a new certificate is documented here: http://java.sun.com/j2ee/tutorial/1_3-fcs/doc/Security10.htm...

The verification of the certificate is delegated to a TrustManager[1]. Certificate paths can be checked[2] and the TrustManager is flexible enough to support a large variety of verification scenarios[3].

[1] http://docs.oracle.com/javase/1.4.2/docs/api/javax/net/ssl/T...

[2] http://www.exampledepot.com/egs/java.security.cert/ValidCert...

[3] http://stackoverflow.com/questions/11681474/how-can-i-trust-...

If you want to check any 3rd party apps or frameworks that you use you can set up a proxy like Fiddler, configure it to do a MITM attack on you, and see if the client/API blows up with an error or just keeps on truckin'.

better than /etc/hosts approach, yeah

Getting this right was a big pain when we were implementing the various Stripe client libraries. We had to resort to hacks even in mainstream languages.

Well, having to use hacks for that is just plain wrong. I think we can consider this as bugs in these mainstream languages. Probably you could file them?

Python: Just check the certificates with Python Requests (or take one of 20 lines urllib solutions):

>>> requests.get('https://exmaple.com, verify=True) requests.exceptions.SSLError: hostname 'exmaple.com' doesn't match either of '*.exmaple.org', 'exmaple.org'

The requests library in Python will check your certs.

docs.python-requests.org/en/latest/user/advanced/#advanced

This will take you straight there: http://docs.python-requests.org/en/latest/user/advanced/#ssl...

Thanks. Did I miss the TOC or was that a hidden anchor?

Most sphinx documentation has a permalink if you hover over the title text. It will show up in this instance to the right.

Ahh. I couldn't figure that out to be honest, so I just viewed the source.

Why the hell is that advanced?

Nice to know, though. Thanks!

Changing which certificates are used and whether to check is advanced. The right thing is the default.

So presumably, decent high-level languages (like Python) do use an encrypted connection - they just don't ensure the right person is on the other end of that connection?

yep

however, encrypted connection without this checks allows, for example, Man-In-The-Middle attack

To make it worse in many languages there not built in OCSP or CRL facilities to go with their standard TLS wrappers. e.g. The best you get in Python is checking against a CA list. So even if you do go to the trouble to turn on CA verification yourself you still accept known bad certificates.

Great Article! Looks like as of Curl 7.10 PHP now checks this by default: CURLOPT_SSL_VERIFYPEER CURLOPT_SSL_VERIFYHOST

http://php.net/manual/en/function.curl-setopt.php

The problem with doing this is that many/most enterprise IT shops use security devices to proxy SSL traffic, altering the cert that the client sees.

So unless you really understand you app's deployment scenario well, you should proceed with caution. Also note that 2% of enterprise IT people understand SSL at all, (and the devices that do the MITM part are usually controlled by the security dept) so troubleshooting will be close to impossible.

Those proxies publish their fake CA=YES certs, so you can just add them to your root and everything will validate.

Java HTTPS server code allows the developer to configure the X.509 trust manager. My API uses Java for both client and server and transmits JSON messages over secure web sockets which is built over HTTPS. Works great. I authenticate on both client and server, and am just now developing an iOS Objective C client that runs wss.

Node.js seems to omit using a certificate as well. From the documentation:

* cert: Public x509 certificate to use. Default null.

[1] http://nodejs.org/docs/latest/api/https.html#https_https_req...

i know in the the world of .NET (atleast on Windows) it'll automatically check using the cert manager. Anyone know how this might work on platforms such as Java? I'd assume the local VM would need to have some OS specific plumbing in place.

The official Oracle/Sun JVM ships its own file-based certificate store containing the root certificates of authorities it trusts. It does not, at any time, interoperate with OS-wide stores, nor does it assume they might exist. This makes it fully portable, but of course adds a further burden in terms of maintenance.

Same for the JRockit JVM.

True, but if you're installing OpenJDK from Debian they have gone to the trouble of integrating it with the system-level certificate store, so it's much easier to manage.

The JDK comes with a program called key tool. You use key tool to build a "trust store". This is basically a collection of certs you trust. There is also a "key store" which contains your cert and its private key. Then when you run your app you must specify what key store and trust store to use.

I've seen Java .Net and Perl code fail with self signed or expired certificates, so I'm pretty certain they check.

many servers are deployed using Linux which doesn't have any cert manager (see citation in the blog)

But most distribution provide a list of trusted root CA. For exemple on Debian: /etc/ssl/certs/

what about with platforms such as Android? iOS? etc.

iOS and OSX both use the system store represented by the "system" keychain. (On the desktop, individual users can also have keychain a with trusted roots.). Apple keeps their root store up to date via software updates and automatic OCSP checks.

NSURLConnection, the higher level resource API, will by default require a valid certificate chain, but provides for explicitly allowing an insecure connection as part of its authentication callbacks. (This is an improvement over previous versions (leopard and before) where you had to explicitly specify hostnames that should be considered safe.)

So, certificate validation is treated, at the API level, like any other sort of authentication challenge.

You can also provide a client certificate using the same mechanism, if requested by the server.

Using the lower level CFHTTP stream API, you can only fail the connection and re attempt it after disabling cert checking.

Validating certificates is a good thing and everybody should do it.

That said ... it really only tells you that a certificate is 'sound'. It by no means tells you with 100% confidence that you are talking to the right party.

SSL/TLS is still pretty fragile.

The twilio-ruby library does this by packaging a set of root ca certs. There is a configuration option to override this with your own set when you construct the client.

This is default behaviour in Ruby, and as mentioned in the article, one can set up a custom store.

With all the languages that support this by default, I'm surprised Python doesn't.

NGiNX reverse proxy can take care of this and does so securely, am I right?

Nginx ssl reverse proxy does not verify certificates. I tested by reverse proxying to an ssl server with a self signed certificate. Nginx did not log any warnings or complain.

No, nginx is on the server side. The article is about client-side.

You can use nginx as a proxy on your side for any APIs. Just proxy_pass it to the API, and listen only on the local side. It can provide caching and other stuff that way.

Yeah, but that's proxy, not reverse proxy. It's possible to do something fiddler-like using apache or nginx to mitm oneself and dump everything that goes through.

Probably they can.

However, I don't know anything about using nginx that way. If you give me a link or something like that, I'll happily add it to my article. Or I'll just investigate later.

Yeah my bad I read your article assuming you were an API provider not a consumer. In the case of consuming API’s I suggest python-requests, an excellent library that is extremely well maintained.

Reply on news.ycombinator.com