Avoiding downtime: modern alternatives to outdated certificate pinning practices | Hacker News Reader

Avoiding downtime: modern alternatives to outdated certificate pinning practices

blog.cloudflare.com

45 pointscpach36 comments

Surprisingly low quality article from Cloudflare.

None of the solutions offered address the risk that pinning tries to mitigate, namely that you have a client who trusts a misbehaving/rogue/malicious CA (perhaps you installed a VPN or AV or something that added a system-wide CA to your machine?) and their connection to your service gets mitm-d.

Shorter certificate lifetimes - does nothing to mitigate that risk.

CAA records - not checked by client, only ensures that if a malicious actor somehow gets a hold of your domain, they need to use a particular CA to issue their publicly-trusted malicious cert.

CT monitoring - yes, for WebPKI certs, but does not protect against the risk above, as the bad CA-s are likely not sending certs to CT log.

Distributed DCV checks - sure, this is something CA-s should be doing, not something you have control over.

ACME URI in CAA - operational issues quite likely, good if you can pull it off, but again, does not address the risk above.

What is an actual alternative to cert pinning? Your API should have an additional layer of encryption with keys that you have full control over.

Yes

Also, the "Understanding the trade-offs of different certificate pinning implementations" misses one important thing : you can pin the CA, and still requires the cert to be valid for your FQDN

I've seen this a couple of time, and this is wonderful : fully secure, and denies the managed mitm setup by the security team.

And the only one time where I have seen this (in a private discussion of the mobile app with a bank employee), they said that they see no valid use case for a managed MitM against their production website. Preventing proxies like Charles and mitmproxy from working was one of their explicit goals for the app.

While I agree this is a low quality article in desperate need of an editor (there were large sections that were essentially just repeated), I'm not sure I agree with your analysis, especially as it relates to CAA records.

I think the fundamental issue is that all of the players in the PKI world want to essentially enforce that services must be able to deal with short-lived certificates and certificate revocations. As an example, see the recent HN article where Digicert tried to do mass certificate revocations due to a bug with their certificate issuance, and were then sued by a client (Alegeus) who did not have things set up to deal with revocations quickly, so Digicert postponed the revocation: https://news.ycombinator.com/item?id=41114794 . So basically, the powers that be are trying to say "certificate cycling is part of the process - 'set it and forget it' is a thing of the past."

Now, in your objections, you state "None of the solutions offered address the risk that pinning tries to mitigate, namely that you have a client who trusts a misbehaving/rogue/malicious CA", but CAA with CT monitoring should largely prevent that. With CAA monitoring you state "only ensures that if a malicious actor somehow gets a hold of your domain..." - well, if a malicious actor somehow gets ahold of your domain, you're generally fucked in any case - if they've got control of your domain it's probably not a huge step up to get control of your signing cert.

I think your comment brings up good points to someone like me who's not an expert, but it still feels like the overall point that needs to be addressed is that "certificates should be expected to expire or be revoked at any instant" and certificate pinning breaks that.

There are ways a bad actor can get enough temporary control of your domain that they can issue a domain-validated certificate without actually breaching your service (BGP, DNS hijacks, Cloud provider vulnerabilities, etc). These cases you really need to catch with CT monitoring and CAA does help somewhat as well. And ACME uri pinning should mitigate this completely.

Edit: just to clarify, I agree with the article in principle, pinning public certs is just not a workable solution in the current TLS ecosystem and should be avoided, but the author of the article does not understand what attacks pinning is supposed to prevent in the first place, as the solutions offered do not offer the same security guarantees.

Thanks for the reply! Helped me understand your points better, and the larger issues at hand.

If you pin it, you may as well use self signed cert, with whatever expiration date you want. It has nothing to do with what browsers or popular CAs do - you may be the CA in this case. Not saying this is a good idea, just that it's the same idea as before (vide their "why now" argument and shortening rotation times).

Is pinning to certs signed by CAs you don't control, but that are public and trusted, beneficial over using self signed ones? I guess you won't (easily) land in CT log. Anything else?

This won't really work technically, because pins can only be installed after the client has formed a TLS connection to the server, so if you have a self-signed certificate, the client will refuse to connect. In principle you can override this refusal, but in practice browsers make it very hard. Moreover, this isn't what you want people to do, because unless you have some out of band way of distributing the certificate or its fingerprint (and users check it) then they are open to a MITM attack by anyone on the network on first use, not just by attackers who have a valid (but misissued) WebPKI certificate.

The article mentions that pinning is often used by mobile apps and IoT devices. In those contexts, you have an easy out-of-band mechanism to deliver the cert: baking it into the app/firmware binary. (The user generally receives this from the App Store, or in a cardboard box from a retailer, so no initial TLS connection to your server is required.)

If you're pinning the leaf certificate this way, really the only benefit I see of using a WebPKI cert is if you want to reuse the same API endpoint for a web app. Otherwise you're mostly getting a bunch of restrictions and downsides (information leaks from CT, revocation drama, etc) that don't make sense if the cert is hard coded in the client.

I was thinking about certificates distributed with the software itself, so extra bootstrapping connection is not needed. Although in some cases it could still be done, just using other, trusted and not pinned context, I guess.

But yes, for the specific narrow use case of web pages and web browsers, this means out of band installation into the browser trust store. And if we focus on this particular use case, what difference does it make if the cert is self signed or not?

I'm not following you. The way pinning (specifically HPKP) works is that you're supposed to use a valid WebPKI certificate, so the initial connection works, and then you send a header to pin that certificate (or more likely it's CA).

Pinning in mobile apps/IoT is usually done the way the grandparent comment suggests. The fingerprint of the key is hard coded into the app somewhere,[1] so there's no initial connection without it.

IMO, pinning only solves the corner case scenario where a public CA is compromised or issuing fake certs under the table. Everything else is make-the-blue-team-feel-good measures like trying to keep users from intercepting their own traffic, or actively owner-hostile effects like devices no longer working when the cert is rotated. I'm happy to see CloudFlare calling for it to be retired.

[1] sometimes it's more DIY than that, where the code will require that the issuing cert have a particular string in its name, etc.

WebPKI still has the "devices stop working if certs are rotated" problem, just on a longer timescale. If the user takes an old IoT device out of the box, and the auto-update server is using a newer root cert that wasn't trusted when the device first shipped, they'll have a bad time.

Any solution to this requires something that basically looks like certificate pinning. You've gotta rely on your server provider to use a dwindling set of "trusted by the original device firmware and also still trusted now" root certs forever (which gets fun when all of those certs are mandated to expire), or you've gotta create some other mechanism to establish trust on software updates (delivering them via unencrypted HTTP and then verifying an OpenPGP signature is a common choice).

Ah, yes, I was wondering if you have meant HPKP. But then I checked it's deprecated in FF, removed in Chrome and was never present in Edge or Safari. At least according to Wikipedia. MDN mentions it as "obsolete". So it doesn't look like something usable in practice these days, right?

The article mentions certificate pinning is used to prevent MITM attacks, but none of the alternatives listed actually prevent this class of attacks.

I’ve heard multiple times that “certificate pinning is obsolete.” Who is pushing this narrative?

Well, requiring a WebPKI valid certificate itself prevents MITM attacks. What pinning does is prevent MITM attacks in the case where the attacker has a valid (but misissued) certificate.

It's true that CT doesn't prevent this class of attack; instead what it does is require that valid certificates be public, thus -- at least in principle -- allowing for detection of misissuance.

Not really true. Please also consider the case where the user and the attacker are one and the same person trying to reverse-engineer your mobile API. Then the certificate is misissued from the viewpoint of the app developer but not from the user's viewpoint.

If you don’t want people to reverse engineer your software provide it on tamper proof hardware.

How would Facebook, say, in this case work? No longer available on mobile devices and you have to buy Facebook specific hardware?

I am. Pinning is a footgun with negligible real world security impact:

https://tprynn.github.io/2022/12/06/cert-pinning-bad.html

A few thoughts:

1. Almost all mobile devices used by adults to access their work email have provisioning profiles that allow trusted certificates to be installed by one’s employer.

2. Plenty of authoritarian counties require trusting CAs operated by the government. If you have users in those countries, they are vulnerable to snooping.

Your blog post makes it seem like users vulnerable to MITM attacks are in the minority, when in fact they are likely in the vast majority.

How I'd like to replace certificate pinning:

* The client requests additional certificate chains

* The server returns multiple certificate chains (typically two), one chaining to a public CA, once chaining to a private CA

* The connection only succeeds if all expected chains are present and valid

* The public-key of the leaf certificates must be identical

This means:

* The public chain can follow standard practices around certificate transparency, renewal, revocation, expiration, etc.

* Clients which don't care about pinning but expect a standard public chain will work

* The chain to the private CA protects against a compromised public CA for clients which request/validate it

* Requiring identical public-keys means that this only impacts chain validation, not the key-exchange

I have a question. How does the client learn about certificate update?

I understand the "leaf" or "domain" update is done when the request is made, and that I as the end user don't need to do anything.

However if an intermediate certificate is updated, do I now need to update the app or client (OS/browser etc)?

Because I can sure see it in Google's interest to ensure clients only use the latest browsers, OS'. I suppose it becomes another "control" point.

If your client cannot resolve a domain (outdated hardware/software) you're out of luck.

If Google pushes changes to, for example, data collection within Chrome, they're going to want all users to update ASAP (e.g manifest V3, break older versions of chrome by rotating certificates).

> I have a question. How does the client learn about certificate update?

This is what I want to know as well especially with respect to many applications running in Kubernetes.

The client must be upgraded when the certificate changes

I've dealt with cert pinning in the past, and the most frustrating part was invisible pins - any client can pin your certificate in their code, without ever having told you about it. If you run a large enough API over TLS, always assume that some random client will have pinned your certificate at their end. The only way to deal with it is to setup fast rotating certs, so these enterprise clients just give up, but that might not always be possible

We ended up doing slow brown-outs over a long time period to deal with it.

I got Certificate Transparency notifications for a domain of a relatively small company for a while. After you setup custom domains for a few SaaS products, and have a few hosting providers issuing short lived certs, you’re getting notifications fairly regularly. The notifications don’t really come with enough information for you have to any idea if the certs are being issued by trusted parties.

At a large company I have to imagine that monitoring them would just end up being noise.

> However, what we’ve learned is that almost all customers that were impacted by the change were unaware that they had a certificate pin in place.

All that lengthy blog post and no explanation of what configuration is causing this and in what kinds of software. And how to find certificate pins in your infrastructure.

Anyone know?

Well, it's on a software component by component basis, so not much they could have done here, I guess. Look for any settings or in-code references related to ssl/tls. You then may find references to some local paths, pointing to certificate files. Or to cert strings compiled into the binaries. Depending on the details, this may indicate pinned certs.

This is a super disappointing read coming from cloudflare. Even if I didn't disagree with the author about who exactly 'caused' these outages (I'm not blaming the users for the breaking change). I have to disagree with the suggestions. If the goal is availability; why is automation and constant cert and key rotation preferred? It's better to have to get it right every single time, over getting it right once? That's a wild take. I guess it's the behavior and recommendation I should expect from cloudflare, they are after all looking to sell, what's effectively a maintenance contract in concert with their expertise. But if I was gonna buy cloudflare, it would be for their expertise. They're industry leading for a reason. They used to be right.

So I guess it's disappointing not only that they're wrong here in this article, but it's also disappointing they'd sell off some of their credibility earned by their expertise to trick people into an implementation with a much larger maintenance cost.

Anyone have any alternatives to cloudflare, is there anyone who still values correctness over making money?

Every single CA - including non-profit CAs like Let's Encrypt - have made mistakes that require cert rotation.

And sometimes it isn't mistakes that require cert rotation, but cryptographic breakthroughs that render the algorithms in the cert insecure.

It's much better engineering to accept that certs will need to be rotated and make sure it happens regularly so it's not a huge emergency when it's required.

> And sometimes it isn't mistakes that require cert rotation, but cryptographic breakthroughs that render the algorithms in the cert insecure.

Do you have an example of this happening? I know I remember a few cases that sound similar, but weakening of a signature, or key is very different from it breaking.

I'll also point out, if you find the signature made by a CA is weakened such that it can be forged; certificate pinning would actually protect existing users from such an attack.

> It's much better engineering to accept that certs will need to be rotated

that's a very debatable assertion, but I don't disagree outright

> and make sure it happens regularly so it's not a huge emergency when it's required.

No, that's not good engineering. Do something needlessly, so that it's easy to do it when it becomes required is operations not engineering. Engineering would be designing a system that requires the minimal amount of operational support. Certs do need to be updated, and maintanted, they don't need to be rotated.

512 bit RSA used to be common in publicly-trusted certificates. That's very easily breakable on today's hardware. If quantum computers happen, all existing keys in the WebPKI will be broken.

> Do you have an example of this happening? I know I remember a few cases that sound similar, but weakening of a signature, or key is very different from it breaking.

Flame (back in 2012) was a pretty famous example of certificate forgery due to weak hash choice (MD5) and other operational weaknesses in the PKI (predictable serial numbers, if I remember correctly).

> Do something needlessly, so that it's easy to do it when it becomes required is operations not engineering.

I don't think that's a good interpretation of what 'agwa meant. I'm pretty sure he meant that it's considered good engineering practice to build systems that don't encourage the normalization of deviance, which long-lived certificates combined with manual rotation procedures historically have.

Or put another way: automatic rotation procedures ensure the isostability of a PKI, rather than its brittle stability.

cloudflare continues to destroy the free web, this time around they are gaslighting you into thinking that an extremely important security feature, that happens to break cloudflare mitm is actually useless and outdated and there's all kinds of solutions, that incidentally all rely on trusting cloudflare, that you should be using instead.

Can we please just take X.509 out and shoot it?

Reply on news.ycombinator.com