Hacker News Reader

Comment by dmix | Hacker News Reader

Banks can just switch to better SSL services...

ha.ha.ha.

I worked at a financial institution for several years. There are many, many IT folks, internal auditors, and others who are probably wishing they wore their brown pants to work today.

SSL certificates are cheap in contrast to the labour intensive management practices that exist around them, especially around legacy platforms that may have been hardcoded to use certificates from a specific issuer (not that I have ever seen that before, no one would be that foolish right? :/)

>especially around legacy platforms that may have been hardcoded to use certificates from a specific issuer (not that I have ever seen that before, no one would be that foolish right? :/)

D'ya know, I would have naively assumed this wasn't technically possible. I shudder not only to think of the code, but also of the thought process that could compel someone to undergo the effort of bricking themselves into this corner because honestly, that sounds a lot harder than doing it right.

It happened to me - I was working on an application for the pre-paid electricty system in Texas: the server-side code connects to a data-source over a TLS connection (complete with client-side certificates too), except the server-side used a self-signed certificate, and my code didn't have admin/root rights on the client hardware so it had to use in-app certification verification code. I had to hard-code the root CA's fingerprints directly into my code so it would be baked into the signed binary (this was a project requirement). That thing is going to fall-apart if the root CA ever changes - fortunately it won't expire until 2099.

(Yes, I gave the client a strongly worded statement of my misgivings - and all the other bad code-smells in the project)

So basically you're pinning the server certificate chain into your signed client code? Actually that sounds like a good thing!

https://www.owasp.org/index.php/Certificate_and_Public_Key_P...

Except that that likely ended up baked into a system that is hard or impossible to upgrade. So when the certificate is revoked or changed the system stops working.

Pinning is a good thing, assuming you have built in a reliable upgrade path (oh hey, browsers update themselves automagically now so yay!) but when you have occasionally connected devices that have low bandwidth that have a long deploy lifecycle, it's not always feasible, or even a good idea.

It's even worse if said device is embedded and has severe hardware constraints for compliance and regulatory reasons, and those same reasons prevent you from deploying patches on the regular.

Layer in some "built by the lowest bidder" and "accumulation of 30-odd years of poor practices for smart payment cards" and you have a big mess of a situation to unravel.

Oh yeah, and lest you think this is not an issue because this is browser related, may I introduce you to the world of services that proxy legacy green terminal applications into web front ends, ranging from IBM HATS to bespoke AJAXy things that unwrap screen scraped terminal emultor content from XMLHttpRequests.

Jeez. Remembering why I used to drink more when I worked in fintech :)

Hardcoding certificates is actually way too easy. SSL libraries don't necessarily use the system ca store or even know about it. OpenSSL has the option of disabling certifcate validation, providing your own certificate list or pointing to some system-supplied certificates which you need to find first. So in a way you even have to count yourself lucky if the hardcoded one instead of choosing to just disabling validation.

If you're hard coding certificates in your own client software, this issue doesn't affect you.

It's pretty common in the embedded world where you don't have enough flash space-- or don't want to use what flash you have-- to hold a root CA store.

Such devices are usually great targets for learning how not to secure things, because when they care that little they often don't get anything else right either.

This is essentially what HTTP Public Key Pinning (HPKP) does [0]. HPKP is a header your web server responds with that tells the client's browser, "Only trust my domain if the certificate presented is in a chain that goes up through one of these CA public keys". This is so an attacker can't go buy a certificate from a less-diligent CA and use that to MITM HTTPS traffic between clients and your domain.

You pick two or more CAs you trust, and brick yourself into a corner saying "these are the only CAs you should ever trust w.r.t. this domain".

[0] https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning

A number of people recommend doing that as best practice.

https://raymii.org/s/articles/HTTP_Public_Key_Pinning_Extens...

suddenly everyone gets hauled into a change control board meeting to chance a cert on an F5

Well, you guys here love capitalism, and this is capitalism.

If you make such astonishingly obviously poor decisions, you deserve to fail.

The problem is that kludges like this (Symantec's handling) work perfectly up until the point they explode.

And market consumers lack the technical ability to discriminate between anything more finely grained than "works" / "doesn't work."

Any large organization lacks the ability to "just switch" from one thing to another.

I'm fairly sure at least some of the "policy violations" that Symantec did were done exactly as a service to their large bank-or-close-enough customers.

It's not that banks want to switch to a "better" SSL service, it's Symantec being a "better" service for them that got Symantec into trouble.

(IIRC, from reading some of the incident reports)

Can you name some specific examples?

IIRC several of the mis-issued SHA1 certs.

Symantec took one of their widely trusted root certificates and declared that it was now "off the reservation", meaning they may choose to not comply with the BRs for its leaf certificates.

I don't know if they have actively used it to issue SHA-1 certificates, but they certainly could.

You will notice that was also their SGC root, and one of the oldest roots that browsers trusted.

But how or why was this done as a favour for their banking customers?

Banks and companies like First Data were the (sole?) source of exception requests for SHA-1 issuance past the date it was prohibited.

Given the G1 root they pulled has an intermediary called "Symantec Trust Services Private SHA1 Root CA", I can make some guesses...

The exception process used by payment processors such as First Data were for SHA-1 certificates chaining to a root that was still publicly-trusted. They couldn't use an off-reservation root because their client devices didn't trust them.

The roots that Symantec took off-reservation are regularly issuing SHA-1 certificates to anyone whose check clears. Got $1,699 to spare? https://www.thesslstore.com/symantec/secure-site-pro-sha1-pr...

I may be being a bit dense, but if your client device contains a root that goes off-reservation, how does it ever receive a revocation notice? Doesn't everything that chains to that root via a valid chain still get trusted?

Side note: Are payment systems required to link to revalidate their roots at any regular intervals?

> I may be being a bit dense, but if your client device contains a root that goes off-reservation, how does it ever receive a revocation notice? Doesn't everything that chains to that root via a valid chain still get trusted?

Yes, if a client isn't receiving root store updates it will continue to trust certificates chaining to the off-reservation root. This is why taking previously-trusted roots off-reservation is bad for the ecosystem and would ideally be prohibited.

> Side note: Are payment systems required to link to revalidate their roots at any regular intervals?

Many payment systems apparently have no automatic update mechanism, so I assume there is no requirement for such.

Wow, the two of those combined with embedded systems with extended lifecycles seem like a recipe for disaster.

It really is :-(

Indeed, I was just speculating that more banks and payment processors might take advantage of the off-reservation root if it was possible, given the type of companies that publicly showed they needed one.

I agree.

However, there's no point in pretending these certs are good if we can't trust the issuer to not put out BAD certs. Painful or not, if the banks are user the certs to show they are trustworthy, then they need to switch when the issuer ISN'T trustworthy.

Not always, although it depends on what you are changing. I work for a 200,000 person company, and switching cert providers would not be a huge task. We have done it before.

Somebody needs to tell Bank of America. They're still presenting a Symantec EV cert.

So is Chase, Citi, PNC, Schwab, TD, ...

Reply on news.ycombinator.com