Hacker News Reader

Comment by michaelt | Hacker News Reader

Traditionally it's difficult for browser vendors to revoke a root CA as they want to grandfather in old certificates, so existing sites don't have the rug pulled out from under their feet when their only crime is using a crap CA.

Partial solutions include blocking the CA's certs based on the issuance date or insisting they hand over a list of the certs they've issued - but if the CA is going down in flames anyway, they have no incentive to cooperate; they can backdate certs and destroy their own customer list.

My theory is [1] this is one of the side benefits of Certificate Transparency - CT will give browser vendors a list of certs to grandfather in if they decide to shut down a CA against its will.

[1] https://www.mjt.me.uk/posts/certificate-transparency/

View on HN

That's exactly the internet equivalent of too big to fail banks. And like for banks it is unacceptable. The internet needs its Lehman moment to become resilient again. We should let a Comodo fail.

So what is the solution to big banks and big CAs? We are just in the process of collectively creating the biggest CA the world ha sver seen.

With CT we could in theory revoke big CAs without impacting sites currently using them.

You need the CA to either have been cooperating with CT all along, or have kept a copy of all the certs it ever issued, neither of which an incompetent CA is likely to have done.

Symantec, whom one would expect to be one of the more competent CAs out there, cannot do this: https://security.googleblog.com/2015/10/sustaining-digital-c...

But Google did require them to use CT for all new certificates, which I think they are enforcing by assuming Symantec won't lie about the issuance date of new certs.

You may not be able to immediately spot a bad CA, but the more major CAs buy in to CT, the easier it is to spot fraudulent certs through fingerprint reporting. SSL clients can then choose to have a side-channel trust revocation mechanism. Once adoption snowballs, the overall level of trust in the system will rise.

The other problem is that CAs need to be regionally confined in which TLD CNs they can issue certs for.

> The other problem is that CAs need to be regionally confined in which TLD CNs they can issue certs for.

Ryan Sleevi makes a good argument here about why that's bad policy for the internet:

https://groups.google.com/d/msg/mozilla.dev.security.policy/...

There are some good arguments in both directions on that thread. I do happen to like the world in which, say, Let's Encrypt is capable of issuing for any TLD. If we don't think WoSign is good enough for signing .com, we probably shouldn't be exposing .cn users to those attacks, either.

I find the arguments from "it's wrong to recognize borders on the Internet" a bit naive, or impractical. The borders recognize us, and state-sponsored attacks will continue, and have very far-reaching implications. We need better tools to fight them, including this one.

I like the discussion in that thread about how to make the scheme reasonably flexible, though. Thanks for the link.

The certificate system - like the federal reserve - doesn't just fail us in these specifics, but is generally incompatible with the underlying philosophies of distributed technology.

I think we should just pull the rug out. Site operators need to be more aware of which CAs they're using, and need to feel some of the pain from the failure if they're going to shop for CAs based on anything except price.

The behavior of consumers in the certificate marketplace is part of the systemic problem: nobody cares very much about their CA, as long as it causes the little padlock to display correctly (and, more importantly, for warnings to not display) in users' browsers. That's it; beyond that it's a race to the bottom on price.

If a few CAs went down in flames and in doing so created a nasty firedrill for all their customers (which, on the scale of firedrills, getting new certs issued really shouldn't be that big of a deal, compared to a zero-day vulnerability in some piece of the server stack, which happens fairly regularly), suddenly there might be an interest in acquiring certificates from somewhere other than the cheapest-possible source.

Site operators need to be more aware of which CAs they're using

How? I've got no way to judge the security / responsibility of any CA. The amount of money that they charge may have no relation to their behaviour. I don't think anyone has claimed that the CAs who issued wrong certs were charging less than their competitors. You can't blame the CA customers for this.

Very often, marketing material and the language used for for that is a good indicator on whether a particular service knows their stuff, or if they appear to peddle fluff... :)

How CAs respond to issues on these mailing lists and in bugzilla is also pretty telling.

In this particular case, there's been concerns raised for several days. An extra vigilant web site operator who sees these discussions might perhaps come to the conclusion that it would be wise to evaluate switching CAs today, perhaps.

Very often, marketing material and the language used for for that is a good indicator on whether a particular service knows their stuff, or if they appear to peddle fluff... :)

We're all doomed then! Asides from LetsEncrypt, most CA web pages are full of marketing fluff :)

I've just been reading through the mailing list discussion, and from that I agree with you completely, WoSign looks pretty incompetent in their responses. If I had a WoSign certificate, I'd definitely be looking around to get an alternative now. However, that's a bit late for 'people power' to force CAs to act better.

One sign is that if Symantec has issued rogue certificates, and Google publishes the problem, then it's time to leave.

Of course, the CA would then be incentivized to publish their procedures and show how they intend to fix the breach, which shouldn't have happened in the first place, to regain customer's trust. Or go bankrupt. Let me tell you they wouldn't let their employees do mistakes.

Customers who stay, in the situation that it is discovered that Symantec issues another rogue cert, which causes Mozilla to completely revoke Symantec, can later sue Symantec for harming their business by not applying their public procedures. And here's how the economy gets rid of bad actors. The rule is much tougher for cloud machine providers (EC2 and DO): Should they lose one customers' data, everyone would leave them and go bankrupt, that's why they don't.

In parallel, losing a CA doesn't harm the customers very much, even for amazon.com. They can buy a new certificate in a matter of hours. bzbarsky even says it's commonplace to have the certificates signed by 2 CAs.

Absolutely agree. Pulling the rug is a powerful incentive both for the customers and the CAs. Customers will have to consider the likelihood the CA will be revoked due to poor security practices, CAs will have to show that they're not rubbish.

You could say the same about your bank - even though you don't really know how they invest your deposit money, you try and gain an intuition based on other factors (stock price, professionalism, reputation, prestige, etc). Not saying these factors -> a good analysis of the creditworthiness of a financial institution but you get the idea.

You could have backup certificates that you install when an untrustworthy CA is booted.

That's a chicken-and-egg problem, but it would be solved when there's suddenly a lot of site operators with a vested interest in knowing which CAs are likely to fail and which aren't. Right now there's not a ton of interest. CAs are basically selling a fungible, commodity product, so you just buy from the cheapest ones. Searching for "best SSL certificates" thus gets you a lot of articles reviewing CAs, but largely on the basis of stuff like price and ease of issuance. Because that's what people care about.

There would be a slew of reviews of CAs, judging their perceived odds of vanishing in a puff of paperwork, if there was an interest in issuer stability. CAs themselves could probably offer value-added features like guarantees backed by outside parties (i.e. an insurance product) that would pay costs associated with certificate reissuance in the event of malfeasance or incompetence on the part of the CA.

The market would provide, but there has to be demand. Right now there's no demand, because the consequences of getting a cert from a crap issuer has, historically, been approximately zero.

> Site operators need to be more aware of which CAs they're using, and need to feel some of the pain from the failure if they're going to shop for CAs based on anything except price.

Why? CAs just operate as a tax on sites. Having a certificate doesn't buy a site much; it's typically more for the _users'_ convenience, since they are the ones relying on the CA.

Really, users should pay CAs (probably indirectly), and CAs should issue certs for free. But I doubt that'll ever happen.

For commercial sites, the users are paying indirectly - since the cost of CA would be surely rolled into the maintenance costs, which are surely part of the price of whatever the site is selling.

I see two solutions to big CAs.

One is to reengineer the CA trust system to delegate limited authority to CAs (e.g. there is no good reason for a CA in the US to be able to sign certificates with Chinese TLD CNs, and vice versa).

The second is to build out the Certificate Transparency project to the point where the revocation of trust due to a mis-issued certificate is immediate, widespread, and automatic.

> The second is to build out the Certificate Transparency project to the point where the revocation of trust due to a mis-issued certificate is immediate, widespread, and automatic.

I'd love to hear more about this. How would it work?

Lets Encrypt gets a lot of flack because their certs need to renew every three months. I can't imagine why bad actors wouldn't immediately try to swamp any system of revocation with requests.

Imagine a new flag for certificates issued by CAs that support CT, indicating that the domain owner (e.g. github.com) intends to continue to only use CT-capable authorities.

Any certificate issued for e.g. github.com by any other CA immediately become evidence of malicious conduct attributable directly to that CA, and grounds for automated revocation of trust.

I'm not sure what the lifetime of Let's Encrypt certs has to do with this. I'm also not certain how the bad actors would exploit this process, can you explain?

Chrome is experimenting with something called Expect-CT[1], which allows site operators to indicate that only certificates with valid SCTs should be trusted. The implementation is quite similar to HSTS. It's report-only for now, but will probably evolve to something like Require-CT (i.e. CT enforcement) in the future.

[1]: https://docs.google.com/document/d/1VDtHiKa5c96ohP_p-V1k6u83...

Sounds like the right way to go. Thanks for the pointer.

As these things develop and become increasingly security-critical parts of the protocol, it would be nice if programs like libcurl and other HTTP client libraries gained support for them.

(e.g. there is no good reason for a CA in the US to be able to sign certificates with Chinese TLD CNs, and vice versa).

Sure there is. If you have a .cn domain but don't trust Chinese companies not to MITM your site, you can get a cert from a foreign CA and then pin it using the HSTS preload list.

That way, even if the registry hijacks the domain and sends people to a server with a valid (but fraudulent) cert, you're still protected by the pin.

If you don't "trust Chinese companies not to MITM your site" then why do you trust CNNIC enough that you decided to register your domain in .cn?

Because by pinning the cert, I can reasonably make sure they can't abuse that power.

But why not do both? It's unreasonable that there is a mountain of CAs out there that never sign certs for .us but they still have the capability to do so. The more CAs that can sign a cert for my domain, the more chances that someone screws up. At the very least I agree with limiting CAs to a subset of TLDs for protection from hypothetical things like the Turkish government demanding that a Turkish CA signs a cert for facebook.com. If it was a countrywide attack pinning the valid cert wouldn't be very effective if the browser has never been to your site before.

But why not do both?

Because then you're stuck using that country's CAs, and so you can't pin the root or intermediate certs without giving them the keys. You could pin your cert, but that has other disadvantages.

A viable solution to big CAs is to have all sites have their certs cross-signed by at least two different CAs. Then any single CA can be revoked without affecting any sites at all.

Note that this is also how a new CA is bootstrapped: initially the certs it issues are cross-signed by some existing CA so they work even in UAs that don't have the new CA in their trust root.

The obvious drawback is that now sites need certs signed by two CAs, and getting them to use even one CA is hard enough...

I was thinking the same thing. Assuming it's technically feasible I don't see this as being a huge problem. If your site is important enough that you can't have any downtime due to a CA revocation then you spend the extra time/money up front to get get cross-signed certs. If you don't care, then you just fix the problem when it comes up. If CAs start getting revoked more often then this will just become standard practice.

The solution for big banks is the orderly failure. Things like bail-in, where a bank is required to be able to go through a flash chapter-11 process over a week end. Or the "living will", basically the bank preparing a contingency plan in advance if it needs to wind down its activity.

In the CA world, I would assume this would require a standard protocol across CA to request, renew and transfer certificates from CA providers automatically. A sort of commercial version of the ACME protocol. But that requires lots of infrastructure change not just on the CA side, all servers would have to be updated.

This is long overdue anyway. It is currently way too complex to request, install and keep updated a certificate. And there isn't enough competition in this market.

Transferring certificates from an untrustworthy CA makes no sense. Burn the certificates. Websites should just obtain brand new ones from a different CA. If you are running a website that's important to you, this should be at most a one hour operation (or, if politics and procedures delay you in obtaining a new cert, maybe you should have planned ahead and kept two different certificates on hand already)

By transferring I meant re-issuing automatically a new certificate, not keeping the signature. Of course this requires to re-authenticate the request. And that process has to be automated. A bit like the renewal of the certificate should also be automated.

No, existing certificates should not be reissued. We can't trust that they were issued to the rightful owner originally by the failed CA. Clients should request new certificates and validate their ownership from scratch.

Getting a certificate is an hour's work tops if you are doing the csr dance manually. If you use ACME and letsencrypt it's done in seconds.

EV certificates are more work but if this matters to you then perhaps you should source multiple certificates to begin with, or act quickly when a CA is dropped (I'm sure browser vendors will give at least a few days notice)

Edit: Maybe I misunderstood you. If you just meant that software in general should offer requesting certificates from any provider, then sure, why not just use ACME. CertBot appears to be designed with multiple providers in mind. But this doesn't really have much to do specificially with the case of CAs being dropped from the root. (At first I got the impression you wanted other CAs to "bail out" certificates from the revoked CA)

The validation requirements for 'Extended Validation' certificates can be onerous - complete with in-person ID checks, and photocopies of passports and driving licenses signed by public notaries. I'm not sure automated transfers would be possible.

I suppose you could downgrade on automatic transfer. I've heard people question the value of EV certs, and certainly a working DV cert is better than a revoked EV cert. Or you could insist every EV cert applicant verify their identity with two different CAs.

> their only crime is using a crap CA

thing is the whole castle is built upon trust.

if you don't punish crap CA and ppl who don't do research first, things will deteriorate rapidly.

> if you don't punish ... ppl who don't do research first, things will deteriorate rapidly.

Is there a way customers would have or could have known beforehand that this CA was fishy? I agree that the CA should be punished/ostracized, but it isn't obvious to me that most of its customers would have known they were a fishy CA.

on first google search:

https://news.ycombinator.com/item?id=8982013

" It's 2015. They're using SHA-1 for everything (NOOOO!). They're based in China, which has just said it wants to ban encryption. It looks like they've messed up OSCP, so even their own cert doesn't pass. Oh, and RC4, TLS 1.0 "

Politicians in the US and UK have also claimed they want to ban encryption. Unless the CA is literally run by the government (which some Chinese CAs are, but not this one), it doesn't make sense to penalize them for dumb things their country's politicians say.

eh it's a longer quote taken out of context, not my words nor my opinion, but shows it's indeed not a respectable CA

I wouldn't say it's out of context, I was clearly stating my worries when this CA was discussed here a couple of years ago.

I think something should be done, but it's not my call of course. Given we do have Let's Encrypt now, I wouldn't feel in the least bit sad in revoking the WoSign certs. Anyone affected can, and should, change.

It's disappointing to see my concerns back then were well-placed, and there are clear indications StartCom's and WoSign's backends are connected together somehow?! I don't know what's going on there exactly, and someone should definitely find out.

+1 revoking a CA using the issuance date seems like the right solution

It would certainly be better than nothing, yes.

But prior to March 2015, CAs could issue certs valid for up to 5 years. So even if browsers stopped accepting WoSign certs with an issuance date after today, WoSign could still issue certs "issued March 2015 valid until to March 2020" and browsers would accept them.

If they start putting fake dates on the certs then the nuclear option is the only option. They are in the business of selling trust and they're outright lying on their only product? That's untenable.

See above, they've already been caught doing that.

Maybe the best would be to build a whitelist of WoSign certs then, scrap it from the publicly available websites and allow people to submit their own. Then some kind of bloom filter could do the trick I guess.

The same company also cheats on issuance date to support SHA1 signatures. I heard it on HN on another topic.

Edit: source https://groups.google.com/forum/#!topic/mozilla.dev.security...

Incident 2 ----------

In July 2016, it became clear that there was some problems with the StartEncrypt automatic issuance service recently deployed by the CA StartCom. As well as other problems it had, which are outside the scope of this discussion, changing a simple API parameter in the POST request on the submission page changed the root certificate to which the resulting certificate chained up. The value "2" made a certificate signed by "StartCom Class 1 DV Server CA", "1" selected "WoSign CA Free SSL Certificate G2" and "0" selected "CA 沃通根证书", another root certificate owned by WoSign and trusted by Firefox.

Using the value "1" led to a certificate which had a notBefore date (usage start date) of 20th December 2015, and which was signed using the SHA-1 checksum algorithm.

* The issuance of certificates using SHA-1 has been banned by the Baseline Requirements since January 1st, 2016. Browsers, including Firefox, planned to enforce this[2] by not trusting certs with a notBefore date after that date, but in the case of Firefox the fix had to be backed out due to web compatibility issues. However, we are considering how/when to reintroduce it, and CAs presumably know this.

* The issuance of backdated certificates is not forbidden, but is listed in Mozilla's list of Problematic Practices[3]. It says "Minor tweaking for technical compatibility reasons is accepted, but backdating certificates in order to avoid some deadline or code-enforced restriction is not."

* WoSign deny that their code backdated the certificates in order to avoid browser-based restrictions - they say "this date is the day we stop to use this code"[4]. If that is true, it is not clear to us how StartCom came to deploy WoSign code that WoSign itself had abandoned.

* It seems clear from publicly available information that StartCom's issuance systems are linked to WoSign's issuance systems in some way. Nevertheless, it should not have been possible for an application for a cert from StartCom to produce a cert signed by WoSign.

* This misissuance incident was not reported to Mozilla by WoSign as it should have been.

This is also not very encouraging:

> R: Sorry, I don't say it clear, please forgive my bad English since my native language is Chinese. As I said this is my fault that we don't understand the Mozilla policy clearly that we don't think we need to report. But now we are clear that all mis-issued certificate case and any reported bug related system change also need to report. I and every related employee all clear now, then we can guarantee we will do it well in the future. Why we log all SSL certificate from July 5th is for full transparency to let all related parties can report to us in the first time after the certificate is issued.

Maybe employ someone with enough English knowledge to read and understand https://www.mozilla.org/en-US/about/governance/policies/secu... ?

Amateur hour. You'd think understanding the rules of the game would be a prerequisite to be a warden of the entire TLS ecosystem, as any unrestricted CA is.

It's not comforting that the entire security of https globally is now in the hands of someone unable to read the CA requirements, and doesn't even seem to worry about that fact.

Trust of a CA doesn't have to be binary - it could be stochastic.

Let's say that when a browser wishes to revoke a CA certificate, it chooses a timeframe for a "deprecation period". Before the deprecation period, the CA is fully trusted. After the deprecation period, it has been completely eliminated.

During the deprecation period, a browser will possibly pop up an error page rather than accepting the certificate. The probability of this happening increases as the deprecation period advances, slowly "turning up the pain" (likely exponentially or quadratically, for slow initial growth).

A reload will clear the error and load the page (or perhaps go through the probability again). Obviously it would be good if the site were notified through a header that this was happening. And user feedback will accomplish the same thing in a cruder manner.

Proactive sites would move off before the deprecation period even began. Less connected sites would get user reports and move off early in the period. Negligent sites would see their users migrate to different sites as their functionality got ever worse.

Would that not just encourage your average user to click through the security warnings and ignore them, potentially numbing them to other more important warnings?

Users aren't the ones who should feel the pain of a rogue CA, we need the CA to feel the pain somehow.

Perhaps it'd be better as a series of progressive informational pages than described in terms of certificate rejection. The central idea is that the site will become gradually less usable.

If a CA is being revoked, that's pretty close to the maximum pain they'll feel. But suddenly revoking a CA will cause users the most pain - I'm trying to make that gradual. A competent site would react in the first week when a few users got a mild warning, get a certificate from a new CA, then hopefully complain to the original CA demanding a refund and whatnot.

> when their only crime is using a crap CA.

Their punishment would be having to pay for another certificate and install it and using a more reputable one and not necessarily the cheapest one around next time. I think in this case the punishment fits the crime perfectly.

more reputable one and not necessarily the cheapest one

The two don't seem to be linked at all; Symantec's cheapest cert (non-EV, no-wildcard) costs $399 - almost 80x the cheapest ones - and yet they were caught issuing unauthorized certs.

Oh, of course, expensive doesn't mean good. Maybe the cheapest one is the best one. I'm just saying the reputation should be a primary consideration, or at least one of. There should be consequences for bad behavior for CA vendors.

Give them a month, publicize in the relevant places, and that's that. If a site gets caught out it's not the end of the world. If they have big money on the line then they can afford to have someone on staff that keeps up with this stuff.

Reply on news.ycombinator.com