Hacker News Reader

Comment by randomf1fan | Hacker News Reader

This is great news for organizations. I work at a large Fortune 150, and there are lots of services that require wildcard certs. We have a process to get these from our internal CA as well as a third-party - the internal CA is automated, but the third-party (for external services) can be slow and cumbersome, to the point where many departments just buy their own cert. And then a year later, they move on, forget, etc, and suddenly we have services that have expired certs and there's a scramble to fix them.

This move by Letsencrypt should hopefully make them the standard for any external service that doesn't require an EV cert.

View on HN

>This move by Letsencrypt should hopefully make them the standard for any external service that doesn't require an EV cert.

I'm kind of worried about this myself.

No matter how well intentioned, secure, or "good" lets encrypt is, having a significant portion of the world's TLS be under one umbrella isn't a good thing.

I'm hoping that we will begin to see other services pop up that are similar to lets encrypt (free, even using the ACME protocol) so that we don't have too many of our eggs in one basket here.

Yes, that's a fair point.

However, it's still better than having expired certs and a team trying to figure out who owns the app, trying to get in touch with them, asking them to update the cert, finding out that they are no longer with the company...

It's even worse when the service is something that a small team created as a POC - which then became customer facing and mission critical, with the team having moved on to something else.

And it's funny how often this happens over a holiday weekend.

Yes, I know that the issues are deeper and more to do with large company process and bureaucracy than anything technical. But at least you can have secure services that don't fall over.

To use a classic example, who's going to go to bat for you when they get a demand from a government agency - the EFF or the Hong Kong Post Office? I know where I'll place my bets.

Go pull up your certificate authority list and ask yourself for each one of them if you trust that company more or less than let's encrypt.

Let's encrypt publishes auditable logs of all issued certificates, they're backed by some of the biggest names in online privacy and I trust them much more than other CAs.

I for one would be happy if I could delete all other providers from my browser.

CAs ultimately are centralized and too trusting, giving that same level of trust to less trustworthy companies just damages the overall security of TLS. There's no distributed trust model for CAs, it's pretty much all or nothing, so in the case of CAs, distribution is not a security benefit like it is in say Tor or Bitcoin, but a problem as it means the attack surface has widened.

Short of going to a new, completely decentralized solution like the proposed DNSSEC extensions or Namecoin, a single, very secure CA is probably better than a lot of not secure, often government influenced CAs.

fortunately, this change should help diversify the number of certificate authorities for people to use. In the explanation of the wilcard support they link to another post that explains it is enabled by their rollout of ACME v2.

Wildcard support is one advantage of ACME v2, but another advantage they list is "ACME v2 was designed with additional input from other CAs besides Let’s Encrypt, so it should be easier for other CAs to use" - https://letsencrypt.org/2017/06/14/acme-v2-api.html

So, in addition to this functional improvement to Lets Encrypt, the change should enable more automated CA options in the future.

> having a significant portion of the world's TLS be under one umbrella isn't a good thing.

Why is that? The damage from a CA being hacked is not proportional to the size of the CA - they are all equally (small number of exceptions notwithstanding) capable of issuing certificates for any domain which will be trusted by all major browsers.

Is there another aspect I'm not considering? While I see how it feels like a troubling thing, I'm struggling to actually come up with any real consequences of it.

A larger CA is a more valuable hacking target.

The parent makes the point that it's not necessarily the case since hacking any trusted CA (no matter the size) lets you generate certificates for anything. If letsencrypt was hacked today it could be used to generate a valid google.com certificate for instance, even though Google's certificate is normally issued by their own authority.

It's a weakness of the current authority architecture really, trusting a CA is an all or nothing decision. If any of the authorities is compromised you're vulnerable until you remove the CA from your browser, regardless of the number of legit certificates it issued.

It's not just about access to their private key, but also downtime (expected or otherwise), and bugs in the cert verification process.

I don't know of anything concrete, but I can imagine an attack that can exploit the process of verification on their servers to have them sign domains they shouldn't, or DDoS attacks on them to prevent people from renewing their certificates. The bigger they are, the juicier of a target they are for these kinds of things. if they were a provider of 50% of the internet's TLS certificates, you could take down half the internet by continually DDoSing a single company!

Hell I can already imagine someone sending a bunch of signing requests spoofed as someone else, locking that person out of renewing due to rate limiting.

Not to mention that even the country they operate in can be a big deal.

Let's Encrypt strongly encourages you to use a tool that does automatic renewal a month before the cert expires. If someone manages to DDoS Let's Encrypt for an entire month, I think we're firmly into "you have bigger problems" territory. (Among other things, if 50% of the internet were in fact on LE, major internet providers like CloudFlare and Akamai and Google would start offering to run LE directly on their own infrastructure after a week or so of this.)

Bugs in the cert verification process are the same amount of risk regardless of whether everyone is using the CA or nobody is, as long as the CA is trusted. There's nothing gained by putting your eggs in multiple baskets.

Also, these all seem like hypotheticals when the old-school CAs have had OCSP downtime, bugs in the cert verification process, incompetent staff signing and publicly logging google.com certs to test their infrastructure, governments asking and receiving unconstrained intermediates, unconstrained intermediates as a publicly advertised product, etc.

You're right but size doesn't really factor in any of your points.

Assume for instance that the country of Hackeristan manages to have one of its authorities accepted in major web browsers. This authority is only meant to sign Hackeristan domains and only signs a tiny amount of certificates.

Now let's imagine that this authority is compromised, maybe the Hackeristan government wants to intercept connections to gmail, maybe the authority is vulnerable to hackers. One way or an other, it signs a bogus *.google.com certificate. Well it's game over, since the authority is trusted by all major browsers everybody's vulnerable, even though it was a tiny CA. Only certificate pinning can save you now.

Yes, but if LE was the only major CA, then if you could attack "Company A" by impersonating them and making lots of signing requests causing them to hit rate limits you could take "Company A" offline.

If LE was found to be incompetent and lost control of their private key, browsers would be much less willing to remove them as trusted if they were a significant portion of the web.

And things like the impact of DDoSing LE to take their OCSP servers down and things like that still grow with their size.

To clarify, I love LE and I use them almost exclusively. But I'd feel better if there were others trying to follow in their footsteps.

Punishing CAs for bad behavior (ie Security Problems) has more collateral damage the bigger a CA is. Right now, if a CA is bad enough browsers just stop accepting their certificates. After a certain size that becomes unfeasible, removing a lot of pressure from that CA

The pressure will come from the public. If they damage their reputation, people will be less willing to donate, which will pretty directly influence their income stream.

99% of the public doesn't know what is a CA

99% of the public doesn't donate either.

I'd be amazed of it wasn't at least five-9s, Facebook has 2B users.

You're underestimating how many web designers and devs are out there. These easily number in the millions.

Also, what grandparent said: "The public", in this case, is people who would donate, of which 100% know what a CA is.

Assuming Facebook's numbers represent two-thirds of all web users then I'm saying I'd be surprised if LetsEncrypt have more than 30,000 donors.

If we're quibbling about "the public" then the GP comments only make sense if "the public" means "people who aren't IT professionals", in which case I'd warrant that there are far fewer donors than 30k who aren't IT professionals, indeed it's got to be ~0.

Can't see donor details on the LE pages though? Mind you at approx.av.300k certs issued daily (https://letsencrypt.org/stats/) I concede I could easily be orders of magnitude out in my guesswork.

No, browsers don't do that. See how WoSign was distrusted[0]. Basically, they still trusted existing certificates, but stopped trusting new certs (both renewed or brand new). Through this, they kept collateral damage to a minimum, while carrying the CA death sentence.

[0] https://blog.mozilla.org/security/2016/10/24/distrusting-new...

The trouble is that's only possible with the CA's cooperation, because they have the ability to backdate the certificates by falsifying the date. In the case of WoSign Mozilla threatened to distrust them completely if they did that, but if it's unfeasible to remove a CA that threat may be ineffectual.

This kind of forgery can be mitigated by requiring all certificates to be published to a Certificate Transparency server upon issuance. You can't backdate a public ledger that is being watched by third parties.

It can be a headache when their OCSP server and issuance process run into problems that break the Internet for a large number of people. https://community.letsencrypt.org/t/may-19-2017-ocsp-and-iss...

It's been awhile since I had to deal with ocsp breakage, but if it breaks due to an ocsp server down, doesn't that mean the browser or web server is misconfigured? Of course, if browsers are misconfigured out of the box, that doesn't help at all...

It wasn't as simple as the ocsp server being down. It was returning bad request (http 400) responses. When the good responses expired from caches, the bad responses started going out and breakage started spreading. LE detailed this in their postmortem which I linked.

Amazon has the capability to do this with their ACM tool, but artificially hold it back to AWS hosted only.

I would use them if they turned this on and offered the service at a low cost.

It's not 'artificial', for multiple reasons. You correctly address one by listing the cost, but the other is that they limit you to only being able to configure a certificate to a resource they manage, so that they can rotate that certificate transparently for you.

Yes, they could branch beyond that, and create a special category of certificates to issue, that has a cost, and that gives you access to the private key, but that isn't really distinguishable from any other certificate provider out there. In fact, Let's Encrypt offers that for free? Why would Amazon decide to compete with a paid product against a free one, when there's no benefit to the consumer to warrant paying?

>Why would Amazon decide to compete with a paid product against a free one, when there's no benefit to the consumer to warrant paying?

They compete with free Cloudflare caching today. And free DNS services. And much cheaper VPS services. There are various reasons customers might choose low cost over free.

For me, since I use ACM already, for AWS hosted resources, I would appreciate the advantage of using it for other resources. Even ones on AWS, like a cert on a Lightsail instance, for example.

I say artificial because all that's missing really is a link to download the cert.

I work on the Let's Encrypt project and this is my own personal opinion, not necessarily the opinion of EFF or ISRG.

There are several reasons that it could be good to have multiple publicly-trusted CAs out there, though most of them don't depend at all on the relative market share of the CAs at a given point in time.

Security risk If a particular CA is compromised through an attack and its intermediate certificates need to be revoked, it would be good to have other CAs already ready to continue issuance to the public. Even if the CA intends to resume operations after a compromise and user-agents are OK with that, it probably wouldn't be prepared to resume operations immediately.

Geographic risk It would be good for availability to have CA datacenters on multiple continents, not just one.

Jurisdictional risk The government of the country where a CA operates could try to compel the CA to stop issuing particular certificates, for example in order to enforce international sanctions, or to facilitate espionage by trying to make it harder for people to get authenticated encrypted connections to certain services. A government could even force a CA in its territory to cease operating entirely. (There's also jurisdictional risk in the other direction, of governments trying to compel misissuance, but this risk is strictly increased by having trusted CAs in more jurisdictions. In general, having more CAs increases the risk of misissuance, while decreasing the risk of certificates being unavailable to a particular site because people don't like that site or its operators for some reason.)

Continuity risk It would be safer to have CAs with more different kinds of funding for their operating expenses.

Institutional/governance risk A particular CA might some day decide to do things that relying parties find improper. Having more CA alternatives can give the relying parties more plausible leverage to get the CA to align its practices with their preferences. (As with the jurisdictional risk point, only a decision not to issue certain certificates can be directly addressed in the short term by having other alternatives. A decision to issue certificates that other people think shouldn't have been issued can probably only be addressed this way by removing trust from a particular issuer.)

Looking over this thread, I do want to emphasize again that misissuance risk gets worse, not better, when there are more CAs. If you're particularly afraid that CAs will be issuing certs improperly because they get attacked or coerced or do a bad job of validation or internal controls, you should probably want fewer CAs rather than more, at least as a response to that particular concern. This is because CAs in the X.509 PKI can't "contradict" another CA's issuance; every assertion about a binding between an identifier and a public key is cumulative and operates in parallel and in addition to every other assertion.

Has a large CA ever had its intermediate certificates compromised? Just curious if that process has ever happened.

I don't recall anything like that having happened before, even in the DigiNotar case, where the CA was thoroughly compromised. The keys must be kept in HSMs, so even with a fully compromised issuance system, the keys themselves are typically safe - which isn't much of a relief at that point.

There were a couple of cases where CAs like Trustwave or CNNIC signed intermediate certificates that were capable of issuing publicly-trusted certificates for organizations who lacked the required audits. They were typically intended for corporate/internal MitM proxies, though there was no technical enforcement in place for this, and they could've been used for any MitM attack. The recent investigations into Symantec's CA showed similar, but slightly more complex cases.

I just reviewed Chapter 4 of Ivan Ristić's book and the only incidents that might be considered compromises on this level were DigiNotar and NICCA, which both led to revocation of intermediates. However, the book doesn't explain technically what the exact nature of the compromises was, so I'm not sure either of them involved an actual compromise of the private key material itself.

There were many other incidents involving problems with behavior of PKI participants, and I'm sure reading this chapter will give people a sense that the ability to remove trust from intermediate CAs is an important ability.

Why are we relying on single providers, shouldn't it be a consensus system of some form, like month old certs get passed on in bulk outside of the main network, then if the original provider is compromised their cert info disagrees with the "consensus" providers indicating a compromise at some point.

High profile sites can buy multiple top level certificates (with mutual signing, say); sites needing less security can fallback on a simplified consensus system (maybe like above).

Workable?

And a quick follow up - our security team does not allow rogue * .company.com wildcards - they hunt down those sites and boot them off the network.

They do, however, allow *.<service>.company.com.

Can you clarify who "our security team" is?

From the parent post:

> I work at a large Fortune 150 [...] we have [an] internal CA

(This was a lot clearer when there weren't so many other comments in between.)

Sure - this was in reference to my top level comment, but I see that this dropped lower down the page.

I work for a large Fortune 150, one that you've heard of, and we have a security team that is constantly scanning our network for weaknesses and potential exploit vectors. They will kill (firewall off) any sites that might compromise the network and tell the application owner to fix the issue before they allow it back on the public net.

That is excellent advice from your security team.

https://certificatemonitor.org/ works quite well

As does any monitoring system (nagios, zabbix, etc) that any organisation should already have running.

This is a helpful use case. Could you message me so I can learn more? press@letsencrypt.org

Unfortunately I'm not authorized to publicly speak for the corporation, so I can't really give you any more details. Sorry about that.

And let's be perfectly clear, EV certs are basically a moneygrab by CAs trying to provide some kind of value-add. It used to be that ecommerce sites would have to get an EV cert, now with Chrome desktop showing "Secure" in the address bar, the visual representation of an EV cert isn't nearly as important.

Reply on news.ycombinator.com