Entrust Certificate Distrust
security.googleblog.com
security.googleblog.com
https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
How dysfunctional does a company have to be to let this happen?
Stuff like this happens when upper management has zero clue about the business they are in. They believe they are in the business of selling certificates, while in reality they are in the business of selling trust. They treat things like the CA/B Forum and the various Root Programs as more like an optional networking event than the combination of judge, jury, and executioner that it actually is - with a completely predictable outcome.
Private certs are not that big a business.
If you asked people from the for-profit CAs about Let's Encrypt before it launched, the impression you'd get was that they're issuing a lot more certificates and this doesn't matter. Millions per day? Ha, we'd barely notice. That was all bluster, they were never doing that.
I think Apple probably had the best shot to turn this into free money. Apple's customers are very willing to pay more than something appears to cost on the basis that it's Apple so it's worth it. I think you'd struggle a lot more to undercut a $10 Apple PKI product with a free offering that's identical because Apple's customers are used to justifying why they spent more money on the same thing with the logo on it, and they are able to be completely irrational about it and it's OK - a brand rep would look unhinged if they violently attack people who point out that it's bullshit, loyal fans will get understanding or even praise.
I actually thought about 10-15 years ago that Apple was about to do this, but they didn't and once Let's Encrypt happens there's no room really. Apple does still make money off some places where they're sole issuer and get to charge arbitrarily for doing nothing, but not like they would if they'd seized the entire Web PKI.
Also roots can be TLD constrained, typically to ccTLD(s).
Which means their business continuity planning was bullshit.
The good news is this caused companies in the healthcare space (at least provider, facility, and insurer sides) to start asking more pointed BCP questions to their SaaS vendors.
Surprised no one pointed to the nature of the business as a source of this behavior.
In a non-innovative, compliance-based industry, you make money by cutting costs.
This affects the entire business, as you find managers who are effective at cutting costs and architects/engineers who will work for lower salary.
Multiply that over enough years, and we know where it leads...
>> I see three possible outcomes:
>> 1. The root programs continue to be lenient with Entrust indefinitely. Nothing changes.
>> 2. The root programs continue to be lenient with Entrust for a while, but eventually the mistakes pile up enough that one of the root programs pushes for distrust.
>> 3. The root programs immediately stop being lenient with Entrust. Entrust is forced to make internal changes to remain a CA.
It raises an interesting point about what constitutes a historical pattern of behavior, sufficient for infering future deficiencies reliably enough to take present action.
Here, the motivation seemed to be that (a) enough history had accumulated to estimate Entrust's rate of process improvement & (b) that rate was deemed insufficient. Which seems a decent metric: if perfection is not presently achieved, then remediation progress needs to be seen.
Here's the email their CEO/President sent to everyone that uses them:
Google Chrome announced yesterday that specific public roots used to issue public certificates by Entrust will no longer be trusted by default after October 31, 2024. This decision comes as a disappointment to us as a long-term member of the CA/Browser Forum community.
To address your concerns, there have been no security implications to the events that led to this distrust event, and you can be assured that your certificates are secure. I also want to assure you that Entrust can and will be able to serve your digital certificate needs now and in the future. And, our ability to do this extends beyond the public roots covered in Google’s decision.
Additionally, there is no impact on our private certificate offerings – including our PKI, PKI as a Service, and managed PKI – nor our code signing, digital signing, and email (S/MIME and VMC) offerings.
While the announcement is disappointing, Entrust has been in the public and private digital certificate business for over 25 years and we continue to bring that expertise and capability to your use cases every day. It is our hope that you will allow us to continue to serve your needs and we stand ready to answer any questions you have regarding your ongoing needs.
Sincerely,
Todd Wilkinson President & CEO
---
My personal take: I don't see why any of their customers (such as ey.com) would want to split their CA needs across multiple suppliers.
I was wondering how Chrome was able to revoke a certificate based on time without trusting the CA to not back date certificates and it looks like this is due to being able to trust certificate transparency logs instead. This is where they get the signed certificate timestamps (SCT) from.
[1] https://groups.google.com/a/ccadb.org/g/public/c/wRs-zec8w7k...
The goal of CT was not to make staged distrust of a CA possible (that's just a happy coincidence). It was to make it possible to detect CAs misissuing certificates, which includes CAs back dating certificates.
I’m also an author on https://webpki.substack.com. I will be writing my thoughts on the distrust soon.
I can try to answer any questions folks may have. I can also help folks find ways they can also be involved!
Root programs can only do so much and need surveillance of the CAs from the community.
Regardless of how Entrust is operated, there appears to be significant complexity in CA program that the browsers operate. On the flip side, Let’s Encrypt is basically effortless for me to use, as an end user of an LE secured site and as a developer. Why misallocate all this toil on root CA compliance on the one hand, when LE could redirect that labor towards something valuable instead? What is so challenging about giving LE full leadership on this issue? Where does the proverbial political strength of Entrust and similar entities come from, in an ecosystem where there are functionally 5 cooperating, more or less transparent entities that decide the trust of certificates for 99% of end users? Why does anyone care about any of the CAs?
I don't get entrust here. It's not like they weren't told what to do.
As an example of how wedded Let's Encrypt is to simplicity, part of routine application process is to show your certificates expire properly, often a CA would manually create a certificate which either was already expired (back dating it but arguably this is allowed for technical procedures) or had a very short lifetime and so would expire by the time the trust programme examines the demo cert. Let's Encrypt instead issued an ordinary 90 day certificate, using the same automatic process as their subscribers - and then just waited months for it to expire. Like a boss.
Let's Encrypt started their operations with _automated_ certificate issuance only. They also do not do OV/EV certificates that are much, much harder to automate without providing any real benefits.
So, LE's mission is to issue certificates under the rules set by CAs and Browsers. (Yes, CAs do participate in setting up rules for CAs.
> Where does the proverbial political strength of Entrust and similar entities come from
Generally supporting non-automated certificate issuance. Effectively, technical debt. A lot of older enterprises have done manual certificate issuance, and they don't feel the pressure/reason to switch.
"treasury.gov" and "uspto.gov" stand out (to me), let alone the lengthy list of banks and other gov places around the world.
https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
The older vendors are even less able to be properly open with their customers (let alone the general public) than Google. At Apple it's probably a firing offence to even confirm obvious decisions - it seemingly took months to get Apple's chosen representative to confirm that Apple's new 398 day rule was an issuance requirement, rather than just something where Apple wouldn't trust longer lived certs in Safari.
Any root program will refer to it for context on issues.
And for a CA, credibility is everything
If you believe https://bimiradar.com/glob
it looks like Entrust is selling on the order of a few dozen certs a week to maybe upwards of 100-200.
EDIT: I've asked Google if Gmail will be discontinuing support for Entrusts VMC certificate (and thus BIMI logos), I would guess not since BIMI has some actual requirements, but assumptions are not the best way to make decisions about risk (like our BIMI logo not working later this fall).
See arguments about red-warning unencrypted HTTP and how that pushed the web to update.
Add in that genAI is going to make plausible-looking phishing emails a lot easier for the world to generate en mass, and giving the everyperson something better than "decide if it looks suspicious" is important.
A dead giveaway would otherwise have been that the BIMI issuers are all the now-panicking EV certificate issuers, which nobody will now buy.
They've pivoted to payment cards.
This continues to annoy me. Chrome (and other browsers) have detailed trust constraints, e.g. SCTNotAfter, in their own root stores. Why can’t administrators do the same thing?
It should be perfectly reasonable and probably required for an employee to be able to order reimbursed things like travel arraingements with a credit card on their org provided device, but that org may MITM any trust chain for some administrative convenience.
The org itself could cross sign with name constraints if they opt to be good, but would probably end up filing a lot of bugs in various software that can't handle it and their being good is the kind of selfless act that rarely happens without a regulatory requirement to pay for consequences of doing a MITM of your employees on the Internet.
Whether CA/B is good or bad at what it does, it puts about a thousand times more effort into the question of whether to install a CA certificate in the browser than a company that just bought the cheapest solution to one of its problems and wants to install the corresponding vendor certs.
For example: https://docs.umbrella.com/deployment-umbrella/docs/install-c...
How many things could be wrong with that system and cause user's traffic to be compromised web wide? What community is checking transparency logs and threatening Cisco to revoke their authority to sell that product? What would that even mean?
In practice, complexity and customizability breeds ossification, because "safe" becomes the tiny sunset of common configuration.
I could definitely see network appliance vendors, IT network security admins, endpoint security vendors, etc. rapidly fucking up everything.
At least with delegation to browser vendors + certificate transparency logs, we have a semi standard path for a detrust like this to be forced without exploding the ecosystem.
Additionally, if there were more wiggle room, you'd alter the balance of power between browsers and CAs, which seems decently calibrated now.
The problem as I see it is that whatever method used is optional and insufficient to protect users until the browser highlights the source is not real public trust. Google knows this and started with the claim they prioritize user security while ending with the work around to prioritizing user security. (And without the slightest warning that sending your users to a bunch of financial institutions using improper trust chains is ethically dubious and requires more consideration than the time it takes to click the settings.)
Directly from Entrust: "Yes, there has been ongoing internal discussion and reflection on the issues found in this and other incidents, which has led to the action items described previously and ongoing changes, including the decision to revoke the certificates affected by this bug. Exceptional circumstances would need to be provided and justified by the Subscribers. Given the nature of the feedback we have received to date, we doubt that the community has any real interest in anything that Entrust could suggest, except to use against Entrust in a destructive, not constructive, way. We therefore would like more explicit and clear guidelines or a definition of “exceptional circumstances” to be adopted and applied equally to all CAs, perhaps through updates in the CA/B Forum requirements."
We’ve been endlessly talking about our repeated screw-ups, which led us to revoke the affected certificates. If subscribers want an exception, they need to come up with an extraordinary excuse. We don't care, so we demand clear and strict rules about what counts as “exceptional circumstances” that apply to all CAs, and these should be updated in the CA/B Forum requirements. We are big, who are you?
... it's not promising
it looks like Entrust is selling on the order of a few dozen certs a week to maybe upwards of 100-200.
EDIT: I've asked Google if Gmail will be discontinuing support for Entrusts VMC certificate (and thus BIMI logos), I would guess not since BIMI has some actual requirements, but assumptions are not the best way to make decisions about risk (like our BIMI logo not working later this fall).
There's now also the problem of competing with a free alternative that increasingly almost everyone knows about.
If you read through some of the incidents in bugzilla, you get the strong impression that Entrust’s market is specifically the people for which something like LetsEncrypt isn’t currently a viable alternative (or at least a difficult one).
In trying to justify not revoking misused certificates, one example they gave for a customer they were granting extended deadlines to was some organization that was contractually obligated to their customers to provide at least 90 days notice of any certificate updates.
While the deliverable is essentially the same, I don’t think Entrust and LetsEncrypt have really been in competition.
Entrust missed/ignored the updates to how certificates were supposed to be formed and when caught declined to revoke the incorrectly issued ones (because it's probably a more or less manual process for many admins working in a pre-Let's Encrypt style of fashion) and they didn't want to inconvenience their customers and assumed that they themselves were the important party in the equation (CA's was that historically compared to site-admins).
The certificate industry has always been quite ad-hoc with CA's being entitled middlemen, we have Let's Encrypt and almost ubiquitous encryption now because browser makers and other internet actors saw security as more important than protecting the CA's business and now that LE is established Google,etc aren't the slightest interested in pampering CAs if they aren't interested in cleaning up the system.
The problem in this case is that Entrust displayed a complete disinterest into actually solving the underlying issues. Doing an oopsie is one thing. Doing an oopsie, lying about it, refusing to take precautions, and failing to take measures to prevent a repeat despite promising to do so? Completely different story.
If they can't be trusted to respond properly to minor administrative issues, why should they be trusted to respond adequately during a real security incident?
The problem is that to be a CA in the root stores you agree to (and entrust voted in support of) a pile of rules, and entrust demonstrated a complete disinterest in complying with those rules.
The reason for removing trust is not the severity of the original errors, it's the the severity of errors in the response.
1. The BRs requires revocation of invalid certificates with 5 days unless there is an exceptional reason not to. Entrust did not.
2. In the event a CA discovers that they are mis-issuing certificates they are required to stop issuing until they have resolved the error. In this case not only did entrust not do this, but they explicitly stated - after the issues were raised, and they were already told they were failing to revoke certs in the required time frame - that they were intentionally continuing to mis-issue
3. They repeatedly made errors in the past, promised to correct them, and then kept making the same errors
4. They made claims they were trying to get customers to prepare for revocation in the require 5 day window, but then it turned out they were telling customers that they had 30+ days (which was only discovered when one of the relevant customers forwarded info to someone else)
5. When a CA discovers miss issuance they are required to file an incident report, and provide detailed information about how it occurred, why it was not caught, what remediating steps are being taken, and what mechanisms are being introduced to ensure a similar failure cannot occur in future. None of Entrust's responses came close to this, until the chrome root store rep came in to say "this is unacceptable", and even then their "improved" reports were incomplete and lacked sufficient detail.
6. Once they were finally doing the basic steps they were meant to have done the moment they learned of the miss-issuance they repeatedly failed to produce an accurate set of the impacted certificates (as in the provided a list and people outside of Entrust were able to immediately turn around and say "but these certs are also broken, why aren't you listing those details")
and so on and so forth.
Google's post to CCADB provides more details than the blog post: https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
This reminds me about discussion about Russian Goverment's NUC Root CA (not trusted by default in Chrome/Firefox, Trusted by Yandex Browser only with some additional verifications to prevent abuse by goverment). Discussion was not about why this cert was necessary in first place, it was about it's creation violating Russian laws and procedures AND violate a lot of technical rules. A lot of people just said - this cert is necessary and it's clear who made it so why we should look to "minor details"? (Links - in Russian https://habr.com/ru/articles/666520/ / https://habr.com/ru/articles/708970/ )
Wonder how secure that is? That has real potential for extracting value.
nCipher make HSMs - a lot are used in banks to encrypt transactions on behalf of devices through APIs like PKCS11.
To answer your question "how secure that is?"; The answer is yes, secure.
chase.com aa.com
washingtonpost.com, cdc.gov, dell.com, jpl.nasa.gov, mastercard.com
This is gonna cause me some headaches, along with everyone else who processes payments through Cybersource, and possibly others :(
We're already working on it. Keep an eye for merchant notifications if you use certificate pinning.
Now, back to rotating certificates....
It'll be interesting to see what, if any, organizations affected by this switch to: Stick with 1 YR certs or go to the future with free 90 days?
I'm sure leaf certificate pinning is very common among your customers. Assuming that pinning is a manual process where customers decide to implicitly trust a specific cert, what's the point of using a third party CA for those customers all?
Does anybody self-sign or use a private CA on specific endpoints with longer certificate validity, and let the pinning customers use those?
We have one customer who has demonstrated enough competence with certificates that we create a private ca endpoint and let them use that. The private root lasts around 5 years, and they pin to that.
It doesn't say much.
[0] https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...
With Let's Encrypt we made a lot of people's certificate management a "fire and forget" thing, which is exactly what we hoped to do, but if they completely forget about it, it may be that there will be lots of targets against whom nobody would notice certificate misissuance.
(I wasn't aware of your credentials when I made my previous comment so I assumed you didn't know about mandatory certificate transparency which is a mistake on my part, sorry! I'll make sure to check profile about sections before I assume again.)
Also, we made Certbot randomize the subject key by default every time it renews, so you have a huge amount of churn in subject keys, so you can't just say "oh, well, this public key has been used for a long time, so it's probably correct!". Every subject key is typically new and is unrelated to every previous subject key.
I hope that won't turn out to have been a poor trade-off. (We thought it was good to have more turnover of keys in order to reduce the impact of successfully stealing or cryptographically attacking one.)
Indeed, the fact that Cloudflare emails out CT warnings due to their own backup certs is rather embarrassing.
It's also why I'm personally against SMIME and think it's a bad idea.
Another example: Yandex Browser ONLY trust Russian NUC certs if they are in public CT logs,not otherwise (https://habr.com/ru/companies/yandex/articles/667300/ - text is in Russian) (as far as I understood, NOT trusting this CA al all is not option for them or their users, and if user is using chrome/firefox and needs access to sites which use this CA - CA will be just be installed manually so Yandex's solution is more secure, thanks to CTs).
Intel people really don't want to get caught (and whatever CA they use really does not want to get caught), CT turns the attack into a gamble. Even if nobody is checking most sites, CT still creates a deterence factor. Not perfect, but a lot better than the previous status quo.