DigiCert: Threat of legal action to stifle Bugzilla discourse
bugzilla.mozilla.org
bugzilla.mozilla.org
DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation.
Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner.
Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.
It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients.
Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-inci...) says clearly:
"Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business.
Make that series of consequences clear, and companies will stop trying that trick.
The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations).
If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
You say that and yet Teams has 320 million people using it, and I bet almost none of them enjoy the experience. Sometimes you just have to work with what you're given. Given "incompetence", we might as well throw in all of Azure.
> Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
That might be true for DV, but there's classes of certificates that take at least weeks to obtain, think BIMI VMC or codesigning.
Yeah, there are tons of companies who chose Teams as a vendor (because it's bundled with other stuff), and inflict it on their employees. It was still absolutely their choice.
Use of a TRO protects you short term, but results in having to migrate to a new CA medium term. You can't stop them from using TRO's but you can make it not worth it.
For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days.
I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43
I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong:
> You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs.
> You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
I'm not sure DigiCert could have done anything about the TRO or the impacted certs, but it should have been able to move forward with the revocation of all other certificates. That IMO is the real issue/failure, alongside the concern/impact of TRO's on security processes in the future.
Yes, I think this would have been appropriate action. If the contractual language is extremely clear between the CA and the subscriber, there is no legal basis on which the customer can prevent revocation. The fact they found a court that doesn't understand technology is frankly irrelevant. This detail is exactly why Tim and other parties are requesting the exact language of the agreement between Digicert and the subscriber that filed the TRO. A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR, or to take actions which are detrimental to the entire Internet. This is exactly the type of circumstance where you do what you are required to do, and then sort it out afterwards. Any appeals court would have easily overturned the TRO as it has no legal basis.
Yes, it absolutely does. "I think the court will agree with my view of what the contract says once the case is heard in full" is not a valid reason to disregard a TRO.
> or to take actions which are detrimental to the entire Internet
That would be harder. But a delayed revocation stemming from a flawed validation process, when the CA is responsible for the flaw and knows that the result of the validation was in fact correct, simply does not cause any detrimental effects to the entire Internet.
Of course, if that system had been in place, DigiCert would probably be facing hundreds of lawsuits from businesses disrupted through no fault of their own rather than inside baseball PKI drama.
Make it clear that if it works, it will only work once.
The CABF should adopt policies that any such legal action or any request for extension will be considered a public declaration that the customer's application is incompatible with the requirements of the Web PKI and that not only will the current CA refuse to renew the certificate but it will be publicly documented in the Bugzilla so that no other CA will issue certificates covering any of those names nor any new names for the same company without an affidavit explicitly stating that the issues preventing compliance have been resolved and the company acknowledges this and commits to never doing so again.
Existing names that were successfully revoked in time can be renewed but neither the problematic one(s) nor any new ones will be allowed.
If they then file for another TRO in the future they may still get a short-lived order but the existence of such an agreement would at least to my non-lawyer brain cause any judge who may have granted a TRO to become very displeased when the CA presented it in their response.
I have put up a sign saying I can instantly crush any crossover vehicles I like, as I consider them ugly and lacking in character.
As I load your car into my crusher, you dispute the legality of my sign. But the court is closed until Monday, and the sign says I can crush your car instantly, no waiting.
Should I be allowed to crush your car today? Or should I have to wait until Monday, so the disputed legality can be resolved?
I think the same applies here.
The court IS deciding, temporarily.
A court can rule that a term of a contract is void because it contradicts public policy, and it certainly can issue a TRO pausing an action which would otherwise be allowed by a contract while resolving a dispute related to it.
For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes.
Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs.
I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.
When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond.
The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.
The policy and its designers aren't incompetent, the intention is to ensure security first when fact-finding can take time to establish a subset of actual abuse. This is common in IT/Security and is clearly communicated in the policy customers agree to as well as something agreed on by all CA's. Security > Uptime is the baseline, and while you may disagree with that and have good arguments for the contrary, the argument is much wider than us and has been extensively hashed out in the working groups and discussions around this and other similar issues and is implemented this way by consensus while weighing valid arguments for different approaches and pros and cons.
Dyndns is not the only use case for having someone control a subdomain without having the full trust of the root domain.
In the past CAs have been more than happy to lie and downplay incidents, because they know their customers don't like revocations and are going to consider switching to a competitor. Maybe not a big deal in practice for a technicality, but it does mean that CAs can not be trusted to always be truthful - and that in turn extends to serious incidents. This is obviously really bad for the web PKI as a whole, so the rule is that any violation of the rules, no matter how small or nitpicky, must be treated as if it were a serious security incident. Don't agree with the rules or think the rule is interpreted the wrong way? File a report afterwards to review the rules.
The entire business of a CA is to be a trusted third party. If you can't be trusted to always follow the rules exactly to the letter, you shouldn't be a CA. Behaving predictably is more important than being right about some technical minutiae regarding the security impact of some incident.
Alegeus, a client of DigiCert, filed a court motion to stop DigiCert from revoking their certificate. The courts granted the temporary restraining order.
There is no "update their legal work to prevent customers from legal action" that can avoid a temporary restraining order that is ordered by the courts to provide time to establish facts. Its simply how the legal process works. "they can't help but" seems unfair as the issue was the client Alegeus was unable to replace their certificates in the revocation timeline.
Further "Going through the courts", they didn't go to the courts a client of theirs did.
Digicert is absolutely not blameless. Outside of the TRO, they have failed to meet revocation windows before and yes, are likely using the TRO in this instance as a shield for that.
However the TRO itself is a concerning development that all CA's need to consider depending on their legal jurisdiction, as it could put companies in a bind of being legally ordered to not to complete something they are contractually and legally required to do within the required timeline and interrupt standard security practices in cases like this.
If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?
In "Common law" when something goes before a court in slightly different situations the second court will look at what the last one decided to see if it makes sense and if they agree. Then the third time looks at both the previous. After many many cases courts will have seen all the arguments and made decisions and so there is no need to educate the court anymore, they will just look at what was done before and decide the same thing.
Not all courts follow the "common law" above though, and I'm not clear how the other systems are different. (I've only lived in common law areas)
Timeline:
Jul 30, 2024 - ORDER granting 2 Ex Parte Motion for TRO. IT IS HEREBY ORDERED that DigiCert is prohibited from revoking the security certificates for the Alegeus Websites for a period of seven (7) day
Jul 31, 2024 - NOTICE of Appearance by Jess M. Krannich on behalf of DigiCert
...
Aug 3, 2024 - DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
Looking at that timeline, I can see why other CA forum members are asking "are you really taking this seriously?" The whole point is that such a restraining order was definitely a bad call by the judge and should absolutely have been contested immediately by any legal team that was interested in representing the security interests of the CA Browser Forum (which, ya know, members of the CA Browser Forum should do in such cases, hence why they're in the CA Browser forum). The fact that DigiCerts legal team did show up for that TRO and then did not act to try to defend their ability to secure their certs, is a bad thing. If you want to be a CA, you gotta be willing to defend your need to act in the name of security, and to defend that in a social, business, and legal context. The point of CAs in the world is not to make money, it is to provide security services. Responding with "hey, but my legal obligations in my country mean I can't always do that" is a valid explanation, but it also means that the rest of the CA Browser Forum should probably not trust you.
It's true that DigiCert could have refused to cooperate with Alegeus and fought the court order instead of cooperating with them to rotate the certificates ASAP. But that would have taken a lot longer than 5 days, even if they eventually won. If the CA Browser Forum has such a strong security interest in swift revocation, it's hard to see how further delaying the revocation in order to provoke a legal battle promotes that interest.
I'll point out notice of appearance doesn't mean they COULD have done anything. The TRO was granted for 7 days already.
From a strategic position, they probably saw they could work with the client and withdraw the TRO faster and cheaper than challenging the TRO in court. Its unlikely a challenge would have significantly decreased the time they were required to not act under the TRO.
Additionally, it seems more prudent to me that they should drop Allegius as a customer than make the guy resign. The current situation seems like if an issue occurs again then DigiCert will not revoke the certs according to the timeline they committed to (24h).
[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c11
> If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?
That would be an excellent reason to make sure to impose strict consequences on CAs without regard to whatever legal orders they're subject to, so that the next CA can point to those consequences when arguing against things like a TRO. "If you grant this TRO, the net effect will not be that you get what you want; the net effect will be to destroy our business and still not get what you want".
https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73
Seems extremely reasonable to me.
Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.
How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things?
It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.
I'd imagine that they wouldn't do it to "take a stand" so much as "avoid the risk of getting their stuff broken in the short term" in this scenario, regardless of whichever party loses the blame game. See: the recent WordPress drama, which has turned customers away from both parties involved.
On the flip side, for user impact, it will play out like this: Some bank or other important entity could possibly, for whatever reason, continue using a (presumably expired? unless DigiCert continues issuing anyways; note that most likely, they will not.) DigiCert certificate after the cut off date, which will lead to users receiving errors. Some of them will have HSTS setup, which will lead to an emergency situation where they have to issue a new certificate ASAP, as it will basically halt their business until they do. For places where there is no HSTS, users may be instructed to simply bypass the certificate warning temporarily, and support lines will be absolutely swamped until they actually fix the problem.
The WordPress situation is quite different. You don't have to use WordPress. Users don't even know what the Web PKI is to find an alternative to it, not that there is one or will be one.
They don't have free rein to do whatever they want. If they chose to remove them, they are going to have to be ready to defend themselves in court.
If the browsers removed DigiCert, DigiCert could certainly sue based on the harm to their business. They could argue that the browser vendors were inconsistent with the application of their rules. They could argue that the browser vendors were unfairly competing by kicking them out.
Not saying they would win all these, but they would certainly fight it - the alternative would be the end of DigiCert, they aren't going to go down without a fight in whatever venue they can find.
... and DigiCert did not try to appeal the TRO.
...unless they notice the breakage, and you tell them to do so to stop it.
That all assumes DigiCert continues issuing certificates they know won't be trusted. I assume most likely what will actually happen is they have to start selling certificates from another CA, but failing that they're probably just going to stop issuing certificates before the cutoff.
Note that this is not a "both" but a "any of them". One can disagree with the other on this and still cause wide breakage.
This gives certificate owners ample time to look for a different issuer and no certificate buyer would deliberately purchase a certificate from an issuer when they know some percentage of users will not trust that cert.
So for the end users, everything will keep working: the existing digicert certs stay valid and newly refreshed certs will be signed by a different authority. There is no need to turn off automatic updates over this.
Between Entrust and Symantec, we've already seen this happen to large well-known CAs and everything remained fine (not for the offenders, but, hey, that's the system working as intended)
“In general” is a broad statement, given that the overwhelming majority of people using the Web PKI have no idea that they’re doing so. DigiCert is not a legible part of the value chain for the average users; they won’t notice that the sites they use switch CAs. This strong disfavoritism towards vendors is arguably one of the Web PKI’s greatest strengths.
The most recent digicert thread smells suspiciously similar to those that lead up to the Entrust debacle.
Their customers would be forced to give their business to Honest Achmed[1].
Where’s a billionaire troll when you need one? (And a form of billionaire trolling that would upset a lot less people than Musk’s version of it.)
Would be even funnier if the billionaire’s name actually was Achmed
Interesting, I hadn’t heard that, is there anywhere I can read more about it?
> You could purchase a root off an existing CA,
As well as a sub-CA, I remember in theory you can have two independent CAs with cross-signing… but do browsers actually support that?
Sub-CAs: Not really. Operational risk to the parent CA is huge, you'd be hard pressed to get any current public CA to sign an issuing CA to be operated externally. Cross-signing still works (though it is the stuff of nightmares in many cases) but again you have to have money and a CA willing to do it!
Anyone can get a cert from lets-encrypt. No users of your website care how trustworthy your CA is. And CA's are too big to fail, so you barely need to worry about your CA being removed by the trust store. So you can only compete on price.
If we want to change this, we need a way for a certificate to be co-signed by multiple CA's. So that the certificate can be presented to the user, and they can figure out if any trusted CA of theirs has signed the certificate. This way, revoking trust in a CA becomes easier, because people should have multiple signatures on their certificate. That means, all of a sudden, that the quality of a CA actually matters.
Whilst it might seem this is already possible, it is not. Cross-signing is only a thing for intermediate certificates, and does not work. You can also have multiple certificates for the same key, but when starting a TLS session, you must present a single certificate. (This seems to have changed for TLS 1.3, so perhaps this is already possible?)
In theory, you could make an intermediate CA and get cross signed certs from multiple CAs (hopefully with Name Constraints), your intermediate CA signs your server cert, and you include all your cross certs and the intermediate certs for those. And the client figures out which chains it can make and if it likes any of them.
But experience has shown, verification may find a chain with signatures that line up, but the CA is expired or revoked in the local trust store, and reject the verification, even though another chain could have been found with the provided certificates.
And, because of the limited information in tls handshakes from real world clients, it's difficult (maybe impossible) to know which CAs a client would accept, so that the server could present an appropriate chain.
> 2. Sub CAs Operated by 3rd Parties
> Honest Achmed's uncles may invite some of their friends to issue certificates as well, in particular their cousins Refik and Abdi or "RA" as they're known. Honest Achmed's uncles assure us that their RA can be trusted, apart from that one time when they lent them the keys to the car, but that was a one-off that won't happen again.
Trust stores and Browsers specifically have more options than simply remove a CA. A more reasonable approach for distrust process of a CA this size would be to stop accepting any new certificates issued after a certain date. Then existing customers will have a heads up and even if asleep will find out the bad news during scheduled renewal rather than while the responsible employees are on vacation.
https://security.googleblog.com/2017/09/chromes-plan-to-dist...
There are a lot of efforts to get name constraints widely supported, so a given CA can only issue for certain TLDs or even domains, but it's still not there. That could be a viable punishment for misbehaving CAs in the future, or even an intentionally chosen restriction for CAs that want less strict requirements. Government CAs could be a very useful application, if a CA cert could be name constrained to only issue for .gov.cctld at that point we could basically just let them do whatever they want without concern. Likewise for any small regional CAs, if they could be limited to only regional TLDs their blast radius is reduced which also could mean less strict requirements than those applied to CAs able to sign for .com.
It would also be useful for localhost. Create a private CA that is constrained to localhost and *.local, and you don't have to worry about it being used to MitM other sites if it gets compromised.
Obviously having a glut of new private CAs would cause scalability issues for CT logs that are already having issues keeping up with current uses, perhaps CT requirements could be reduced for such single-domain CAs in combination with limited lifetimes.
"The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made.
Therefore, this is truly a security-critical incident, "
That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.
The author of that comment is Andrew Ayer, who also does great writeups about CA incidents and processes on his blog:
A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem.
So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open.
There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.
And the experience earned from the mistake.
This is pretty heavy handed and I don't think you've thought through what the consequences of that may look like. The historic way to deal with a problematic CA is to prevent them from issuing new certificates or renewing certificates (after taking care of immediate damage, of course). There are lots of legitimate companies that use Digicert and they should have an expectation of being able to continue business in the short term while they find a different certificate provider.
I specifically say the historical way to deprecate a certificate authority is to prevent them from issuing new certificates. Since certificates have a finite lifetime, the certificate authority would as well and would have immediately no further revenue from certificates.
I am saying "pulling the plug" without notice is going to take down tens of thousands of bystanders and anyone using those certificates for business would be unable to conduct business securely online, which would be disastrous. For example amazon.com has a DigiCert-issued certificate.
> The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier.
and
> The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time.
SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it.
OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues.
EDIT to add: DigiCert has a response in a different thread here: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 that would appear to contradict my speculation. Specifically
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
So did the judge just not read the TOU before signing the TRO?
I wonder if the CAB forum would have standing to sue Alegeus and/or that judge for interfering with the PKI process with an invalid TRO.
Maybe if DigiCert had not decided to make that comment, Sectigo would have been willing to stay quiet...
I understand that the TRO prevented them from revoking approximately 70 certificates, and there really is nothing they could have done differently in that case. Their other revocation failings are inexcusable.
Obviously draw your own conclusions, but I actually laughed out loud at the following statement from DigiCert:
"In reality, our letter to you was consistent with our desire to promote open and honest dialogue."
Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.
As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me.
Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong.
The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.
> The legal world does not move as fast as the demands of our CA ecosystem. Our legal approach was to work with the complainant’s legal team to get the TRO dismissed in 5 days.
The court would not have dissolved the TRO in anywhere close to 5 days without the complainant's cooperation, even if Digicert had an ironclad argument for doing so. Digicert made the right choice to get the certificates rotated as fast as possible - and I don't think Sectigo intends to argue it would be better to stand on principle even if that makes the revocation slower.
... isn't that exactly the legal button the complainant pushed to prevent the revocation?
Temporary restraining orders are the biggest exception. If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
So if DigiCert's irreparable harm was great would that prevent it? Like legally requiring CAs to follow their revocation policies or pay millions in damages?
So a customer could go right ahead and get a TRO but long term it will cost them less than making sure their infrastructure can handle this rare event?
For all the back and forth in the CAB thread, it doesn't seem we're any closer to finding an escape valve for that, which seemingly would have to come from the CAB because there isn't even really a mechanism for courts to decide not to issue TROs about cert revocation, short of somehow taking a case around this to the Supreme Court (which isn't going to happen).
Another commenter mentioned in a sibling thread the possibility of using future-dated revocations. The CA could be mandated to publish such a revocation immediately, such that it takes effect by the 24-hour deadline. Once published, the revocations themselves should be irrevocable. This would also need to be accounted for in the CA's customer contracts.
The legal system is almost certainly going to view future-dating and immediately publishing revocations poorly in any civil action where a customer claims harm.
It sounds to me there are things digicert could have done better, without violating the court order:
- they could have revoked all the other certs that weren't protected by the TRO. They did not.
- they could have contested the TRO sooner. But they didn't.
- Possibly, they could have given customers more warning, since they didn't notify customers until much of the 24 hours was gone
That said, IMO I don't what they did is necessarily irredeemable. But they need to come up with, and execute on, a plan to make sure something like this doesn't happen again. And threatening legal action is really not a good way to re-establish trust.
I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
> and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
In your view, are there any meaningful issues going un-addressed currently?
Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children.
And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies.
Meanwhile:
- Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet.
- Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
Basically the missing '_' was supposed to allow DNS providers who allow programmatic DNS record creation to filter out unauthorised certificate creation. So the certificates created without it could have been unauthorized by the owner of the domain they claim to certify.
For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation:
- Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1894560, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 - incident due to the issuance of some certificates with incorrectly-capitalized phrases in the certificate's Business Category field; baseline requirements require revocation within five days but DigiCert dragged that out much further
- Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1910322, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1910805, DigiCert information page https://www.digicert.com/support/certificate-revocation-inci... - incident due to incorrect CNAME-based domain validation (failure to check that the CNAME started with an underscore); baseline requirements require revocation within 24 hours but DigiCert was stopped by the TRO and revoked after five days.
Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
> Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
:chefs_kiss:
> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
Wot. They had _multiple_ services generating random numbers?
They hiring? :]
To issue a cert for a client (say example.com), the CA sends a random number challenge to the client (say 12345678). The client then responds by making a DNS record for _12345678.example.com but due to a bug, DigiCert accepted 12345678.example.com instead.
Apparently there's a rule that says any such certs must be considered mis-issued and must be revoked by the issuing CA within 24 hours.
The CA talked to some clients who said it would be too much trouble to replace their cert on such short notice. One of those clients filed a lawsuit with the court system, which responded by issuing a restraining order telling the CA not to revoke the cert.
First of all, this is a dumb reason to revoke certs. It's difficult to imagine a situation where any of the mis-issued certificates correspond to an actual compromise of somebody's website by a bad actor. But we can perhaps give the benefit of the doubt: I think the intent here is a fail-safe procedure is implemented so revocation is triggered by a broad class of circumstances, which is constructed so that it's easy to check.
Second, this is a dumb thing for a website to file a lawsuit about. If you have to replace the cert on your website because your CA made a booboo, the engineer-hours needed to change out your new cert are far less than the lawyer-hours needed to try to work through the court system.
Third, it's dumb for people to blame DigiCert for the TRO. The TRO was filed by a service provider called Alegeus. If DigiCert doesn't follow the TRO, they'll be in contempt of court. Theoretically, a DigiCert employee who presses the "Revoke" button at this point could be sent to jail for it!
Fourth, a certificate revocation is basically a signed message that says "News Flash: I think this certificate might not correspond to this website." A court preventing the publication of such a message is equivalent to a court preventing a newspaper from publishing an article. This is called "prior restraint" and it has a high bar in the US. Prior restraint analysis was not performed by the court before the TRO was issued. I think the court violated DigiCert's due process rights (by not analyzing whether prior restraint applies) and DigiCert's free speech rights (by not denying the TRO as it would be prior restraint). (That browsers have a hair-trigger response to certain kinds of news flashes and will take drastic action isn't the fault of the publisher of the news.)
Fifth, it's obvious there ought to be some technical means to prevent this kind of situation in the future: What is the protocol to handle a case of a CA refusing (or court-ordered not to) revoke certs known to be compromised? I think you need to have a system that allows a quorum of CA's not including the issuing CA to speedily revoke certificates, or an entire CA. (A public real-time quorum of a dynamic set of validators passing messages they want to reach consensus on: Perhaps this is a good use case for a blockchain?) Of course you also have to deal with the problem of, what if the next TRO targets the entire validator set? You'd need to be sure those participants are jurisdictionally diverse (or anonymous) so it would be difficult for a single entity to compromise the entire system at once.
An underscore. WTF.
2. It is a pretty dumb thing for a website to file a lawsuit about, that is true.
3. DigiCert deserves a lot of blame for the TRO. DigiCert received one TRO for one client who had 72 certs from DigiCert[1]. What they should have done is continue with revoking the other 80,000+ certificates in the 24 hour time window and then inform the CABF that they have 72 (or however many) holdout certs caused by legally binding temporary restraining order. They should have been proactive in complying and in their communication. DigiCert did not do that though. They delayed all 80,000+ certificates for 5 days instead. That's not good enough.
[1] - https://www.courtlistener.com/docket/68995396/2/alegeus-tech...
1) To DigiCert's point: If certs need an emergency revocation but it will impact a service which say: provides life saving services, or keeps the electricity on for the majority of a country, would it not be wise to file them as a one-off "exceptional circumstance". I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential." I wish that that was the direction the debate had gone in.
For instance, What is considered an 'exceptional circumstance'? What kind of services are covered, and what are not?
Personally, I would think that things like: health, heat, water, electricity, and physical security (prison and law enforcement) are all potentially essential areas. They are industries that ought be able to request an emergency, 48-hour exception if they know they can't meet it within 24 hours and their services will go down as a result. I feel like two days should be enough time for just about any organization to work through a certificate issue, unless it's a long holiday, or something very, very niche.
I think that, to a degree, Tim Callan (Setigo CEO) was being unreasonable in expecting DigiCert to not offer any kind of possibility for exceptions. Some services should not go down, just because it goes against the principals of computer security. It hate saying that, but it's true. Keeping the ICU running matters more than whether the hospital is following best security practices during an emergency.
Could it cause more problems by ignoring best practices? Possibly! Will enforcing best practices possibly kill someone? If the answer is anything other than a firm "No", then it is secondary to protecting that service.
2) To Sectigo's point: We should not allow any CA to hide behind Policies or poorly written MSAs. If things went the way they did because they were allowed to go that way, then that means you should learn from those things in the post mortem! Take steps to shore it up! Try and prevent other companies from following suit, otherwise more will take action whenever it meets their own best interest. It is disappointing that this part seems to fell into snarky retorts too, because there were some legitimate means to discuss this.
For instance: Instead of barring from someone from being allowed to file a TRO, simply have an agreement in place that before any legal action like a TRO is filed, the customer will meet with the CA and a emergency mediator. Just take 30 minutes to one hour to see if you can work things out before the customer submits a TRO!
It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption.
Neither side wants a TRO! It makes DigiCert look weak during an emergency, and it makes Alegeus (the company that filed the TRO) look incompetent, desperate, and underhanded.
The crux of what Tim Callan (Sectigo) was getting at, is that there needs to be a correction to DigiCert's policies. It's blaringly obvious. DigiCert were, in a way, "legally attacked" in a manner that should be prevented in the future, as best they can prevent it.
DigiCert lackadaisically shrugging their shoulders and saying "B-But...that goes against Mozilla policy!" is just deflection and meaningless. DigiCert can go to the trouble of sending legal council after Sectigo for comments on Bugzilla, but they can't use legal council to protect DigiCert from surprise TRO's? Really? Bugzilla feedback...that is the legal issue? Not DigiCert being sucker punched by their own customers?
The whole thing is just so aggravating. Both sides need to get over themselves and try to work together. They don't need to like each other, but they should do what is best for the industry. Each side sending out daddy lawyer to fight for them completely misses the point, and kills the chance for constructive feedback.
Have you read the court filings? Alegeus did indeed have this discussion, explaining that they have a detailed manual process for dozens of distinct websites, and could not complete it within the notice period even though they began working on it immediately. They escalated it all the way up to the DigiCert CEO, and filed for a TRO only after DigiCert told them - exactly as the Bugzilla discussion participants wanted - that no exemption would be offered. That's the key point they seem to be missing: taking a hard line on the revocation timeline does not deter legal action but generates more of it.
Privacy of personal data is also essential.
If anything, such context should warrant expediting the process of revocation.
1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it.
2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it.
3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it.
4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert.
The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice...
Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.
Certificate authorities are required by contract (the CA/Browser Forum Baseline Requirements is a contract they agree to when being granted trust as a CA) to revoke miss-issued certificates within 24 hours of the time they learn of the miss-issuance.
One of their customers filed a court case & got a temporary restraining order (TRO) preventing DigiCert from revoking ≈70 of those certs within 24 hours.
DigiCert then proceeded to delay revocation beyond the 24 hour limit *for all the certs, not just the ≈70 that the TRO covered.
That is a violation of the CA/Browser Forum Baseline Requirements. Failure to revoke ≈70 certs because of a court order could be accepted, but failure to revoke thousands of other certs (putting customers at risk) should not be.
Whether you believe them, or not, there was a valid reason given. They also said they fixed this particular issue internally, so I guess we'll see.
DigiCert had over two million certs issued per the "businessEntity" case sensitivity bug thread, which was 2023.
They revoked ~84,000 out of an abundance of caution - my read was they revoked every cert issued from the (now defunct) system digicert called "OEM" - which had the bug where it wasn't prepending an underscore.
I just read the entirety of the underscore and the case sensitivity threads.
I'm trying very hard to be charitable. I don't know anything about digikey. But it could be incompetence; legal dept getting cold feet because someone filed a TRO in a court against the company; malice?
i don't disagree with what you're saying, either; digicert never really answered why they couldn't get the other 83k revocations done faster, hoping "we couldn't do it, sorry" in the revocation bug report would suffice.
For example, in the first issue (delayed revocation after incorrect capitalisation), DigiCert chose to not revoke several thousand certificates within the required 5-day window for reasons ranging from "is in use in embedded devices and can't be replaced in a timely fashion" (this is a failure of the customer's automation and planning, not a failure of DigiCert, and is not a reason for DigiCert to not revoke) to "is being pinned in a mobile app and the app needs to be rebuilt and pushed to app stores" (this is again a failure of the customer; the customer should have foreseen the event that their certificate would be revoked, and, if they really wanted to pin, should have also had a hot standby certificate from another CA pinned, for example). Further, DigiCert's own certificate policy at time included the following text (in section 1.4.2):
DigiCert strongly discourages key pinning and shall not consider it a sufficient reason to delay revocation.
The other more recent incident (the TRO) would have restrained DigiCert from revoking approximately 70 certificates. DigiCert chose to extend this delay to tens of thousands of certificates, issued to other customers who sought no restraining orders. DigiCert chose to ignore their own policy, and allowed this decision to delay revocation. That was entirely their own choice, to appease their own customers at the expense of complying with both their own policy and the Baseline Requirements.
Are you starting to see a pattern?
You are only suggesting they could have handled it worse. Why would they take legal action against the CCO for statements on a bug report other than to squash transparency?
Can you provide a link to this? I looked Callan's comments on the cited bug[1] and none of it struck me as being even a little off the rails.