Hacker News Reader

Comment by nneonneo | Hacker News Reader

In a nutshell:

DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation.

Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner.

Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.

View on HN

Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far.

For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes.

Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs.

I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.

I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that.

When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't.

Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond.

The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.

>Could have been a problem, if Alegeus was a dyndns provider - but they aren't.

AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.

There was something wrong. They hadn't been issued in accordance with the baseline requirements.

The middle ground here is DigiCert should have been able and continued with the revocation of all the other certificates not impacted by the TRO within the policies and contractual agreements it has.

The policy and its designers aren't incompetent, the intention is to ensure security first when fact-finding can take time to establish a subset of actual abuse. This is common in IT/Security and is clearly communicated in the policy customers agree to as well as something agreed on by all CA's. Security > Uptime is the baseline, and while you may disagree with that and have good arguments for the contrary, the argument is much wider than us and has been extensively hashed out in the working groups and discussions around this and other similar issues and is implemented this way by consensus while weighing valid arguments for different approaches and pros and cons.

The fact that those certificates had been issued without a secure method of verification is the thing that was wrong with those certificates.

Dyndns is not the only use case for having someone control a subdomain without having the full trust of the root domain.

Revocation rules are intentionally very strict, in order to prevent CAs from weaseling out of them. If you leave open any possibility of a CA delaying or refusing revocation because of some form of "it's an administrative issue, it isn't that serious anyways", every incident will inevitably result in a weeks-long discussion about the real impact of the incident and whether this one is "serious enough" to require immediate action.

In the past CAs have been more than happy to lie and downplay incidents, because they know their customers don't like revocations and are going to consider switching to a competitor. Maybe not a big deal in practice for a technicality, but it does mean that CAs can not be trusted to always be truthful - and that in turn extends to serious incidents. This is obviously really bad for the web PKI as a whole, so the rule is that any violation of the rules, no matter how small or nitpicky, must be treated as if it were a serious security incident. Don't agree with the rules or think the rule is interpreted the wrong way? File a report afterwards to review the rules.

The entire business of a CA is to be a trusted third party. If you can't be trusted to always follow the rules exactly to the letter, you shouldn't be a CA. Behaving predictably is more important than being right about some technical minutiae regarding the security impact of some incident.

No. Our cert infrastructure needs to meet the highest standards possible, this is not an area where "welp, no harm done, don't worry about it" is acceptable.

This seems like a misunderstanding of the TRO.

Alegeus, a client of DigiCert, filed a court motion to stop DigiCert from revoking their certificate. The courts granted the temporary restraining order.

There is no "update their legal work to prevent customers from legal action" that can avoid a temporary restraining order that is ordered by the courts to provide time to establish facts. Its simply how the legal process works. "they can't help but" seems unfair as the issue was the client Alegeus was unable to replace their certificates in the revocation timeline.

Further "Going through the courts", they didn't go to the courts a client of theirs did.

Digicert is absolutely not blameless. Outside of the TRO, they have failed to meet revocation windows before and yes, are likely using the TRO in this instance as a shield for that.

However the TRO itself is a concerning development that all CA's need to consider depending on their legal jurisdiction, as it could put companies in a bind of being legally ordered to not to complete something they are contractually and legally required to do within the required timeline and interrupt standard security practices in cases like this.

Would a judge not consider potential harm to each party if the TRO is granted or not?

If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?

They should - but only if a good lawyer points that out to the courts. Remember the courts are not experts in the technical details in question (and should not be - there are far more details that could come before the court than any human has time to learn), the job of lawyers is to quickly give them enough education to figure it out.

In "Common law" when something goes before a court in slightly different situations the second court will look at what the last one decided to see if it makes sense and if they agree. Then the third time looks at both the previous. After many many cases courts will have seen all the arguments and made decisions and so there is no need to educate the court anymore, they will just look at what was done before and decide the same thing.

Not all courts follow the "common law" above though, and I'm not clear how the other systems are different. (I've only lived in common law areas)

Well, if the CA forum had a really clear automatic sanction, I would expect the lawyer to raise that concern.

Other comments have since suggested this was an emergency order where the CA didn't have their lawyer present. The courts said "don't do anything for a week while the real paperwork happens". I don't know the cases in question well enough to figure out if that is correct, but it seems reasonable. But also a week is not very long.

An answer for the civil law countries I know of: The laws tend to be more detailled to avoid the situation you describe. Also, there is still a lot of referencing old cases, but it is considered guiding, not binding.

hmm Isn't there an applicable standard of review for legal cases?

The TRO was filed for and granted without DigiCerts input. By the time they responded, both parties filed to vacate the TRO (3 days later). Judges I expect weigh, to the best of their abilities, the perceived harm in granting/not granting the TRO. Considering the technical literacy of our court system I expect that leaves much to be desired

It was filed and granted without DigiCerts input, yes, but DigiCerts lawyer showed up within 24 hours but then did not do anything. See the courtlistener link: https://www.courtlistener.com/docket/68995396/alegeus-techno...

Timeline:

Jul 30, 2024 - ORDER granting 2 Ex Parte Motion for TRO. IT IS HEREBY ORDERED that DigiCert is prohibited from revoking the security certificates for the Alegeus Websites for a period of seven (7) day

Jul 31, 2024 - NOTICE of Appearance by Jess M. Krannich on behalf of DigiCert

...

Aug 3, 2024 - DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED

Looking at that timeline, I can see why other CA forum members are asking "are you really taking this seriously?" The whole point is that such a restraining order was definitely a bad call by the judge and should absolutely have been contested immediately by any legal team that was interested in representing the security interests of the CA Browser Forum (which, ya know, members of the CA Browser Forum should do in such cases, hence why they're in the CA Browser forum). The fact that DigiCerts legal team did show up for that TRO and then did not act to try to defend their ability to secure their certs, is a bad thing. If you want to be a CA, you gotta be willing to defend your need to act in the name of security, and to defend that in a social, business, and legal context. The point of CAs in the world is not to make money, it is to provide security services. Responding with "hey, but my legal obligations in my country mean I can't always do that" is a valid explanation, but it also means that the rest of the CA Browser Forum should probably not trust you.

Any CA who believes that the courts in their country could not prevent them from revoking certificates is confused at best.

It's true that DigiCert could have refused to cooperate with Alegeus and fought the court order instead of cooperating with them to rotate the certificates ASAP. But that would have taken a lot longer than 5 days, even if they eventually won. If the CA Browser Forum has such a strong security interest in swift revocation, it's hard to see how further delaying the revocation in order to provoke a legal battle promotes that interest.

Ah yes, I missed that note, thanks.

I'll point out notice of appearance doesn't mean they COULD have done anything. The TRO was granted for 7 days already.

From a strategic position, they probably saw they could work with the client and withdraw the TRO faster and cheaper than challenging the TRO in court. Its unlikely a challenge would have significantly decreased the time they were required to not act under the TRO.

They could have revoked the certificates of companies not listed in the TRO. Instead revocations for _all_ customers were delayed by 3 days over the time limit [1].

Additionally, it seems more prudent to me that they should drop Allegius as a customer than make the guy resign. The current situation seems like if an issue occurs again then DigiCert will not revoke the certs according to the timeline they committed to (24h).

[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c11

Agreed on both counts (I made the same points in other comments :))

> Would a judge not consider potential harm to each party if the TRO is granted or not?

> If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?

That would be an excellent reason to make sure to impose strict consequences on CAs without regard to whatever legal orders they're subject to, so that the next CA can point to those consequences when arguing against things like a TRO. "If you grant this TRO, the net effect will not be that you get what you want; the net effect will be to destroy our business and still not get what you want".

Here is the last reply from Callan:

https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73

Seems extremely reasonable to me.

What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?

DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days.

It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients.

Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-inci...) says clearly:

"Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."

I think the eventual correct option is pretty clear.

A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business.

Make that series of consequences clear, and companies will stop trying that trick.

I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs.

The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?

The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor. If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.

A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough.

If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations).

If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.

> A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done.

You say that and yet Teams has 320 million people using it, and I bet almost none of them enjoy the experience. Sometimes you just have to work with what you're given. Given "incompetence", we might as well throw in all of Azure.

> Let Salesforce figure out how to automate certificates, they've been in the business for long enough.

That might be true for DV, but there's classes of certificates that take at least weeks to obtain, think BIMI VMC or codesigning.

> You say that and yet Teams has 320 million people using it

Yeah, there are tons of companies who chose Teams as a vendor (because it's bundled with other stuff), and inflict it on their employees. It was still absolutely their choice.

DigiCert and other CA's need to write in their terms that failure to follow the policies and revocation timelines can/will result in termination of the contract. Alegeus should have been dropped as a customer as soon as the incident was resolved and refused further products/renewals.

Use of a TRO protects you short term, but results in having to migrate to a new CA medium term. You can't stop them from using TRO's but you can make it not worth it.

Is the CA allowed to stipulate that the customer (before being dropped) needs to pay a significant sum for, say, "expenses" if they resort to this kind of TRO?

Possibly, I'm not sure if there is precedent for charging a company for taking valid legal actions, only vexatious in contract law. Probably depends on the legal jurisdiction and courts.

How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?

I can declare my right to do something as much as I want... that doesn't mean I get to do it, certainly not if a judge decides I can't.

The better question is, how do you prevent this tactic from working?

For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.

Well, you can contest the motion for a TRO, for a start. Digicert failed to do so.

You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.

You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.

Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted.

They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days.

I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43

I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong:

> You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.

Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs.

> You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.

By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.

Just to be clear, the whole incident covered over 80,000 certificates. The TRO was applicable to only those of one subscriber - just over 70 certificates, yet caused the revocations of all 80k+ to be delayed.

To add to this, 3 days after the TRO was filed both parties moved to vacate the TRO.

DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED

I'm not sure DigiCert could have done anything about the TRO or the impacted certs, but it should have been able to move forward with the revocation of all other certificates. That IMO is the real issue/failure, alongside the concern/impact of TRO's on security processes in the future.

> By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.

Yes, I think this would have been appropriate action. If the contractual language is extremely clear between the CA and the subscriber, there is no legal basis on which the customer can prevent revocation. The fact they found a court that doesn't understand technology is frankly irrelevant. This detail is exactly why Tim and other parties are requesting the exact language of the agreement between Digicert and the subscriber that filed the TRO. A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR, or to take actions which are detrimental to the entire Internet. This is exactly the type of circumstance where you do what you are required to do, and then sort it out afterwards. Any appeals court would have easily overturned the TRO as it has no legal basis.

> A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR

Yes, it absolutely does. "I think the court will agree with my view of what the contract says once the case is heard in full" is not a valid reason to disregard a TRO.

> or to take actions which are detrimental to the entire Internet

That would be harder. But a delayed revocation stemming from a flawed validation process, when the CA is responsible for the flaw and knows that the result of the validation was in fact correct, simply does not cause any detrimental effects to the entire Internet.

You could just require publishing the revocation immediately with an effective date in the future.

Of course, if that system had been in place, DigiCert would probably be facing hundreds of lawsuits from businesses disrupted through no fault of their own rather than inside baseball PKI drama.

> The better question is, how do you prevent this tactic from working?

Make it clear that if it works, it will only work once.

The CABF should adopt policies that any such legal action or any request for extension will be considered a public declaration that the customer's application is incompatible with the requirements of the Web PKI and that not only will the current CA refuse to renew the certificate but it will be publicly documented in the Bugzilla so that no other CA will issue certificates covering any of those names nor any new names for the same company without an affidavit explicitly stating that the issues preventing compliance have been resolved and the company acknowledges this and commits to never doing so again.

Existing names that were successfully revoked in time can be renewed but neither the problematic one(s) nor any new ones will be allowed.

If they then file for another TRO in the future they may still get a short-lived order but the existence of such an agreement would at least to my non-lawyer brain cause any judge who may have granted a TRO to become very displeased when the CA presented it in their response.

It’s Saturday and the courts are closed. You park in a car park I own.

I have put up a sign saying I can instantly crush any crossover vehicles I like, as I consider them ugly and lacking in character.

As I load your car into my crusher, you dispute the legality of my sign. But the court is closed until Monday, and the sign says I can crush your car instantly, no waiting.

Should I be allowed to crush your car today? Or should I have to wait until Monday, so the disputed legality can be resolved?

That’s actually a question for you. You won’t truly know if you’re allowed to until a court decides. You can choose to proceed and crush it, and then deal with the consequences of doing so if it turns out you were wrong. Likely you’ll end up owing damages to the owner of the car, perhaps even punitive damages on top.

I think the same applies here.

The TRO actually means you aren't allowed, that's the point. It's an ordered injunction that legally obliges you from not acting until the courts can review the facts.

The court IS deciding, temporarily.

In this case, yes. That’s pretty cut and dry for the time being. However regarding the analogy I was replying to, I was pointing out that it’s less a situation of what you’re allowed to do and more one of what you believe you’re allowed to do, and weighing the consequences of being wrong against upholding your stated terms. In other words, something you should probably discuss with a lawyer.

There is no TRO in the car park analogy.

But there is here, so its a flawed analogy.

> How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?

A court can rule that a term of a contract is void because it contradicts public policy, and it certainly can issue a TRO pausing an action which would otherwise be allowed by a contract while resolving a dispute related to it.

Because the judge doesn't know the details of how PKI works, and either the ToS for digicert doesn't spell out that they can revoke certs at any time (which would be problematic for a CA), or the judge didn't read, or didn't understand the ToS.

The order would normally be a matter of public record, and the ecosystem should follow these events carefully and ensure never to issue a certificate to such a company again as it undermines the entire system.

Isn't Sectigo Comodo? Extra rich coming from them.

No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.

Reply on news.ycombinator.com