Health industry company sues to prevent certificate revocation
hezmatt.org
hezmatt.org
It seems obvious enough how this would come about:
1. The healthcare company knows better than to do a Crowdstrike, where "just a configuration change" is rushed out with inadequate testing, for "security". As even a configuration change could trigger a latent bug, they agree configuration changes should be rolled out gradually, and not applied during clients' peak operating hours.
2. The healthcare company could have used "Let's Encrypt" to get certificates for free, if they wanted short-term certificates. But instead they paid $$$$ to DigiCert for certificates with a 12 month duration.
3. Even if Alegeus had read the small print saying mis-issued certificates can be revoked with only 24 hours of notice - they quite reasonably assumed DigiCert would have controls in place to avoid mis-issued certificates. Hasn't DigiCert told the browser vendors precisely that?
I mean, sure, in theory there should be controls in place, but no company or process is perfect. That's why there's a process in place to respond to problems with issuance in the first place.
DigiCert is responding in the way the browser vendors mandate; 24-hour revocation, per the article, is a baseline requirement to be a CA.
This makes no sense. The revoke certs when there's been an incident. Telling customers that they might need to revoke certs in the event of an incident should mean that they are going to be free from incidents??
If Stripe tells me webhooks might be delivered more than once, that does NOT mean I would assume they have controls in place to prevent webhooks from being delivered more than once. That's just wild to assume.
To make it clear what I mean, imagine this hypothetical conversation:
Alegeus: "We need a certificate with a longer life than the 3 months we can get from Lets Encrypt"
DigiCert: "That is our core business. We will sell you a 12 month certificate. It may be revoked early, if it was mis-issued."
Alegeus: "I particularly need a 12 month certificate, as I mentioned. Is revocation likely? Do you often mis-issue certificates? To your knowledge, are you currently mis-issuing certificates?"
CA/Browser Forum: "Yeah DigiCert, is it likely? Are you currently mis-issuing certificates?"
DigiCert: "We have a number of organisational and technical controls in place to prevent certificates being mis-issued. We do not have a history of mis-issuing certificates, and to our knowledge we are not currently mis-issuing certificates. Our controls have passed third-party audits."
Alegeus: "OK, so to the best of your knowledge as certificate industry experts, this 12 month cert you're selling me will be valid for 12 months?"
DigiCert: "To the best of our knowledge, yes."
Supermarket: "We need a candy bar they we can sell to our customers"
Hershey: "That is our core business. We will sell you candy. It may be recalled, if we discover problems."
Supermarket: "Is a recall likely? Do you often recall candy? To your knowledge, are you currently selling bad candy?"
FDA: "Yeah Hershey, is it likely? Are you currently making toxic candy?"
Hershey: "We have a number of organisational and technical controls in place to prevent contaminated candy. We do not have a history of contamination, and to our knowledge we are not currently contaminating candy. Our controls have passed third-party audits."
Supermarket: "OK, so to the best of your knowledge as candy experts, this candy you're selling me will be good to sell?"
Hershey: "To the best of our knowledge, yes."
If someone warns you about a possible eventuality, you simply don't pretend that the eventuality is impossible. That's why they warned you. Obviously nobody wants that. It's just plain ignorant to have a shocked Pikachu face when the thing they warned you might happen in extenuating circumstances happens.
2. It's also true that there are rules, they exist for a reason, these Alegeus guys aren't special, and dragging this stuff into the courts is antisocial in the extreme. So it wouldn't be unreasonable for browsers to, say, explicitly blacklist their domains for as long as there conceivably might be noncompliant certs out there.
You know who else thinks there are rules, that the rules exist for a reason, and that draconian punishments should be applied to people who try to derail attempts to enforce the rules?
Judges. Like the judge that granted this temporary restraining order.
Browsers do have a relationship with their users, who may very well be relying on them to enforce the CA/BF validity rules, or if they can't do that to provide equivalent security guarantees. And in particular many of the users rely on their browsers to protect them by blocking suspicious domains in various ways. So blocking the domains wouldn't be arbitrary, capricious, discriminatory, or anything like that.
Judges don't have infinite sua sponte powers to order just anybody to do just anything. Although I admit they sometimes act as though they did, especially when they feel like their authoriteh has been disrespecked.
Anyway the browsers aren't gonna do it. There's zero chance. So legalities don't really arise. I'm just saying it wouldn't be unreasonable.
If a browser vendor disrupted the websites, because they knew of this court order and decided to take deliberate action to defy the authority of the court, that's contempt of court.
The browser vendor then gets pulled up in front of the judge who granted the restraining order. The browser vendor says "ah, but the court order only says the certificate can't be revoked, we didn't revoke it we blacklisted it" and the judge says "that's a great argument so I'm not going to send you to jail, I'm going to send you to prison instead"
> IT IS HEREBY ORDERED that DigiCert is prohibited from revoking the security certificates for the Alegeus Websites for a period of seven (7) days, or until the Court is able to schedule a hearing on the Motion, whichever is earlier.
The order specifically mentioned Digicert and nobody else. The order doesn't magically apply to everyone else in the world.
1. Alegeus creates white-labeled sites for their customers, but those sites are hosted on customer domains (usually for ease of sharing login cookies across subdomains on the customer), e.g. alegeus.customerdomain.com.
2. So the issue isn't necessarily that Alegeus can't update their systems, it's that their many customers can't update their DNS records fast enough to point to the new certs.
IMO, if you built a system (and I'm talking about DigiCert here) that expects thousands of customers can update everything at the drop of a hat to update their code, you built a bad system. I don't think it's too different from CrowdStrike, except it's as if CrowdStrike gave 24 hours notice instead, saying "Hey, all of our customers, just a heads up we're going to break all your machines in 24 hours, you really should have planned better if you can't handle that."
Alegeus is the one suing here because they've got tons of customers, but make no mistake this would cause outtages for tons of other companies that use DigiCert.
Unless you use DANE (which nobody does, and the browsers don't check it even if they did), you do not have to change any DNS records, or touch the DNS at all, to update a certificate. There is no DNS involvement. Period.
DANE should be the norm, but it's not.
You can also verify over HTTP or email (egad...). You would think that if Alegeus controlled the HTTP servers and not the DNS servers, Alegeus would opt for HTTP verification, but I concede that they may have built everything around DNS verification.
Why inadequate (in the absolute)? This can be automated and let's encrypt allows verification through DNS, moreover this allows verification for wildcard certificates.
Now in this particular case maybe they should have gone through HTTP, and even automated with ACME. But there is nothing inadequate in the absolute in DNS verification. Besides allowing wildcard it also allows verification when you don't control the web server(s), when you don't even have a webserver at all, when the standard ports are occupied for something else, etc.
The point of X.509 certificates is that you can't rely on information you get either from the DNS or from the HTTP server. If you could, you wouldn't need the whole mess in the first place.
Sure, the verification helps, because you have to successfully fool both the client and the CA. But if you can fool one, there's a strong chance that you can fool the other. In the end, the CA is still relying on exactly the same information that the client isn't supposed to have to rely on.
The original idea behind X.509 was that verification would be "out of band", but that turned out to be expensive and non-scalable, so the X.509 world, including the CA/BF, resorted to this very weak kind of verification. They try to backstop it with stuff like certificate transparency, but it's just adding epicycles that aren't particularly reassuring.
If everybody used DNSSEC, then DNS-based verification would be OK. But at that point you ought to just distribute key hashes through the DNS and dispense with X.509 entirely. That's actually what should have happened, and probably what would have happened if X.509 hadn't still been such a cash cow at the times when the various standards solidified beyond all chance of improvement. Because of that "cash cow" status, there was a lot of obvious sabotage aimed at entrenching X.509 and fighting any attempt to improve the situation. And now we're stuck with it.
The practical: you can't reliably run DNSSEC everywhere Chrome runs. Networks get really fucky with any even slightly unusual DNS messages.
The principle: because you can't realistically ever declare a "flag day" and deprecate the X.509 WebPKI, you have to support both systems, so DANE doesn't collapse your trust anchors down to a smaller set; it actually adds to the number of things you have to trust.
It's really tragic that the Internet is so ossified. (Not just in this regard, but in many others.)
Admittedly DNSSEC is an essential prerequisite and wasn't really baked, but it was being delayed by essentially the same FUD. And DNSSEC got quite usable, from a pure technology readiness point of view, pretty fast once people started putting a little urgency behind it later on. It's still relatively hard to use, but that would evaporate in six months if there were some impetus for it to get more users.
Not that the browsers helped, mind you (Mozilla wasn't really any better). A switch to DNS as root of trust still could have been done when DANE was finished if there'd been any real will. Much less will than it takes to establish this or that bad-idea standard that's of no value to anybody but advertisers. Or for that matter to set up weird, mutant, off-the-wall, who-asked-for-that, solving-the-wrong-problem-in-the-wrong-way hackery like DNS over HTTP.
Still, by that time, the browsers could point not only at an entrenched system of practice, but also at a ton of broken, clearly-non-standards-compliant middleboxes on the network screwing up DNS, especially in "The Enterprise(TM)". Personally, I had, and still have, zero sympathy for the people who put those boxes there or put themselves behind them. I would have given them exactly zero consideration. But a lot of people somehow seem to think that one system's bad behavior obliges everybody else to bend over backwards to accommodate that system forever after.
There is actually one legitimate knock against DNSSEC: it makes replies bigger, which makes DNS a worse DoS amplifier. I think it's worth it, and if it weren't my reponse would probably have just been to start moving DNS over TCP. But at least there's a somewhat respectable argument there. Strangely, it was never the main argument you heard.
The fundamental problem with this whole line of argument though is that, even if things worked well (they did not; ask Slack) you're still just trading the WebPKI --- with all its warts --- for a system that is even less transparent, and that is de jure operated by world governments. There will, for instance, never be a "DANE Transparency" log; not only because DANE will never be deployed for real, but also because the market forces that coerced CAs into adopting CT don't exist in the DNS.
Per the article, this is required to be a CA. It's not a DigiCert thing, it's a baseline requirement. If you can't revoke your certificates quickly, you risk being delisted by Google/Mozilla/Microsoft et al. Entrust was just recently dropped in part because of this very thing.
1 - they made the screwup;
2 - DigiCert appear not to have put language into their MSAs and so forth that this was a thing that could happen and customers had better be prepared to do it. Why shouldn't Alegeus sue DigiCert to force DigiCert to conform to the terms of the agreement that DigiCert agreed to with Alegeus?
if you don't have a fast process for replacing this stuff, you're going to have a bad day eventually
This is almost certainly wrong. With the exception of DNSSEC, there's no interaction between DNS and certificates. There's no DNS record that points at a certificate. The HTTP server that terminates the connection is the server that has the certificate.
TXT records can validate a domain, but you can also validate a domain by hosting a file on that domain to show that you control it.
I find it more likely that Alegeus is giving customers appliances or VMs that they run themselves, and those are each configured with a baked-in certificate. Which is bad anyway because it means the customer needs to be involved in the certificate update process.
The lawsuit smells like them trying to buy time to fix their IT problems. Even if it loses or gets tossed out all they needed was that preliminary injunction. It's really sucky for DigiCert who have to choose which party to anger, the courts or the browser manufacturers, just because this healthcare company doesn't have its act together.
> The Service Specific Terms are incorporated by reference into this Agreement
> “Service Specific Terms” mean additional terms specific to certain Services as set forth at www.digicert.com/service-specific-terms, which are incorporated herein to the extent applicable to any specific Services procured by Customer hereunder (which includes the Certificate Terms of Use with respect to Customer’s use of any Certificates).
https://www.digicert.com/service-specific-terms states:
> The Certificate Terms of Use available at www.digicert.com/certificate-terms (the “Certificate Terms of Use”) apply to Certificates other than Qualified Certificates and PKIoverheid Certificates requested or issued by Customer. The applicable Certificate Terms of Use are incorporated herein by reference.
https://www.digicert.com/certificate-terms states:
> DigiCert may revoke a Certificate without notice for the reasons stated in the CPS, including if DigiCert reasonably believes that:
> Industry Standards or DigiCert’s CPS require Certificate revocation, or revocation is necessary to protect the rights, confidential information, operations, or reputation of DigiCert or a third party.
Does that revocation requirement actually apply to these particular certificates?
If I've understood the problem it is that DigiCert was issuing challenge keys for DNS based validation that did not start with the required underscore.
The underscore is required to prevent an attack that only works against sites that allow users to create subdomains.
The particular sites of the plaintiff's clients are not such sites and so has the CA actually obtained "evidence that the validation [...] should not be relied upon"?
I wonder if we could come up with a process where a customer could push another DNS record to redo the verification. They'd need to act fast to beat the 24 hour revocation window, but maybe that's better than a full re-issuance? Probably better to invest in re-issuance agility though.
DigiCert's blog post…
DigiCert Revocation Incident (CNAME Domain Validation) (digicert.com) — https://news.ycombinator.com/item?id=41104504 — 136 points by vitaliyf 1 day ago
… and the Bugzilla entry where DigiCert reported this to Mozilla:
Digicert pauses mass certificate revocation due to legal action from customer (bugzilla.mozilla.org) — https://news.ycombinator.com/item?id=41114794 — 9 points by frakkingcylons 23 hours ago
Edit:
> Therefore, this is truly a security-critical incident, as there is a real risk (not a negligible 2^-150 risk as implied by DigiCert) that this flaw could have been exploited to get unauthorized certificates. Revocation of the improperly validated certificates is security-critical.
Yeah... have they proven that this wasn't exploited? With 83,000 certificates getting revoked this isn't a small scale mistake.
Presumably it would be fairly easy to test, for the vast majority of cases.
HTTP file validation is allowed for DV certificates, so you can test a domain by connecting to it. Testing from a variety of locations around the world avoids the risk of MITM. If you issued a certificate for example.com and it's now being used by example.com, doesn't that prove the same thing?
But whether it's been exploited is, in a sense, irrelevant. The baseline requirements don't say "when you detect a mis-issued cert carry out more checks and decide for yourself whether to revoke" - they say the CA must revoke.
> HTTP file validation is allowed for DV certificates, so you can test a domain by connecting to it.
This was validation over DNS using a CNAME record. You don't need to keep the verification records around after the certificate is issued, so post-hoc analysis won't tell you much. But worse, how does the CA know if a RANDOM.example.com CNAME record was created by a customer vs the domain owner? The underscore is supposed to solve that, and they didn't use it.
You might not win - but if you just need a few extra days? The court paperwork will take time.
IANAL.
> DigiCert knew or should have known about its failure to properly complete the Security Certificates for Alegeus weeks ago.
'You should have known there were bugs in your code!'
> DigiCert is a voluntary member of the Certification Authority Browser Forum (CABF), which has bylaws stating that certificates with an issue in their domain validation must be revoked within 24 hours
"voluntary" is technically correct, I suppose. But let's assume they'd like to be a CA?
But then we go
> despite that DigiCert has arbitrarily chosen this less than 24-hour window
It wasn't arbitrary, it was a requirement, as the complaint itself pointed out…
> Alegeus has hundreds of such Clients. Alegeus is generally required by contract to give its clients much longer than 24 hours’ notice before executing such a change regarding certification.
Yet another example of that nobody ever thinks about cert rotation, ever.
> Consequently, due to the errors and delays of DigiCert alone, Alegeus cannot practically make all the required changes
Yeah, no, your error was not having a process to account for this in place.
The TRO also states,
> The threatened injury to Alegeus outweighs any injury DigiCert might suffer under the injunction.
Well, if DigiCert is removed as a CA because they've violated the CA/B BRs (at least, AIUI, they have) — that could be a pretty big injury. They're probably too big to fail, and "we tried but we had a TRO" is a pretty good excuse.
The TRO is scoped to only their certs, and only for 7 days (barring further orders), so it's at least pretty minimal? So I suppose DigiCert could revoke the rest.
So they filed this suit, got the TRO, will do the work while TRO is underway and at hearing, declare it all moot, pay the lawyers and move on.
Appealing, I imagine?
> would likely cause
> has the possibility of
Doesn't sound like it's "likely" in this case, so said process probably wouldn't work?
I'm wondering what the process is for showing that a TRO could potentially cause irreparable harm.
I don't think the legal challenges shown here are unique to this CA. A good question for the CABF is how a CA can demonstrate that they've reduced this sort of legal risk. It's a hard problem because there's CAs in so many different legal jurisdictions.
To be clear, I think a TRO is also a pretty good excuse for technical of the rules, and the most likely thing is that people bend to accommodate here.
> how their agreements should be re-written to reduce the risk of customers filing TRO
I actually am not sure a clear agreement matters here. It could be they already have that (I didn't see evidence in the complaint that convinced me one way or the other); if the case proceeds long enough, this I expect will be evident in the ruling: either the agreement was clear, and they should have expected a possible revocation since it is inherently part of the system they're interoperating with (… this would be my default stance; ignorance of certs doesn't get you infinite leeway here) or the agreement wasn't clear at it's a contract breach as alleged.
However, I also agree that, assuming they can rotate in 3 days like they said in the complaint, — but that is an assumption. I mean, IDK about your workplace, but I'm sure you've seen a deadline or ~~two~~ 100% slip? — then the case becomes moot and we'll never know, as there will be no ruling. That could very well happen, we'll have to see.
> I don't think the legal challenges shown here are unique to this CA. A good question for the CABF is how a CA can demonstrate that they've reduced this sort of legal risk. It's a hard problem because there's CAs in so many different legal jurisdictions.
I agree, though I think it is difficult to show that. Like I said, I'm not sure an ironclad agreement saves you from the TRO, here. The judge would still issue a TRO while deciding the agreement is, actually, ironclad, since the judge cannot start from the assumption that it is, and will err on the side of caution.
I think you are right, even if the agreement was perfectly clear, a TRO can be issued. Either the CA system lives with this risk or we only trust CAs that operate in countries without this risk (do any exist?)
Another approach is for the CA to revoke the cert first, then notify the customer. This avoids the TRO as the customer wouldn't know about the planned revocation until it was too late to act.
Does the CAs desire to help customers swiftly move to a new certificate outweigh the risk that a customer would sue to get more time (causing the CA to fail to revoke quickly enough)?