23,000 HTTPS certificates axed after CEO e-mails private keys | Hacker News Reader

23,000 HTTPS certificates axed after CEO e-mails private keys

arstechnica.com

276 pointsartsandsci54 comments

Oops Trustico server allows arbitrary RCE incl root shell: https://twitter.com/svblxyz/status/969220402768736258

did they really generate the certificate pair server-side?

pkiJS / WebCrypto² API are a keygen¹ replacement so that you can let the browser generate the certificate pair client-side. If the user isn't informed that key generation happens locally then maybe it looks as if the certificate is "downloaded". But with these methods when implemented correctly, certificates never goes to the server (and is saved locally).

I'm not a fan of the above approach even though Mozilla sells it as safe: e.g. internal direct access to the OS RNG (instead of doing it in JavaScript Math-random() which led to a lot of critic throughout history of browser based crypto) , since the browser is still a sandbox, and IMO unsuitable for private key storage (even just for a few seconds).

Would be interesting to know how trustico implemented this. It's somehow so crazy and negligent a thing to imagine. There is no explanation to why they didn't just implement it using standard software stacks for PKI (primekey has a jboss thing they could have used). Literally copy/pasting things from SO or some example code found in the manuals from WebCrypto would have been better than what they did. These guys have "trust" literally in their name.

EDIT: Tippfehler

¹ https://github.com/w3ctag/wiki/wiki/keygen

² https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_...

> These guys have "trust" literally in their name.

Generally speaking, this is more a red flag than an endorsement.

Stupid question: I thought it was accepted practice [1] to generate the private key server-side so long as you don't store after the user's downloaded it? (Not that Trustico followed that part either.)

[1] though not something I would endorse!

Why would it be? It's going to end up in lots of places it's not supposed to be. Better generate the private key where it's needed and never have it leave the system.

I said I don't endorse it and don't think it's a good idea but e.g. you can have AWS generate your private key (for SSH though not for a cert AFAIK) with the warning that they can't recover it if you lose it.

I just figured there might be some compromise to accommodate users who don't know how to generate their own.

A question I still have is what Trustico _hoped_ was going to happen here. Was there some arrangement where if the certs have to be revoked they get a refund?

Trustico also don't seem very clear on what exactly they're accusing Symantec of, the Ars article quotes them

"We believe Symantec to have operated our account in a manner whereby it had been compromised" and "We believe the orders placed via our Symantec account were at risk and were poorly managed"

In the relationship we're talking about here, a reseller to a CA, basically the only valuable information that can be compromised is payment instructions. All the important seeming cryptographic stuff? All of that is allowed to be public by design, the CSR, the Certificates themselves, any validation messages used - all public thanks to asymmetric cryptography.

Symantec got in trouble last year for inadequate (in fact basically non-existent) oversight of their RAs, particularly the Korean CrossCert. Unlike a reseller the RAs were trusted to do the domain validation themselves, so if CrossCert said "We checked and it's OK to issue a Thawte cert for www.example.com with this key" then Symantec just did it and didn't ask any questions (though they do seem to have checked they got paid...). Resellers aren't trusted with validation, they're just glorified sales people who just take a cut and help with price discrimination, what's there to "compromise" months after you cease the relationship?

I suspect the logic was something along the lines of "Symantec has acted in an untrustworthy manner (enough to get them distrusted by several major browser vendors) therefore we consider all our customer's certificates that were issued by them to be untrustworthy ("compromised") and want them revoked."

Still doesn't explain why they thought that revoking 50k of their customers certificates with less than 24-hours notice would be a good idea though. That really just seems like shooting themselves in the foot.

Can we take a step back even further? I can't figure out the logic that got this started:

"These Private Keys are stored in cold storage, for the purpose of revocation."

Like what? Why? Since when do you need a private key to revoke a certificate?

I feel like I understand certificates really well, but .. am I missing something?

Those are Trustico's customer's certificates. The only way Symantec could have fraudulently issues those was if Trustico told them to or lied about who their customers were.

There's no logic here to understand.

I think the previous thread about Trustico had it right: Digicert bought all Symantec's customers but Trustico wants their customers to stay with them instead and wants Digicert to look untrustworthy. If none of this went pubilc and Digicert revoked all their certificates, Trustico could step in to save the day.

This was very irresponsible from the CEO, but should not have been possible at all.

The private key should be generated where it's used, a certificate signing request created and sent to a certificate authority. After the CA makes sure the CSR is legit, they can sign the certificate without seeing the private key.

Copying the private key outside the server is an unnecessary risk. Giving it to a third party such as a CA or allowing them to generate it, moreso.

They said in the article they kept the private key to simplify the revokation process. Thus, very likely they also generated the private key.

It's a bad practice promoted by issuers that want to sell a "managed" or "value added" service corporate IT is so fond of. The lesson here is that there's a lot of IT people that don't grasp the basics of key management.

The road to security hell is paved with intentions to simplify.

That's the most succinct phrasing of the problem of security vs. users (and customers) complaining about what they perceive as the inconvenience and complexity that accompany securing digital assets.

I like to say (and it's not original) that security and convenience are two endpoints of one continuum.

You can't get more of one without moving away from the other.

There are a few rare near-freebies, but they always have a gotcha. And you'd better understand what they are before taking advantage of them.

The road to usability hell is paved with intentions to securify.

Why did they need customer private keys for revocation? That didn't make sense to me.

[edit: thanks, cjalmeida]

You need private key for the certificate management protocol. Maybe the had some arrangements for automatic revocation back when Symantec was running the CA.

Anyway, you are compromising security over convenience. A trade-off IT should never accept.

Never is a bit harsh, we're all doing it every day. Why are we still using passwords in 2018? They have been proved unsafe by now.

Granted, but in corporate IT, where stakes are higher, those concession should be minimal. Like, setting up a certificate with only a CSR is server setup 101.

Yes, everything is a matter of finding the right compromise (ofc I agree that sending 23000 private keys over email should never be considered)

Yes, they apparently provided a service that generates private keys because that's easier for the customer.

Having the private key on a server for a certificate authority in 2018 is an unnecessary risk. They should have had a hardware security module where the private key can not be extracted out.

When you can buy one relatively cheaply even from yubico, there is simply no excuse to not using one as a CA.

https://www.yubico.com/products/yubihsm/

All of their certs should be revoked and they should go out of business for technical incompetence plain and simple.

Forgive my ignorance, but how does a device capable of storing something like 127 keys help a CA whom should have been storing zero client keys? Do you mean for their own keys?

TL;DNR: I'm an idiot and misread the story thinking the executive sent the ca private key, not the client keys

This is even more egregious. Thanks for the cluebat hit to the face!

Speaking of things that shouldn't be possible: arbitrary code execution as root https://twitter.com/svblxyz/status/969220402768736258 Looks like the whole site is now offline.

One part that is missing is why did the CEO say those certs were already compromised before they he mailed them? He mailed them sort of as proof they are compromised?? Im confused. Any thoughts?

From the article:

"When Rowley asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates"

He's arguing that the various issues that lead to Symantec being gradually distrusted mean that they believed all their existing non-expired, non-revoked certificate were compromised (in the sense that they should not be trusted) and should be revoked.

Maybe it's just me, but a company who stores their customer's private keys, sends them around in a ZIP file via email and has a trivial code injection vulnerability on their site, with a process running as root, is probably not the one I want to make that call.

> Maybe it's just me, but a company who stores their customer's private keys, sends them around in a ZIP file via email and has a trivial code injection vulnerability on their site, with a process running as root, is probably not the one I want to make that call.

On the flip side, if a company with such little regard for security thinks there's a problem with the Symantec, you've got to wonder. They're basically saying "we're dangerously incompetent, however even _we_ can see that Symantec are broken" - and might actually have a point.

Symantec is already getting distrusted and the Web PKI community has decided on a plan that minimizes user impact while doing that. It's very doubtful that a company as incompetent as Trustico has the expertise to come up with a more well-informed decision or a better approach. Certainly attempting to get 50,000 certificates revoked immediately without doing any work to reissue those certificates prior to that is a terrible approach.

Considering the fact that they recently switched to a new CA partner, a more likely explanation is that this was a misguided attempt to somehow keep their existing business rather than let DigiCert pick them up somehow. Obviously it failed in a spectacular fashion.

But this is all nonsense. Symantec never had the private keys, so there was nothing at all unsafe about the certificates.

I dunno. If a company knows that it does all those things, I think that it is 100% correct in making the call that all of its active certificates should be treated as compromised.

They're not claiming the certificates were compromised because of their own practices, in fact they repeatedly emphasise that there was no issue on their end.

I'm not questioning that the certificates became compromised the moment they sent them via email, or really the moment they generated and stored them given the level of security competence they've shown thus far.

> They're not claiming the certificates were compromised because of their own practices

I didn't say their reasoning was correct, only the conclusion.

> As the holder of the root certificate used to sign the TLS certificates Trustico was reselling, Symantec was ultimately responsible for ensuring this requirement was being followed, although in fairness, there was probably no way for Symantec to detect a violation.

This is in some sense the scariest part in the whole mess. Root authorities might have responsibility, but the basic logic of cert resellers makes an error like this easy and undetectable.

> When Rowley asked for proof the certificates were compromised, the Trustico CEO emailed the private keys of 23,000 certificates

Voilà, compromised certs!

Couldn’t tell if that was supposed to be the idea. But amusing nonetheless.

I can't tell if the logic was "we shouldn't have these keys so the certs are blown" or "watch, I just compromised the certs!"

Either way, it's very much a case of "my neighborhood isn't safe - because I keep committing crimes there."

edit: Oh, reading the Mozilla page it's also "we allowed code injection on the cert generation page". That's even worse.

Also, that script was running as root

https://mobile.twitter.com/taviso/status/969230138956136448

Hah, seriously?

...I wonder whether the keys would have been more or less safe if, instead of all that, they just uploaded them unsecured to Pastebin. At least then they might have gone unnoticed, instead of being handed straight to possible bad actors.

"We haven't finished getting to the bottom of this and don't have the post-mortem yet but this is very bad and you need to revoke all of these ASAP"

"We're not revoking 50k certs without a damn good reason"

"Attached is your reason"

You gotta admit, it's a terribly compelling reason. And I do mean terribly.

God I hope my co-workers in the PKI group would say "hell no" if our CEO came and asked for the private keys. Show some damn backbone.

Though on 2nd thought, maybe having a society with a good social safety net and collective bargaining for workers would be a good foundation for a positive outcome in that situation.

If workers are afraid of losing their jobs because they'd lose health insurance and other vital benefits then they'd be more likely to cave.

Sorry but being from a former socialist country in the top 5 world ranking of living standards I can't help but make it a social discussion.

The issue wasn't "CEO gets what CEO wants," but that the private keys were even retrievable by someone else than their holder.

When these things come up they often have this sort of comment attached.

The comments always make it look like the conversation went "Hey can I get them private keys?" and our sysadmin hero either says "no" or our oppressed sysadmin afraid to lose their job says "sure, here you go boss!"

I guess where I'm going with this is - do people really do that? Either say yes or no? Where's the part where you explain the reason behind no? "If I do that it compromises xyz putting us at risk of exploits a,b,c and lawsuits 1,2,3" and then a more informed CEO either asks again or drops whatever they were trying to do

The way I see it, most undergrads would write malware, spyware, adware, and bloatware, for a low six figure salary. Likewise I think most developers outside of SV, SEA, NY, etc., would do the same for that amount.

I have little faith in software developers and administrators to show a backbone when their salary is on the line.

Looks like Trustico is a UK company.

>would say "hell no" if our CEO came and asked for the private keys

what if he asked for bigger furniture budget? we all know how that ends

The mozilla.dev.security.policy thread linked in the article is a much better read:

https://groups.google.com/forum/#!topic/mozilla.dev.security...

tl;dr: Trustico told Digicert that all 50k of their certificates should be revoked, without specifying a reason as to why they were compromised. 50k is a LOT of certificates, so Digicert asked for details. Trustico then sent Digicert the private keys, and Digicert said "Welp, guess now they're compromised".

> Finally -- the CSR/key generation form page incorporates JavaScript from at least 5 or 6 different companies (including ad servers), which would allow any of those third parties (intentionally or through compromise of their own) to capture generated keys. This is a reckless amount of exposure, to the point that even if the keys were generated completely inside the browser and never exposed to the server (which does not appear to be the case), I would consider them compromised at the time of generation.

Also, wow. That's extremely reckless.

Jesus. That means it's not just about "offering revocation" and storing them on the wrong side; even people who didn't use that option are completely compromised.

If you generated your own Private Key and used that to make one or more CSRs, you can show any fool the CSR and no harm will come to you, at least on the cryptographic side. Upload it to 4chan, print it out on a T-shirt, doesn't matter.

Any certificates produced are worthless to everybody except you, because you have the Private Key, and a _competent_ Certificate Authority (in this story that's Symantec or DigiCert, not Trustico) should only issue certificates for the names requested in the CSR, not any other names.

A customer who did things sensibly (their own Private Key, make a CSR etcetera) and used Trustico say in 2016, needn't be concerned that these development will end with their certificates unilaterally revoked on Trustico's say-so. They might not want to do any more business with Trustico, but there's no particular reason this incident should affect their feelings about DigiCert or Comodo or any of the Symantec brands (Verisign / Thawte / RapidSSL / etcetera) now controlled by DigiCert.

> Currently, we are only revoking the certificates if we received the private keys. There are additional certificates the reseller requested to have revoked, but DigiCert has decided to disregard that request until we receive proof of compromise or more information about the cause of this incident.

I feel like given Trustico's behavior and the fact they already emailed 23k private keys to Digicert, I would just be conservative and revoke them all. Why leave the other ~25k keys when Trustico has specifically stated they are compromised?

Seems to me that Digicert is tactically forgetting to say what they said to Trustico.

Additionally there's not an issue with Trustico having access to the private keys. It simply means you are trusting Trustico and if you accept that it's no different from trusting Google or Facebook or your bank with your private data.

So what's your response to this?

Reply on news.ycombinator.com