Certificate Revocation Issue | Hacker News Reader

Certificate Revocation Issue

downloads.globalsign.com

178 pointsdirectionless81 comments

The handling of this has been quite terrible. The article is the first "good" communication about an event that started 7 hours prior. And according to the communication will take another 4 days to solve completely.

Before this, the Technical Solutions Director tweeted solutions that did not work for end users, but highlighted a typical IT centric approach to problem resolution ("Works, What's the problem?") [1]

For anyone not already aware, check out Let's Encrypt. I am evaluating it for about 200 domains now in earnest after having it on my horizon for some time. At least to have it ready as a fallback. [2]

Getting 200 EV certificates in a hurry from a different CA has been costly this morning.

[1] - https://twitter.com/vanbroup/status/786548172864626690 [2] - https://letsencrypt.org/

Let's Encrypt doesn't offer EV, and AFAIK, they don't plan to.

That is a good point. I guess the move to Let's Encrypt is a change in mindset completely. Away from Marketing snake-oil to a simple solution to a technical issue.

EV certificates do not add additional security as far as I am aware, it's simply a measure of extended validation of the customer, which in my experience is only fluff.

So going with Let's Encrypt may as well do the full cycle and sell this free solution to non-technical people who believe EV provides additional value.

Hard sell... I know.

> EV certificates do not add additional security as far as I am aware, it's simply a measure of extended validation of the customer,

Correct.

> ...which in my experience is only fluff.

Depends on what you want to accomplish. If you just care about getting to SOME website with a secure connection than a DV certificate is all you need.

But take my employer: Capital One. We're a bank. When people go to https://www.capitalone.com it is important that they get our website. ALSO, though, it is important that when people go to https://www.capitalοne.com that they do NOT get some malicious website. (Can't see the difference? What glyph did I use for the letter "o"?) This is just one example of how an EV certificates can be useful for certain purposes, although it may not apply to your business.

CertSimple only does EV (and we happily recommend Lets Encrypt, CloudFlare, Heroku and other free sources if you only need a DV cert).

We're 18 months old and the main reason we exist is to make the identity verification EV adds not a pain in the arse.

Here's a list of ways we do EV differently from other providers: https://certsimple.com/about

I've been on HN for nearly as long as pg has so feel free to give me a shout here if you have any questions. It's past midnight in the UK right now so I'll answer them tomorrow morning. If you prefer email: mike@certsimple.

Was checking out your site, and the form to start verification doesn't appear when in portrait mode on mobile?

Landscape seems to work fine though.

Thanks bakavic. Our UI is currently built for desktop since most of our customers are coming from desktops - there's some data entry and we use things like webcrypto and HTML5 clipboard that may not be available on mobile platforms. We're going to improve the mobile experience in future but have other higher priority items (like better DBA support!) on our roadmap first.

Tried it out for the heck of it. Unfortunately Dun & Bradstreet has old address information for my company and their account creation system is broken (error 'CEPDEV101' when creating an account). So the "3 hour" validation will take a lot longer, through no fault of CertSimple's.

Hi Steven! Check the 'Next steps' email which has specific instructions for your company - we don't rely on Dun & Bradstreet to update.

In case you can't access their website

Dear Valued GlobalSign Customer,

As most of you are aware, we are experiencing an internal process issue (details below) that is impacting your business. While we have identified the root-cause, we deeply apologize for the problems this is causing you and wanted to ensure you that we are actively resolving the issue.

GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms. As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for 1 week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.

GlobalSign has since removed the cross-certificate from the OCSP database and cleared all caches. However, the global nature of CDNs and effectiveness of caching continued to push some of those responses out as far as end users. End users cannot always easily clear their caches, either through lack of knowledge or lack of permission. New users (visitors) are not affected as they will now receive good responses.

The problem will correct itself in 4 days as the cached responses expire, which we know is not ideal. However, in the meantime, GlobalSign will be providing an alternative issuing CA for customers to use instead, issued by a different root which was not affected by the cross that was revoked, but offering the same ubiquity and does not require to reissue the certificate itself.

We are currently working on the detailed instructions to help you resolve the issue and will communicate those instruction to you shortly.

Thank you for your patience.

Lila Kee Chief Product Officer GMO GlobalSign

US +1 603-570-7060 | UK +44 1622 766 766 | EU +32 16 89 1900 www.globalsign.com/en

One thing that shocks me about this is the browser responses. Opera would not allow me to visit the sites affected, citing a revoked cert and possible attack. While I appreciate the warning, it was odd to see the site fully disabled.

Safari on the other hand silently failed the https auth and served the page regardless. Concerning behavior for a revoked cert.

That might depend on the specific Safari version. I am using Safari 10 on macOS Sierra, and it would not allow me to access any site using one of the affected certs: instead it would just serve an error page.

I think that was more because of the site, not the certificate - the grey failure page would appear for sites that use HSTS[0]. If the site didn't use HSTS or if you visited it for the first time, you'd get an ordinary certificate error alert.

[0] https://en.wikipedia.org/wiki/HSTS

The cert wasn't revoked though, just the cross signed intermediate.

Possibly your Safari had already cached an 'ok' result for the intermediate?

The os already has an ocsp cache.

It baffles me why do all browsers pretend that https with an untrusted certificate is worse than plain http.

I think because with plain http there should be no illusion of security.

Do you mean "we expect people not to have an illusion of security" or "we would want people not to have an illusion of security"?

I posit that this actually creates an illusion that http is better than "insecure" https.

This worked for me on OS X:

  sqlite3 ~/Library/Keychains/*/ocspcache.sqlite3 'DELETE FROM ocsp WHERE hex(serialNum) IN ("040000000001444EF03E20", "040000000001444EF04247");'

What a pain in the behind :/

This did not work for me on macOS Sierra

I am on macOS Sierra...

That didn't work for me, but this did:

  sudo rm /var/db/crls/*cache.db

From: https://support.globalsign.com/customer/portal/articles/1353...

That removes all caches for OCSP. That seems a little heavy handed IMHO.

but if it works.

Interesting, for me it was the opposite. The rm command did not work, but the sqlite3 command did.

Not helping us that are running >30 sites using globalsign, but I have finished buying RapidSSL for all of them.

Nope :-(

Ouch that sucks.

Thank you! I tried the rm that GlobalSign recommended to no effect, but your sqlite3 command immediately fixed the problem (on macOS Sierra).

How does their SSL warranty play into this? Will they have to pay $1.250.000 for each OrganizationSSL certificate? https://www.globalsign.com/repository/globalsign-warranty-po...

Interesting - 3.3 seems to be applicable here:

"3.3 Intentional or accidental errors This Warranty Policy warrants against intentional or accidental errors introduced into a Certificate by any personnel of a GlobalSign RA or a GlobalSign LRA due to failure to use reasonable care as required under the CPS."

Not a lawyer, but will push this with our legal team for sure.

This could destroy GlobalSign, so there must be a clause to exempt them from liability ... ah, here it is:

"8.1 Total amount for warranty exhausted When the maximum limit allocated for warranty payments is exhausted, GlobalSign shall have no further obligation to refund any Beneficiary, unless required otherwise by applicable law."

So basically if you ask them for $1.250.000 in potential lost sales and damage to your reputation because your website is unavailable and appears to be compromised, you'll get it? Nice.

Not if they allocated $0 for warranty payments :-)

Unless required otherwise by applicable law. :-)

"introduced into a Certificate"

They didn't introduce any errors _into_ a certificate. They affected the certificate chain, but not the certificate directly. I'm certain this is how their own lawyers view this.

If THIS isn't covered by warranty, what would it take to invoke this warranty?

Btw I've always thought these SSL insurance warranties sounded like a scam.

It seems likely we'll find out if they are, at least for GlobalSign's.

This is going to get hilariously legal. A certificate from GlobalSign or anybody else is the product of the chain, if the chain isn't there and the cert isn't valid it's no more valid than a self-signed one.

Tell me again, anybody, why we trust these people?

"When the maximum limit allocated for warranty payments is exhausted..."

I'm gonna go out on a limb and guess they allocated $0.

I bought new certificates, for a new set of domains, through AlphaSSL today. One hour later customers starts calling, complaining about revoked certificates. Initially I assumed they had screwed up somehow and revoked our old certs, but after reports saying that it worked with some browsers and failed on others I started googling for recent related issues, and found out about GlobalSign.

Man, do they suck at communication. We're now 14 hours into the incident. 6-7 hours ago they posted a trouble shooting guid, promising new intermediate certificates for AlphaSSL and I've just been informed by their support that it'll be another hour before they're ready.

It's now 02:00 in dk, so I can expect the new certs at 3 and be done by 4.

Fun night.

Thanks GlobalSign.

P.S. Also thanks to the guy who made their marketing department stop tweeting iot crap while this is going on. That pissed me off.

This broke bootstrapcdn.com, bootstrap's official CDN. So the effects are extremely widespread even for non-globalsign clients.

Since this may take days to resolve completely, here's a temporary workaround for Chrome users -- launch your browser with this flag:

    chrome --ignore-certificate-errors

You'll still know when sites have bad certificates due to the red line drawn through the https:// portion of the URL. But you will be able to access these sites. But be sure to stop using this flag as soon as you can. It could leak secure cookies to a MitM with a fake cert. Very slim odds of that, but still undesirable. Yet Chrome left us with no other option here.

I must say though, I'm increasingly frustrated by software vendors trying to strip away control over our own machines. There is no option at all from the standard error message, even under advanced, to indicate that you know about this problem and wish to proceed. And I'm sure it's only a matter of time before they remove this command-line option as well.

I get that novices probably need some protection, but I really wish there were a way to say that, "yes, I really do know what I'm doing, please stop treating me like a toddler." So instead, I'm forced to use a much less safe, hidden command-line option or be locked out of various sites for four whole days.

> I must say though, I'm increasingly frustrated by software vendors trying to strip away control over our own machines.

I get the sentiment, but if you look at how common malware is on Windows versus (walled-garden) platforms like iOS, it becomes pretty clear that it's the most effective way to keep users safe. Inconveniencing tech-savvy folk is a small price to pay for that, IMO.

> I get that novices probably need some protection, but I really wish there were a way to say that, "yes, I really do know what I'm doing, please stop treating me like a toddler."

Typing "badidea" when faced with the interstitial should do the trick.

But that's just it, I'd have to download the source to Chromium, find where this error is thrown, and add a UI element to allow me to bypass the error. That's ... a whole lot more than just inconveniencing me. Flipping a switch in chrome://flags would be an acceptable inconvenience.

This command-line switch works, but it requires accepting a larger risk with respect to secure cookies and MitM attacks.

That's why "badidea" exists.

... wow, I thought you were saying that would be a good way of implementing it. Didn't realize that actually worked o_O

Thanks, that's certainly superior to the command-line flag! A touch patronizing, but I guess it could be worse :P

What reuptable certificate authorities are left besides LetsEncrypt?

Symantec is expensive but they seem to have their act together.

Not particularly.

https://security.googleblog.com/2015/09/improved-digital-cer...

https://security.googleblog.com/2015/10/sustaining-digital-c...

Also:

https://groups.google.com/forum/#!topic/mozilla.dev.security...

https://cabforum.org/pipermail/public/2016-January/006519.ht...

https://groups.google.com/d/msg/mozilla.dev.security.policy/...

https://security.googleblog.com/2015/12/proactive-measures-i...

https://sslmate.com/blog/post/ct_redaction_in_chrome_53

https://www.agwa.name/blog/post/domain_validation_vulnerabil...

And all of this is just from the last year.

Symantec is one of the worst CAs there is. Avoid them like the plague.

Now i have ten tabs open catching up on tweets from you and @sleevi_

https://bugzilla.mozilla.org/show_bug.cgi?id=1261919 is fun

Updates https://news.ycombinator.com/item?id=12712279 https://downloads.globalsign.com/acton/attachment/2674/f-06d...

>some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.

So it's not their failure but browsers'? Which ones then and what versions?

At least Safari and IE, it seems.

Beta chrome as well. Stable chrome shows revocation if you click into details; beta marks the cert as revoked.

Stable Chrome for me was definitely broken. And in fact, trying to view the details on the revoked certificate caused Chrome to beachball.

Weird! Completely different than my experience.

That's what I'm wondering, too - is this actually true, or is it just a deflection?

I didn't even know that this was what was going on until I saw this on HN and I've been experiencing it all day today.

I thought it was Opera's fault: sites would load fine on Firefox or Chrome, but not on my primary browser.

This killed one of our main webapps, but since the site was hosted on AWS, I provisioned an ACM certificate (free) in about 5 minutes and manually applied it to the ELB listener. Couldn't have been easier.

Today's weirdness and communication around it has made me trust Globalsign a lot less now.

Affecting SoundCloud, though I'm not seeing any ssl issues on my end.

http://status.soundcloud.com/day/2016/10/13

Revoking keys (and the necessary checking that it requires) will never work IMHO. The only way to solve this problem is short key signature lifetimes, automated signatures and, if compromised, just no re-signature.

Let's Encrypt is one way, although the lifetime with 3 months is a bit too long. One month or even less would be better. Additional verification and checks via DANE/DNSSEC help to shorten the impact of a compromised key. Constant checking for revocations do not. Again: IMHO.

Root CAs can't have 3 month validity. Any computer left offline for 3 months, or even 1 day (if coming up close to expiration) wouldn't be able to bootstrap an updated trust store.

If you're going to do that properly, why not more like 1 hour?

That'd be pretty rough if Let's Encrypt had an outage.

It'd also be incredibly hard on Certificate Transparency servers.

A 1 hour server certificate might work if all clients had the proper time set and all libraries were comparing the notAfter to the UTC current time.

Sadly, lots of phones and computers have incorrect time settings, off by one hour to work around incorrect DST settings is pretty common, off by several days isn't super rare either.

I have run into some SSL client libraries that check notBefore against the current time, in the current timezone, and therefore those in the western hemisphere would fail to work for several hours after a certificate was issued. I imagine they're using the same for notAfter, so those in the eastern hemisphere would stop working early.

My rule of thumb is a certificate that is less than 30 days old or has less than 30 days of validity left is going to cause problems for a subset of your users, and shouldn't be used unless you have a good reason (old certificate compromised would be a good reason).

> The only way to solve this problem is short key signature lifetimes, automated signatures and, if compromised, just no re-signature.

I wonder if Let's Encrypt is thinking ahead to a federated future where instead of issuing leaf certs, they also issue intermediate certs to domain owners, who can use those to mint very short-term leaf certificates (like 1-4h) in an automated fashion.

That would strictly require that all TLS clients correctly handled intermediate certs with a scope limitated to a few hardcoded second level domains.

Right now there's barely support for limiting a root CA to a single TLD...

This issue affected me as soon as I upgraded my chrome to V54 yesterday. It broke all my CDN hosted files which were using the AlphaSSL Wildcard certificate. We were experiencing low traffic and realized this may have been the issue. Got into Chat support with ssl2buy who provided me with a Comodo Wildcard certificate. It was a pain to recreate and install the certs everywhere. But we didn't want to lose any more traffic.

https://support.globalsign.com/customer/portal/articles/2599...

They are in the process of fixing certificates. I know of many that are now ok. Too bad I already bought new ones, won't be going back,.

Once we understood that this was caused by the Chrome update we contacted them and we got a free Komodo certificate from AlphaSSL.

it looks like this is impacting sites hosted on google cloud using the load-balancer (you upload your cert, and I'm using a globalsign cert). I am getting 502 errors via mobile but via desktop it's fine.

anyone else use globalsign via google load balancer who can confirm?

downvoted eh? tough crowd.

FYI the google load balancer issue seems to be real, but not related.

https://status.cloud.google.com/incident/compute/16020

I've been wanting to switch all our certificates to Let's Encrypt for almost one year. But, there was always something else which needed my attention more urgently.

So, I guess, thank you Global Sign for forcing me to finally make the switch!

Luckily I don't have many tenants yet: I replaced my wildcard AlphaSSL with a couple of Lets Encrypt certs to fill the 4 day gap. Four days of inaccessibility is just not acceptable.

This is a real mess.

Did the issue directly affect your AlphaSSL site? So far mine's continued to work. May have dodged a bullet by using OCSP stapling, though.

Yes it did. I have had multiple emails from clients who could not access the platform.

My company uses Google Firebase and Fastly CDN, and we're affected by this issue through both hosts.

I assume this explains why so many people are experiencing random SSL issues today.

Easy, they just need to revoke the revocation.

What do you mean, X.509 doesn't support that? :P

Reply on news.ycombinator.com