WoSign Incidents Report Update [pdf] | Hacker News Reader

WoSign Incidents Report Update [pdf]

wosign.com

113 pointsxnyhps74 comments

Awesome use of load balancing for request retry across multiple backend servers:

> This was caused by the CMS (Certificate Management System), when it sent the signing request of the certificate to the signing server A, which had no response, then the CMS sent it to the other newly added signing server B. After a while the signing server A signed the certificate and sent to the CMS and also to the subscriber, then the subscriber installed the cert in its website and hat's why Censys recorded this certificate; in the meantime, the signing server B also signed this certificate some time later (in seconds) and sent it to the CMS, the CMS accepted it and rewrote it in the DB.

> This issue happened after adding another signing server on Jan 5th 2015, and found it on April 9th 2015. When had the two signing servers added a load balancer, but the configuration was not properly done because it didn't lock the request.

Mind you, that's a perfectly legit technical bug. Maybe they were using nginx for load balancing POST requests? :)

https://news.ycombinator.com/item?id=11217477

Yup, retrying PUT/POST is a bad idea unless you're really careful.

Cross linking the earlier WoSign scandal reports: https://news.ycombinator.com/item?id=12389573 https://news.ycombinator.com/item?id=12582534 https://news.ycombinator.com/item?id=12617659

So if I get it right, WoSign will cease to exist as a CA given the 1y probation proposed by Mozilla and the general distrust that will follow?

That was Mozilla's action plan, this is Wosign's response, now it is back in Mozilla's court.

> "Many customers in China find it important to use a domestic CA for purposes of security."

That's not how the CA system works. Your security is unaffected by what CA you choose; it is invariably the minimum of all trusted CAs.

It certainly is if you strip down the certs trusted.

Well, true. But that would mainly affect availability :)

Which is a good reason to trust fewer CAs, especially those from countries which have poor transparency.

I honestly don't think the location should be a factor. How many countries are there that don't have things like national security letters?

I'd rather have a system where the same criteria apply to all CAs independent of their jurisdiction, coupled with mechanisms that guarantee transparency (like Certificate Transparency) and stuff like key pinning.

I don't think the general public realizes how important this is.

I only recently found out that anyone that can reliably man in the middle your server can get a valid HTTPS cert from CA's like Let's Encrypt. That includes cloud providers, backbone providers and possibly the local government where the server is located. [edit: It also includes anyone that can temporarily modify or spoof your DNS records.]

True! As someone working on Let's Encrypt, I'd like just to point out that this possibility didn't start with Let's Encrypt, but is generally true of CAs that issue using domain validation (DV).

The CA/Browser Forum has endorsed a variety of ways of proving control over a domain; for each of them, if some CA uses it and you can spoof it, you might be able to get that CA to misissue a cert for that domain.

The desire to somehow tighten this up is in tension with the desire to make HTTPS ubiquitous, and an underlying problem is that we're on the Internet where there is no central identity mechanism or source of proof of identity assertions -- except perhaps the DNS root and all of its associated registration and delegation mechanisms.

Maybe someday we could slightly clean up DV by making the proof of domain control go through DNS domain registries, since they're the underlying source of authority or ground truth in the DNS system right now. We could imagine a protocol where a CA has to ask the registry whether a particular certificate request is really from an entity that the domain registrant has approved, and both the CA and registry could publicly log the question and answer. (Perhaps we'll also have certs for other kinds of naming systems as well.)

In the meantime, you can make a CAA record, use HPKP, like pfg said, and check the Certificate Transparency logs. None of those are perfect, but they're a lot better than what we had with DV five years ago.

> imagine a protocol where a CA has to ask the registry

Or maybe registries could directly act as CAs for their own TLDs (and no others). It would be an interesting discussion about how this would be better or worse than what we have now.

I imagine a lot of registries wouldn't want to pay for the infrastructure to do this, but maybe it should be regarded as a basic part of their role. Right now it would be very annoying because you could no longer get a cert for multiple names under different TLDs, at least if they were managed by different registries (so you couldn't get a single cert covering example.io, example.cc, and example.net, even if you controlled all three names). Maybe if SNI becomes more ubiquitous this limitation will seem less annoying in the future.

This idea is DANE with DNSSEC, and it is unlikely to be adopted. tqbf has a good blog post[0] about why it's probably not a good idea (disclosure: I agree).

[0] https://sockpuppet.org/blog/2015/01/15/against-dnssec/

And EasyDNS have a blog post[0] about why it actually is a good idea (disclosure: I agree).

[0] http://blog.easydns.org/2015/08/06/for-dnssec/

That article starts off by saying that DNSSEC slightly improves the security of DV certs, but not sufficiently.

What I was suggesting above was having domain registrars or registries use a new protocol (not DNSSEC) specifically to authenticate domain registrants to CAs for improving the security of DV certs -- and in order to replace existing DV mechanisms. My thought is that registries and registrars together possess the ground truth about domain ownership in the DNS system, and so it would be helpful if they had a way to securely communicate that directly to CAs. I didn't mean to suggest that DNSSEC should be that way.

I don't think this would be vulnerable to any of Thomas Ptacek's particular criticisms of DNSSEC, unless the first section implies that he wanted to completely eliminate DV certs or stop trusting them as a general matter. I didn't read it that way. However, we might take the article to imply that DV certs, even if they could be guaranteed to match up perfectly with the DNS ground truth at all times, may never be enough as the main or only means of authenticating some entities for some purposes, which is also fair.

The issue really isn't using DV (domain validation) or EV (extended validation), but the fact that the validation chain a particular client sees can be different from what everybody sees. The CA system could by augmented by various distributed trust and result consensus systems.

I wonder if this is a service Akamai/Amazon/Google/whoever-wlse-runs-lots-of-cdn-endpoints could sell to CAs? A kind of "reverse CDN request", where they can return (over TLS connections) a web page fetched from tens or hundreds of different edge servers all at once?

That'd at least mean an attacker would need to be MITMing the connection close to your server/loadbalancer instead of near to the CA?

Wouldn't help if someone was doing a Quantum Insert style attack from within your hosting infrastructure. (I wonder if there's pre-build AMIs to let you quickly spin up open source implementations of Qantum Insert on AWS already?)

'All at once' is being DDOS'd so you wouldn't want that.

In general, you are going to be MITM'd near your endpoint (ISP level DNS tricks, or on your WiFi) or near the destination endpoint, very rarely somewhere in the core (although that has happened with some BGP hacking). To prevent the client being mitmd you can ask something else on the internet what their cert resolution chain was.

If the server can be mitmd with a valid cert the only solution is for all clients to contribute to a distributed consensus -- appending the cert chain they see to a distributed log.

[deleted]

I'm not sure which part of my reply you think is not possible - if it's the "guaranteed transparency" bit, that is very much possible and is the end-goal for Certificate Transparency. There is no way to "bypass" this with laws.

Most CAs that operate today are located in countries where mechanisms exist that could very well be used to force a CA to hand over private keys and/or sign certificates, not to mention that some intelligence agencies (or other actors) might use not-so-legal means to achieve the same thing. So why bother trying to enforce some kind of "NSLs (&co.) are bad unless you're one of The Good Guys" rule rather than embracing a mechanism that guarantees that CAs will be caught when they engage in such behaviour (willingly or not)?

[deleted]

Are you implying nation states are going to prevent browser vendors from implementing mandatory Certificate Transparency? Why haven't they done that for HPKP, which would allow ISIS to prevent being MitM'd as well? What about all those E2E-crypto messengers out there?

Whoever is deleting or editing to erase, can you please stop? It is disrespectful to people who have replied.

Cross-nationality cert validation would be good. Ie a cert signed by US and DE and RU registrars, with that list in the cert, combined with advertising via a distributed ledger what had been signed by whom.

True. But I've heard this before by European providers avoiding US-based certificates. That doesn't make it any better, but this is a persistent myth, not limited to chinese customers.

Interesting aside that came out of this case is the issue of cross-signing intermediary and root certs and it not being disclosed.

The WoSign roots were cross-signed by[0] Comodo and StartCom (owned by WoSign, but we didn't know that), so even with WoSign roots being revoked, there would still be a verification path.

Nice to see that now there is an effort to disclose all of these[2][3], and[1]:

> Mozilla now requires the disclosue of all intermedidate certificates, including those cross-certificates.

[0] https://wiki.mozilla.org/CA:WoSign_Issues#Cross_Signing

[1] https://groups.google.com/d/msg/mozilla.dev.security.policy/...

[2] https://crt.sh/mozilla-disclosures

[3] https://secure.comodo.com/products/publiclyDisclosedSubCACer...

Is the CEO stepping down because fraudulent certificates were signed, or is he stepping down because they got caught?

The latter. Otherwise he would have stepped down weeks ago.

He is being "relieved of his duties as CEO" according to the report. Saying he's "stepping down" is IMHO like saying a fugitive "turned himself in" when police arrived at his house with an arrest warrant. (Although I don't think he is being accused of anything criminal here.) Most of the incidents appear to stem from technical errors and poorly implemented processes, but the whole list points to his inability to lead this kind of company effectively.

I'm always skeptical of whether replacing a few executives can actually fix the cultural problems that were fostered/ignored in a company over time. The new leadership has to overcome a lot of inertia, and that's assuming they're any better than the old. They're also still answering to the same investors and pressures.

When you have a business that effectively prints money, why do something this stupid?

I also wonder how much more effort they thought it was to write the code to backdate the certs (rather than use "now") v.s. code for upgrading to SHA-256.

It's not about a lack of support for SHA-256 on the CA side. But backdated SHA-1 certs are in demand because they allow other systems to keep working, systems like embedded/PoS computers that lack support for SHA-256, while simultaneously dodging errors on modern browsers that only allow SHA-1 certs if they were issued before a certain date.

Wosign was certainly capable of issuing SHA-256 certificates. But the customers needed SHA-1 certificates with a backdated issue timestamp. And Wosign was willing to fake the issue timestamp on new certificates, probably a lucrative market because no other reputable CA would be willing to do so.

> Wosign was certainly capable of issuing SHA-256 certificates. But the customers needed SHA-1 certificates with a backdated issue timestamp.

So they had a special side deal to get backdated certs? There's no way they were doing this for everybody or for "regular price" right?

Doesn't matter they violated the chain of trust, the fact that they took extra money to do this doesn't make it any more or less of an offense.

EDIT: Was reading out of date info :)

~~However like some one else pointed out out of the discussion in the group looks like it was a technical bug, which is still bad, but at least not maliciously bad.~~ Gah HN why you still no do markdown....

It was definitively done on purpose based on customer requests. Look at the linked PDF, section 9.2.

Oh nvm, looks like link changed or i have clicked a different link I was reading the google groups thingie.

Interestingly, several western sites also have side deals to get SHA1 certs backdated or otherwise whitelisted, including Google and CloudFlare.

This is a very interesting allegation. Do you have a link to any such backdated or whitelisted certificates? Where is the whitelist, and how does it work? Does all major SSL libraries, browsers, http clients (openssl, libressl, polarssl, gnutls, microsoft, apple, java, etc etc) implement the same whitelist?

A common approach is to get SHA-1 certificates signed by roots that have been pulled from various root programs, but are still (likely) present on outdated devices that do not support SHA-2 (like XP <= SP2). I think CloudFlare is using one of Comodo's removed roots[1] for this purpose.

Once a root certificate has been pulled from a root program, they stop being in scope for the root program policies and the Baseline Requirements (at least that's the common interpretation), which would prohibit SHA-1 issuance from those roots. There's been some discussion about changing this.

[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1208461

So the allegation that "western" sites somehow do similar shady deals falls flat on its face, because the Wosign situation is all about the actions of a trusted CA. There is no comparison between trusted CAs and other random/untrusted/removed CAs.

I'm sure plenty of "western" sites have attempted to get similar deals (FWIW, Tyro, one of the companies that got SHA-1 certificates from WoSign, is based in Australia), but I'm not aware of any other CA incidents like this one. Other CAs seem to either use pulled roots or go through the proper exception process for SHA-1 issuance in the CA/B Forum. No one else has been caught backdating SHA-1 certificates to my knowledge. (For now?)

(Symantec accidentally signed a SHA-1 certificate from a publicly-trusted root while preparing their SHA-1 issuance exception application, but that's not quite as bad, I guess.)

I did not claim they were doing anything shady, just that they got those certs through out-of-the-ordinary deals.

It would be a lot more justifiable (and far better for everyone involved) if, for example, the SHA1 certs through Comodo’s pulled root were available to everyone.

That's at best off topic. We're discussing the actions and trustworthyness of a currently trusted root CA here.

I dunno - there's an argument to be made (which I'm not sure which side I'd come down on" that having a trusted CA say "Oh _those_ certs? They're ones we issued from a no-longer-trusted-root of ours! We can do whatever we want with that!" is at best disingenuous and quite possibly just timeshifting abuse of trust. The "trust" lies in the organisation, not in the individual certs. Pulling them from the root program shouldn't absolve the organisation from continuing to treat them as security-critical and shouldn't absolve them from abiding by the rules for them.

I don't want to pick on Cloudflare or Comodo right now (see my other comment in here), but if someone were to propose a rule change that says a CA's trust obligations extend to all their old retired trusted roots as well as their current and future ones, I'd be inclined to agree unless I heard a very convincing argument against it (that had some strong protections in place to stop some of the obvious abuse possibiulities).

Thought experiment: what'd happen to a CA with root trust, if they negotiated to update a root cert, replaced their old one in all the trust stores, then sold their old private key to $badpeople?

There is a public exception process to handle SHA1 certificates, and for the rest, they get special certificates of old root certificates that are only on older devices.

Obviously, all these options are not available to anyone except a handful of large companies.

That's at best off topic again. We're discussing the actions and trustworthyness of a currently trusted root CA here.

Edit: The exception processes also do not involve fraudulently backdating anything. Which is kind of a big deal when you are in the trust business.

Google is not doing this.

CloudFlare made a deal with Comodo to issue (non-backdated) SHA-1 certificates from a "legacy" root that is mostly no longer trusted by modern clients.

Symantec and Entrust are also issuing SHA-1 certificates from "legacy" roots to large enterprises.

"CloudFlare made a deal with Comodo to issue (non-backdated) SHA-1 certificates from a "legacy" root that is mostly no longer trusted by modern clients."

That's quite a nice solution.

The upside seems to be there's very little additional risk if certs are issued that "modern clients" won't accept - while still allowing "legacy clients" to commuicate with the level of security/encryption they always have.

I guess the downside is - allowing it to contniue means nobody will _ever_ upgrade or turn off "legacy client" equipment - which is probably all riddled with huge numbers of other "known exploits"... If you make it possible to put off upgrading your WindowsXP (or equivalent early 2000's linux based) POS system or industrial/SCADA gear - it'll stay around randomly switching which botnet it's DDoSing for forever...

A shame. WoSign were super generous with their free certificate offering long before LetsEncrypt was a thing. They were a a handy alternative.

We should be thanking them for their free certs, and thanking them again now for giving us another example of how the PKI is a farce. The chances are there are a bunch other 'reputable' CAs out there playing these games.

But that's why browsers are increasingly requiring CAs to implement https://en.wikipedia.org/wiki/Certificate_Transparency

Interestingly - StartCom's free certs used to only have 1yr validity - one I got last week is good for 3 years. Cynical-me suspects they're doing that to improve their chances of sitting out a limited 1yr timeout. It'd be almost indistinguishable from a death penalty if all their existing client certs expired while they were unable to renew them - I wonder how long they've been signing certs for 3yrs instead of 1?

This story is a bit of a mess to make sense of coming in cold and reading a Google Groups summary. Here's my read, which may help clarify the story for others.

Mozilla have an excellent explanation document covering the backdated certs in detail here: https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG...

(Thanks to @xnyhps for the link in a reply to this comment.)

WoSign, described elsewhere as China's largest certificates authority, are a CA who have been found to have backdated SHA1 ceritificates to work around browser restrictions on SHA1 cert issueances. SHA1 is no longer considered secure. Resolution of that issue is discussed in new mozilla.dev.security.policy Usenet group peered by Google Groups: https://groups.google.com/forum/#!msg/mozilla.dev.security.p...

A better source for WoSign's update to the story is in the PDF posted to the newsgroup, here: https://www.wosign.com/report/WoSign_Incident_Report_Update_...

Titled "WoSign Incidents Report Update". Which is even less descriptive than the title presently given on this HN post, though perhaps what HN posting guidelines prefer. I'll let @dang wrestle his conscience on that one.

In that document are several issues listed, the one relevant to this HN post appears to be:

"9. Issue S: Backdated SHA-1 Certs (January 2016)

"WoSign has issued certificates after January 1st 2016 but backdated the notBefore date to be in December 2015. This has the effect of avoiding the blocks in browsers regarding SHA-1 certs issued after January 1st 2016. The number of certs affected is probably 67, but may be a few more or less."

Following down from there, several corporate restructuring steps are mentioned, including:

360’s Corporate Development team has been notified to execute the process to legally separate Wosign and Startcom and to begin executing personnel reassignments. StartCom’s chairman will be Xiaosheng Tan (Chief Security Officer of Qihoo 360). StartCom’s CEO will be Inigo Barreira (formerly GM of StartCom Europe). Richard Wang will be relieved of his duties as CEO of WoSign.

There is background on the story from:

"WoSign Mis-Issued SHA-1 SSL Certificates [Updated]" (August 24, 2016) https://www.thesslstore.com/blog/wosign-mis-issued-sha-1-ssl...

"Mozilla Ready to Ban WoSign Certificates for One Year After Shady Behavior" (September 26, 2016)

The second article details Mozilla's issues with WoSign, including purchase of an Israeli CA, StartCom http://news.softpedia.com/news/mozilla-ready-to-ban-wosign-c...

I'm not claiming anything other than a 15 minute familiarity with the situation here. I may have heard earlier rumblings but really haven't followed this at all and wasn't consciously aware of particulars.

The best place to start with this story would be this very well writen document by Mozilla: https://docs.google.com/document/d/1C6BlmbeQfn4a9zydVi2UvjBG.... This is WoSign's first public response after that document was published.

Thanks, I've updated my comment with that.

Mirror of the PDF version of the incident report if they take down the original or it goes down: http://clicky.strapr.com/3g1s1y1k0Y2J

Direct link to PDF report: https://www.wosign.com/report/WoSign_Incident_Report_Update_...

So much money they are making and they can't hire anybody who can write English properly or write a PDF that is not composed of a handful of font faces and sizes.

I also find it funny they have fired the CEO (the PDF does not say he stepped down voluntarily) but he's the one sending that link to the mailing list. I call bs.

> So much money they are making and they can't hire anybody who can write English properly

If you're going to make fun of someone's English then you should probably write better English yourself. At the end of the day English isn't their first language and they'll probably very rarely have to use English. I think some slack should be given on this front.

I don't speak English natively, and any corrections are welcome. On the other hand I don't run a multi-million business based on writing comments on Hacker News so currently hiring a PR person to proofread these comments is not within the boundaries of my budget.

WoSign runs a business that is basically based on trust, and Nigerian scammers have sent me PDFs that looked far more convincing and trustable than that one WoSign posted.

tarancato isn't making any money from these comments, so it seems reasonable that they are not using some form of proofreader.

I am not bothered by their English or using a proof reader. I just think it's hypocritical to throw stones at people for not having good English when they don't operate in an English speaking market when yours isn't perfect either.

Also do you honestly think that was written by the business team, do you think the business team would have been able to properly edit a technical document in another language?

He's not throwing stones at people, he's throwing them at the company, WoSign. FWIW, I agree. They should have gotten (i.e. paid) someone to write a coherent document for this important communication.

> FWIW, I agree. They should have gotten (i.e. paid) someone to write a coherent document for this important communication.

Seriously you'll be hard press to find a paid translator that can translate technical documents.

Also the document is actually reasonable coherent. Picking on the language level of the document over the technical content of the document seems kind petty.

The CA Browser Forum puts out a lot of documents in english - often full of both technical terms and complex multi-layered clauses and subclauses.

You'd hope they'd have a good technical translator already on their payroll.

I'm capable of reading and writing a number of languages, but I'm not proficient at it. This also means that I probably have the writing level of a lower grade school child in those languages, and if I was to try and speak, I'd sound like I was mentally disabled since my vocabulary and general knowledge of the language is lacking.

Japanese is one. I've many times read comments written in Japanese in the Ruby language source code and understood their meaning, but there is absolutely zero ability for me to visit that country and be able to interact with anyone.

The author of this incident report is leagues more capable than I am in writing a report in a foreign language. They didn't insult my mother, or my dog, or call me names by accident. I don't see the justification in insulting their English.

WoSign owns Israel-based CA StartCom (from the shady event where WoSign bought Startcom, didn't disclose it, and started issuing invalid certificates through StartCom). Both operate in multiple English language markets.

> and they'll probably very rarely have to use English.

The CA/Browser Forum Baseline Requirements and the mailing list discussions all happen in English. The major root programs all use English as their language of communication, to the best of my knowledge. The SSL/TLS and X.509 specifications are in English. The major browsers and TLS stacks have technical documentation, code comments, and code review in English. mozilla.dev.security.policy is in English. The international language of scientific research (including cryptographic research) is in English. The CVE program is in English. All of these are reasons not only to have people who are fluent in technical English, but to make hiring such people a priority.

In particular, the inability to communicate precise, subtle technical concepts was very relevant to some of the problems here, like "You may not issue SHA-1 certs after this date" meaning the actual, calendar date of issuance and not the recorded validity period in the certificate. (There was also an element of malice, but it could have gotten sorted faster if communication were easier.) Even if all the others were in some other language, the CA/Browser Forum alone would be enough to make anyone who wants to be a CA (or an HTTPS client implementor) must make fluency in technical English a priority.

It is, of course, highly unfortunate that people from non-Anglophone countries are at a disadvantage here. In an ideal world we wouldn't have that disadvantage. But there doesn't seem to be a way around picking a language to be the scientific lingua franca.

There are a lot of reasons to criticize Wosign. Lack of English skills isn't one of them.

I didn't realize this until I first visited China, but there are parts of the world where it's really hard to find people fluent in English. If you require people to be fluent in English in order to participate in running the Internet you're locking a large number of world regions out.

Learn. You only need a handful of people to read the language in which 99% of all scientific publications are written in.

Reading knowledge of a language can be gotten fairly quickly.

This wasn't about reading knowledge, it was about writing knowledge. The original poster was complaining about grammar errors in their own publications. And even learning reading knowledge of a language that is totally different from your own isn't so easy. I spent quite some time trying to learn Mandarin.

Also out of curiosity: How many languages that are not related to your native language can you fluently read?

There are millions and millions of people in Canada and the US that are fluent in both English and Mandarin. Hire one of them.

I wouldn't hold it against a Chinese company for not having a native English speaker. But I would hold it against them for not at least having a native speaker on retainer to proofread a document that they're about to send out to an international community. It just looks unprofessional.

And that's all it is, really. Stuff that just screams, "Did no one proofread this thing before sending it out?" Probably not. Because whoever wrote it was probably the most fluent person at the company.

---

FWIW, I have native-English speaking programmer friends who ask me to proofread their stuff. I don't mind. One is an amazing programmer, but his writing ability is somewhere around 9th-grade proficiency. The fact remains that being able to write correctly is considered a "bare minimum" quality in the professional world, and it doesn't matter if you wrote your own virtual machine in Assembly in your spare time--if you sound like a college dropout in your resume, they'll skip over you.

It is similarly difficult to inspire confidence in a CA when their "transparency report" sounds unprofessionally cobbled together.

So I am wondering how this mess got started

Is it just incredible incompetence on WoSign's part or is there a stronger reason why China's largest CA is trying to keep issuing weak certificates? Is is tinfoil-hattery to assume that the Chinese government is not ready for SSL to start getting unreadable again?

Compatibility with legacy software that only understands the older certs is my guess.

> The charged 42 backdated certificates are an intentional activity that we try to help the desperate customers since there are more than 3M users still using Windows XP sp2 in China. We like to make things simple that don’t realize how serious this solution was.

I think this is all there really is to it. Some customers wanted to have backdated SHA1 certificates so they could continue supporting platforms that do not accept SHA256 certificates. WoSign decided to oblige them.

Reply on news.ycombinator.com