Symantec Backs Its CA
symantec.com
symantec.com
Problems since Oct 2015 and the action unexpected? see 1)
> We hope it was not calculated to create uncertainty and doubt within the Internet community about our SSL/TLS certificates.
Symantec took no ownership of the issue. Snarky underhanded remarks are not a professional way to address shortcomings in managing their product.
> For example, Google’s claim that we have mis-issued 30,000 SSL/TLS certificates is not true. In the event Google is referring to, 127 certificates – not 30,000 – were identified as mis-issued, and they resulted in no consumer harm.
Per Chrome's team an initial set of reportedly 127 certificates has expanded to include at least 30,000 certificates, issued over a period spanning several years see 2)
Summary: No ownership and no action plan conveyed in Symantec's 421 word message.
1) https://security.googleblog.com/2015/10/sustaining-digital-c...
2) https://groups.google.com/a/chromium.org/forum/#!msg/blink-d...
"23 test certificates had been issued without the domain owner’s knowledge covering five organizations, including Google"
Guess that explains part of why this particular CA incident has Google's full attention.
The snarky comments were probably not meant as snarky, they just happen to be the basis upon which one can seek damages from a 3rd party for damaging your business or costing you customers.
I would guess that Symantec's lawyers and O-level execs are in deep discussions whether to sue regardless of Google's follow-up actions or retraction.
Not saying a lawsuit would help them, but they are laying the groundwork for it here to keep their options open. And send a message to Google's legal team.
Will be very interesting to see where this goes. Really hope for everyone's sake it doesn't go to court because it will just end up being a tax on users in the end (both Google's and Symantec's).
[1] "브라우저 호환성 99.9%" http://www.crosscert.com/symantec/02_0_00.jsp
There are rules for inclusion in Google's cert store, and those rules were broken IIRC.
EDIT: I think that's really the crux of the issue. These 127 certs which Symantec claims are "harmless" are merely the ones which were stumbled across and obviously very "how is this even possible" wrong.
That's why the 30,000 is the "size of the risk". The big "Symantec" problem is that there's no good way to distinguish these 30,000 from the many more certificates issued by Symantec under different brands. For Google it's all-Symantec-or-nothing. So they're coming up with measures that apply to all-Symantec.
Though it doesn't mention the 30000 certs or 127 certs, it does say:
(long quote from Ryan Sleevi:)
In the current misissuance, my understanding is that Symantec asserts that the totality of the misissuance was related to RAs. Symantec's initial response to the set of questions posed by Google [5] indicated that " At this time we do not have evidence that warrants suspension of privileges granted to any other RA besides CrossCert" in the same message that provided the CP/CPS for other RAs besides CrossCert, and itself a follow-up to Symantec's initial response to the Mozilla community, [6], which acknowledged for the potential of audit issues in the statement "We are reviewing E&Y’s audit work, including E&Y’s detailed approach to ascertaining how CrossCert met the required control objectives.". This appears to be similar to the previous event, in that the proposed remediation was first a termination of relationship with specific individuals. However, in Symantec's most recently reply, [1], it seems that again, on the basis of browser questions from a simple cursory examination that such a statement was not consistent with the data - that is, that the full set of issues were not identified by Symantec in their initial investigation, and only upon prompting by Browsers with a specific deadline did Symantec later recognize the scope of the issues. In recognizing the scope, it was clear that the issues did not simply relate to the use of a particular RA or auditor, but also to the practices of RAs with respect to asserting things were correct when they were not.
It appears that, similar to the Testing Tool's failure to ensure that certificates were adhering to the fulsome standards of authentication, Symantec's newly established compliance team was failing to perform even a cursory review of the CP, CPS, and audit statements presented - despite Symantec having found it necessary in that introspective process themselves in response to [3], as noted above.
Symantec's also stated that, in response to the past misissuance, it deployed a compliance assessment tool, which functionally serves a role similar to a Validation Specialist. However, such compliance assessment was designed in a way that it could be bypassed or overridden without following appropriate policies.
The major CAs outsource to partner companies called Registration Authorities (RAs) to perform the task of verifying that people requesting certs are who they say they are --- this is especially important for markets where the company running the CA is has thin on-the-ground support. Such is the case with Symantec/Verisign and CrossCert, their partner RA in Korea.
The technical relationship between the RA and the CA probably varies a lot from firm to firm, but generally the RA has some ability to cause issuance of certificates through automated requests to the CA's infrastructure.
What Ryan and others discovered in repeated rounds of questioning to Symantec was that Symantec had been relying entirely on 3rd party WebTrust audits (these are technical and process audits for CAs conducted by Big 5 accounting firms) without doing any of its own technical due diligence. But the WebTrust audits Symantec's RA's had been doing were delivered by auditors nobody has any faith in, including (as it turns out) Symantec.
Further, Symantec was required to have technical and process controls for specific kinds of issuance requests from their RAs. And it did. But it turned out those controls were designed so that the RAs could override them on their own recognizance. Which is basically the same as running process controls on the honor system --- not OK in this environment.
It is likely Mozilla policy (or the BRs) will forbid letting the local RA do the domain validation. So, a future CrossCert could lie about whether their subscriber is really Foo Corp, but not about whether they control foo-corp.example
Oh, and it's not the Big Five any more, one of the Five collapsed in scandal because it happily signed off on Enron's obviously bogus accounts. So now we have a Big Four, until another one blows up. For those taking bets, the RA was audited by a local EY, whereas Symantec are audited by a KPMG.
Lol.
This is like being court-ordered to do community support and then bragging about all the volunteering you do. Symantec was forced by Google to do CT. See https://security.googleblog.com/2015/10/sustaining-digital-c... , specifically:
> Therefore we are firstly going to require that as of June 1st, 2016, all certificates issued by Symantec itself will be required to support Certificate Transparency.
(By the end of this year, all CAs will be forced to do CT, but Symantec was forced into this last year, because of the stupid shit they keep doing)
https://www.certificate-transparency.org/faq
Also
>Anyone can submit a certificate to a log, although most certificates will be submitted by certificate authorities and server operators.
Most logs don't talk to random civilians about this, but at the ct-policy meeting the one log which did speak about their policies said that they either cut a $$$ deal OR they accept a mutual logging arrangement, on the rationale that if you eat the cost of logging their stuff and they eat the cost of logging your stuff, that works out fine for everybody.
* Root CA practice allows delegating validation to 3rd parties
* However, the Root CA must accept all responsibility for any mis-validation the 3rd parties do. No throwing them under the bus
* Symantec delegates validation to 4 different companies to serve local markets
* Said companies fail to adequately validate domain ownership
* Symantec attempts to throw them under the bus
Further compounding the issue is that there is no way to separate the certificates that had more rigorous validation than the ones validated by these 4 companies
I also wonder what the exact consequences will be (Symantec post fails to explain this). I mean, which big sites will be hit? When? For how many users?
https://bug1334377.bmoattachments.org/attachment.cgi?id=8836...
What they will vigorously defend is disruption to their reputation and their bottom line. What benefit does a business get from Symantec that they do not get from Let's Encrypt? EV?
Let's encrypt is great but let's not pretend it's the right solution for all websites.
What's their stance on wildcard certs with SANs? Is it simply too risky / ripe for abuse?
GitHub, for instance, has three million users. Provisioning a cert for each github.io domain would take 29 years. Yes, you can contact LE and ask for a limit increase, but three million certs would still spam the public cert transparency logs, LE's internal records, etc. significantly. That is a cost. LE rounds down the cost of an individual cert, even 20 certs per week, to zero, but the cost of three million certs is very nonzero.
2. Wildcards work in mass-hosting situations where individual certs don't. Again, for my college computer club's server, putting a few thousand <VirtualHost> blocks in Apache's configuration makes it take forever to start. I tried this. It was a bad plan. The better plan is mass-hosting tools like mod_vhost_ldap or even RewriteRules based on the hostname, but they don't let you interact with SNI, because as far as Apache (and thus mod_ssl) see, it's just a single <VirtualHost>.
3. In order to issue a cert, you need to publicly respond for the name on the cert. If you're issuing public-CA certs for private-network websites (which is generally a best practice over running an enterprise CA), you need to briefly configure a DNS entry for the private name to respond to either the DNS challenge or the HTTP challenge. You can't just configure a DNS entry internally. So your internal deployment process now involves updating public DNS. For anyone who's not running publicly-facing infrastructure (i.e., most companies, whose public Internet presence is just a website), there may not even be an automated means of updating public DNS.
A wildcard cert, meanwhile, will just email the owner of the registered domain via e.g. WHOIS contact info, which requires no external configuration, and is a one-time setup process. Everything else is internal at that point.
I love Let's Encrypt, but it's ill-suited to use cases where wildcards are popular.
Administratively, we use mod_vhost_ldap, and it hasn't been complicated. If anything it's less complicated, because a number of larger websites (departments, courses, etc.) get somename.example.edu CNAMEs from the IT department, and those are virtual hosts too. Back when we were using only webhost.example.edu/~username URLs, the separate virtual hosts were a special case. Now they're just an entry in LDAP with a different DocumentRoot, that's all.
Not that I would really suggest this, it's a DOS waiting to happen.
https://letsencrypt.org/docs/rate-limits/
> Combined with the above limit, that means you can issue certificates containing up to 2,000 unique subdomains per week.
dokku config:set --no-restart myapp DOKKU_LETSENCRYPT_EMAIL=mail@example.com
dokku letsencrypt appWildcards certs have value in some settings.
If you have a service that serves to a fleet of STBs, TVs, or other embedded devices, they have a fixed set of RootCAs and getting the OEM to update those (and possibly through a ton of different resellers) is a nightmare.
While the embedded devices won't be impacted by this announcement (if the deprecation goes ahead), there might be some additional work for anyone that services web browsers AND embedded devices from the same endpoints.
Of course, there's not a good way for clients to indicate which root CAs they accept, and so they don't. If you get into the situation where newer devices don't support Symantec certs, and older devices don't support non-Symantec certs, it's time to figure out how to guess if you're talking to a newer or an older device and give appropriate certs. :(
And as mockable as that might sound, it comes in handy for businesses with special requirements.
(Sorry, I couldn't resist.)
EDIT: Apologies, didn't notice the other poster beat me to it.
You don't need a HTTP server to do authentication, it's also possible to use DNS and TLS SNI. There's certainly no need to spawn a server per domain, even for HTTP.
If your implementation is complex enough to require many certs, these problems are already solvable. Either fork over the money for a wildcard cert, or account for the extra work in your planning.
The setup we use is to have our own authoritative nameserver for a 'dyn' subdomain, and do DNS auth. We auth a subdomain with LE in less than 10 seconds.
PS: It's funny that Symantec's first google hit for "Encryption Everywhere" prompts for my browser's geolocation unsolicited. If your product is trust, maybe you should think a little bit more about how you present your product.
No earlier than this week has Chrome 57 been rolled out. I know that because a customer reported to me that one of my website was distrusted – I had forgotten a StartSSL certificate over there, and sure enough I didn't notice it because I was still on Chrome 56. Sounds like a timely coincidence: If I were Google I would exactly try to distrust a small CA before attacking a bigger monster.
[1]: https://groups.google.com/a/chromium.org/d/msg/blink-dev/eUA...
Their response speaks volumes.
"We are not the only one doing this, why us Google, why us?"
What a shitty excuse. Laughable. Big F to Symantec and wish you bankrupt :)
The job is on the side of CA companies to run audits at the extent of possible damages, just like in nuclear plants: You just can't do a mistake, and certainly not 127.
Easy, right? Just like a nuclear plant.
And honestly, banks should have the expertise and resources to change certs in a day or two, and if they don't that's on them.
With this history of mis-issued certs in mind, Symantec's CA business should be kicked to the ground, left bleeding and never be trusted again.
They aren't helping themselves any with this kind of post, imho.
Regarding TLDs coming under control of Govts, Its solved by independent mirror nameservers run by app devs (Firefox/Chrome/etc) and NGOs (EFF/etc).
If DNSSEC had been deployed 10 years ago, Muammar Ghaddafi would have been BIT.LY's CA.
Maybe you wanted to say that tree-structured DNS with TLDs owned by governments is bad, but this is a problem of DNS, not DNSSEC.
DANE reduces parties able to issue rouge certificate from TLD owner + all CAs to TLD owner only.
Which can be solved by having bit.ly key validated by .ly key + Firefox key + LetsEncrypt key.
DNSSEC may have other architectural/mplementational faults but this property is not one of them. I believe this way of certification is the way forword for internet scale web of trust.
Firstly: "With TLS properly configured, DNSSEC adds nothing."
DNSSEC presents you with a valid, secure chain to know that the meta-data (encoded as TLSA) about the end-point you intend to communicate with is valid.
i.e. Grab the TLSA record for a hostname, validate the hostname (and TLSA record) via DNSSEC and then if the TLS certificate matches the details in the TLSA record you have confirmation that things are valid.
Secondly: "Securing DNS lookups isn’t a high-priority".
Then why there is a whole IETF WG dedicated to addressing it: https://datatracker.ietf.org/wg/dprive/about/
Thirdly: "The real threat to CAs is “the global adversary”
i.e. organisations like the NSA, GCHQ collating everything, everywhere.
Any reasonable person could spend an afternoon - not being paid - providing a counterpoint to each item raised. None of them are (currently) relevant and the only one that is/was is actually being address in the IETF working group linked above.
I agree: DNSSEC does do what it does do. The problem is that what it does do isn't meaningful and doesn't add real security. At the same time, DANE/TLSA does create a system in which we replace certificate authorities with an organization run by the US Government.
Having a CT-like feature for DNSSEC would be nice, though.
Obviously, there's a reason nobody is issuing Google certificates, despite the insecure DNS that appears to be the only obstacle to that happening. Flesh out why, and try to be specific. The point isn't really about Google, but about the fact that the Web PKI is more complicated than the CA BRs suggest, and has safeguards at multiple levels.
I am not arguing that the CA PKI is a good system. It is not. It's a wretched system. It's only my contention that DNSSEC is worse.
Let's Encrypt absolutely would be happy to issue you a certificate for my web site if you can use DNS to convince them you control it.
Why not for GOOGLE.IO? Well Google is very famous, and CAs are obliged to have a "High risk" policy that identifies some names as "High risk" and requires human oversight for those names. Google is probably on every high risk list ever formulated. For Let's Encrypt since they have zero humans in their validation process, the "High risk" list is a blacklist, you just get an error "Policy forbids issuing" instead of your certificate.
But "High risk" doesn't scale. It protects Google, Microsoft, HSBC, Amazon, but it's not going to protect me or most other people.
CAA protects more broadly, but it doesn't protect you against somebody with control over DNS, because it needs DNS records to work, so they could just remove or alter the records. Also CAA is only just recently becoming mandatory, it can't save you if it's ignored by the CA.
I suspect the real issue is that the DNS protocol makes it extremely awkward for client machines to do full DNSSEC validation -- the normal approach is to see the AD bit and to blindly trust the upstream resolver, and this is obviously a complete non-starter for DANE.
Second, why is this a compelling argument? The first 2 TLSA usages are essentially what we already have: HPKP, bootstrapped from the DNS. HPKP's competition, TACK, worked the same way and didn't require DNSSEC at all. Further: all the problems people have with pinning (certificate suicide) remain in place with TLSA 0 and 1.
So basically this argument is: "that's a straw man because there's a way to deploy DANE where it doesn't accomplish anything and that's probably the only way DANE will ever be deployed so none of this matters". I am willing to stipulate that's the case.
Whoops. Usages 0 and 1, then.
> Second, why is this a compelling argument? The first 2 TLSA usages are essentially what we already have: HPKP, bootstrapped from the DNS.
TLSA usages 1 and 2 bootstrap off something that still works even if you've never visited the site before (or if you're in private browsing mode, etc). HPKP can't do that unless you preload.
> Further: all the problems people have with pinning (certificate suicide) remain in place with TLSA 0 and 1.
That's just not true. You can set a reasonable TTL and rescue yourself in a timely manner from a lost certificate without significantly impacting security. With HPKP, if you set a very short expiration, you lost almost all of your security.
What people overlook is that on the modern Internet, browsers themselves form an ad hoc certificate surveillance network. If you get a certificate issued for a site that pins, or create a CT discrepancy, there is a very good chance Google and Mozilla are going to discover the CA responsible. And, as you can see, even if you operate the largest, most important CA on the Internet, Google will, if that happens, fuck your shit up.
If you want one (of many) dispositive arguments against DNSSEC, just consider that Google and Apple and Mozilla are not in fact in a position to fuck shit up for .COM. You can't revoke or distrust a TLD: the use of those TLDs is encoded into the brains of hundreds of millions of users. Not so a CA!