Symantec CA Response to Google Proposal and Community Feedback
symantec.com
symantec.com
Aka "updated".
The entire post is basically "ok how about we be really good from now on and suffer no consequences, cause it'd be really shitty for us if we had to be penalised".
They also posture a lot talking about how big their customers are, almost boasting about how inflexible and slow these big companies are, as if that's somehow Google's or general Internet users' problem.
The current Symantec SHA1 intermediate CA cert is valid between Feb 2010 and Feb 2020, so it seems like there was probably a cert change ~7 years ago too. The horror!
How hard is it to run some version of Chrome or Firefox that is within the last 5 major releases?
I don't care how big your organisation is, it's not that hard.
Yet they want to off-load the costs to the rest of the world to keep sites compatible with their IE8.
This is happening because Symantec was grossly negligent. They're going to be held to account for that.
All those giant companies should be firing up their lawyers and starting emergency procedures to change to a different provider. Anyone still using Symantec at this point is absolutely playing with fire.
The manufacturer made customizations to Android for their phone. Android isn't a cleanly-separated stack where the manufacturer modifies only a certain part of the stack for their phone but Google could go and update another part without risk of breaking anything (is any software project?)
Then, the manufacturers sell the phones to carriers, who usually request extra customizations to (at best) add their logo, or replace default apps with the carrier's own version. This means that any updates to Android likely require updates to this customization.
But even if that customization was free (from a development perspective), Carriers still have maintenance/service contracts with the manufacturers such that any manufacturer's updates must go through the carrier's "quality control" procedure. You wouldn't want an update to make the, say, LTE radio misbehave and break the carrier's network would you? Oh no... (and you certainly wouldn't want to "forget" to replace the good default app with the Carrier's crappy one)
So there you go, a quick glimpse into the complex economic/contractual obligations that delays or blocks Android updates. FWIW, Google has been working both technically and contractually to untangle this mess, and is one reason for the existence of their "own" line of Nexus and now Pixel phones.
The SoC / OEMs also have to sign off because they are the ones with access to the driver source code which needs to be updated and recompiled against a new release.
If Google plays hardball, these parties can always threaten Google with "We're leaving the OHA; we will be going with Amazon's Android fork and bundle Kindle and Prime apps." I'm sure Bezos will throw in Amazon Maps for free.
Edit: Yes, really. https://arstechnica.com/gadgets/2014/02/new-android-oem-lice...
The CA system is composed of alow-moving companies like Symantec dependent on, or more specifically, beholden to fast-moving companies like Google.
In the mobile space, the situation is inverted. Android can move quickly, but they have little to no leverage over the slow-moving carriers.
1) The point of the first part of their post is not "we're too big", it's "a move like this would disrupt the lives of an awful lot of people." Even more so, consider what would happen if Google were to update Chrome to not accept these certs. For internal applications, the IT departments of all these companies—which likely total a few hundreds of thousands of users in total—would simply mandate that Chrome is no longer an acceptable browser. For external applications, companies would have to engage in an enormous PR effort to communicate to customers that their product doesn't work with Chrome. These companies have done nothing wrong, but they're the one being punished, as customers don't know and don't care about cert issues.
2) "Updating" can be an enormous effort. As the blog post states explicitly, there are a lot of use cases where updating a deployed app is very difficult, almost impossible, or cost prohibitive. Again, remember that the person being punished here is not Symantec; you're publishing some poor development company who happened to use Symantec—one of the most popular cert providers on the planet—for an SSL cert. Not good.
I'm not taking sides on the issue, because I honestly don't know that much about it. But comments like the above really trivialize the huge impact that something like this will have.
The real question is, what effect would doing nothing have on all of those people? If Symantec continues to mis-issue certificates (as they have done repeatedly: https://wiki.mozilla.org/CA:Symantec_Issues), keeping their EV status intact could be much, much worse for them.
> As the blog post states explicitly, there are a lot of use cases where updating a deployed app is very difficult, almost impossible, or cost prohibitive.
As others have pointed out, this change would actually have no effect on the 'complex dependencies' Symantec identified:
* This has absolutely no effect on embedded devices, which are certainly not running Chrome. * Mobile applications that pin certificates shouldn't be affected, since the very fact that they're pinning certificates means they're not using Chrome (there's no way for an app to tell Chrome to pin a specific certificate).
On the other hand, Symantec's CA was caught mis-issuing certificates. That affects everyone, not just Symantec's own customers.
not doing something is allowing the previous lack of security standards to put all of those people in danger. there is a tradeoff with every decision.
the entire PKI ecoysystem relies on self management and I think google is making a tough decision, but one that ultimately has already led to better decisions regarding security.
So glad others are making this point. Not to mention that the CA system is one who's sole feature and only purpose is trust. There is zero security if you can't trust the root. We'd love to live in a world where providing security didn't require a trusted third-party, but that's not the world the internet operates on, today. Transparency is provided by establishing strict guidelines and expectations for CAs. Maintaining this trust is enforcing those guidelines equally among the CAs regardless of size/scope and punishing (preferably severely) any violations. Otherwise the old saying of "rules that are not enforced are just suggestions" comes into play.
Because it's a trust-based model, the consequences for violations of that trust have to be strong enough to make a compromised actor willing to refuse government intelligence divisions, corporate criminal actors or even organized crime that has the resources and could stoop to any level of blackmail/bribery to squeeze out a certificate that would allow one to spoof a connection to a high-value target like Amazon, Google, Paypal or Microsoft.
https://medium.com/@sleevi_/a-history-of-hard-choices-c1e1cc...
A suitable compromise could be an override switch that can only be activated via Group Policy. That'd at least be a reasonable suggestion from Symantec. Not their current offer of nothing.
As far as "updating" my point was that they described a routine mobile app update as "recoding" as if it'd be lots of actual code to rewrite. And again, the fact a large company has no way to easily update in face of a security problem really is their problem, not ours.
Those are precisely equivalent.
> Again, remember that the person being punished here is not Symantec;
Yes, it is, though you make exactly the standard "too big to fail" argument. yes, customers who relied on the misbehaving CA will also incur some costs, but if misbehaving CAs don't see reduced or eliminated trust at the level of browsers and similar trust stores, the entire ecosystem -- including the good actors and their customers -- pays the price as end-user trust in the ecosystem weakens because certificates do not guarantee what they are supposed to. And if any actor is recognized as too big to fail, that creates an unlimited license for bad behavior on the part of that actor at the price of the whole ecosystem.
Sometimes doing nothing is the wrong thing. They did wrong by supporting Symantec (or whatever shitty CA they originally did business with that Symantec acquired) with their money.
Or if initially they did not know about how shitty Symantec is, now they ought to know, and yes, it's a business risk (counterparty/vendor risk basically).
> It turns out WoSign has repeatedly been lying to Mozilla about backdating SHA-1 SSL Certificates
Source: https://www.riskiq.com/blog/labs/wosign-and-startcom-caught-...
Certs could be checked via Certificate Transparency but currently this is not required for DV certs.
I suppose that's true for mobile apps with embedded certs. Bet there's an app store / play store approval logjam for Symantec customers that miss this news, and all come crashing in after the expiry.
> Mobile applications that have pinned certificates. Replacing server certificates would require these applications to be recoded, recompiled and redistributed.
Are either of these relevant? Isn't google's proposal to eventually have Chrome stop trusting the Symantec CA system? That change seems like it would have no effect on 1) embedded devices that aren't running Chrome 2) things that already are in place that trust the CA.
In this case the story is more complicated - I think new certificates signed by existing Symantec CA roots would be trusted if their duration is 90 days or less. Historically, most CAs offered one, two, or three year duration. Presumably, the "big customers" would not accept the need to rotate their certificates for this purpose in less than 90 days, they need their 3 years because it takes over 90 days to get something through their bureaucracy. And presumably there are other things they can't do in 90 days, like update an embedded device configuration or firmware, or a mobile app.
Of course it depends on the particulars, but of course these "big customers" can't figure any particulars out in 90 days. But that's why Google proposed a slow sliding scale that would gradually reduce the max duration allowed over the course of a year, so random unknown stuff breaks more gradually throughout the year, not in one huge event like Symantec's blog post suggests.
Embedded devices => https://api.example.com (Symantec cert)
Browsers => https://new-api.example.com (New trusted cert)
Another approach is that if the devices are old enough they're probably using a deprecated version of TLS, so the existing https://api.example.com server could choose which certificate to issue based on the version of TLS offered by the client.
If Chrome ships this at the same time as re-enabling TLS 1.3 support, that would be a decent way to target, since presumably very few of your distributed clients are using TLS 1.3.
Or maybe I've misunderstood the situation.
I've always been of the opinion that the rogue/careless CA problem can be solved by mandating N root sigs and recommending M>N. In a system with multiple independent LetsEncrypt like entities this is doable. If a CA needs to be revoked you can do it in a much easier way here, since it's still ok to transitively allow sites with N-1 certs and one now-disallowed cert signed by that CA. Doesn't fix all the problems but it certainly removes the inertia behind kicking out bad CAs.
Looks like we'd need a x509 extension to do this then. :/
But I don't think Google is going to be swayed by "you can't distrust us if we do wrong because too much depends on us" because if they do, it encourages laxness at big CAs.
I could see someone translating "these devices use Symantec's server certs and serve web traffic" to "these devices are pinned to certs issued by Symantec". But if that's the case, they should pay more attention in announcements like this one...
Are these large organizations somehow incapable of putting out official statements regarding CAs? If they're being 'under-represented', it's their own fault for not speaking up.
I'm at a loss for a good analogy so I'll pick a lousy one: it'd be like if Samsung's response to the exploding Note 7 recall mandate were "Don't do this -- you're going to keep some of the biggest customers -- users of our phones -- from being able to make emergency calls/check their e-mail/respond to texts" ...yeah, but I'm still not OK with the small possibility that carrying my phone in my front pocket might result in third-degree genital burns.
[1] https://groups.google.com/a/chromium.org/forum/m/#!topic/bli...
Edit: They did ask for community feedback, comments on risk, etc. But they do already have a timeline. See link above.
As a user, you need your bank (or other large org) more than you need your preference of browser.
"Why isn't my browser working??" "So sorry about that, those darn nerds made a change on us."
I have seen this exact scenario play out. Mozilla and Google might be respected in the tech community; to most people they're an extremely tiny part of their life. "Your the IT man, just make it work"
As you get older, you transition from "this is how it should be!" to "this is how it works, what is a realistic amount of change that can occur?"
But that's just business as usual. You never know when your vendor goes rouge. And really Big Corps should have plans for these events.
I think the world is changing in ways which may cause the affected organizations to reconsider what they consider an acceptable CA solution.
Technically-correct well-actuallys lose to real-world perception ninety-nine percent of the time.
Works for me without Javascript.
For a bank maybe a user would temporarily accommodate but it would affect online stores and SaaS business a lot.
... well that's their problem, right?
You can't simultaneously say "These are some of the most important organizations in the world and you'll cause worldwide chaos" and "Won't someone listen to these poor companies, I am the Symantorax, I speak for the cohort, for the cohort has no tongues."
Thanks for that -- reading this comment connected a few thoughts I was struggling with.
Their response felt like "if you don't follow through with your plan to punish us we'll do all of these nice things for you." It reminds me of when my kids get caught doing something wrong and I get "but I'll do all of my chores and treat my siblings nice" along with a handful of other promises of future deeds which I know will be partially or completely not-done. How different the result would have been if they came to me with the chores done and some/all of the promises fulfilled already. It would have been much better for Symantec to have responded with these items in the "completed" category with a smaller number of "we will's".
But to the quote I highlighted: the issue at hand is the past failures indicate that "some of the most important organizations in the world" need to not be trusting Symantec in the first place. They've had a big enough history of failures as a CA to get the attention of Google, who I'm sure isn't terribly interested in degrading the browsing experience of their customers. The choice becomes: (1) Banhammer Symantec, take the resulting pain all at once, visibly and obviously, and get the problem fixed painfully (but it's fixed) or (2) Trust that things will be different this time and if they're not, have another security failure of some kind that results in these critically important company's customers' data being exposed, which is a circumstance that will almost certainly go unnoticed for a period of time, will be covered up if the exposure is minimal and covering up the exposure is possible or may not be discovered at all. The CA system is centered around trusting a handful of rather powerful bodies with an enormous responsibility around the overall security of the Internet. It's not a great model to begin with, but it's what we have and I haven't read about any alternatives that are viable. The only way to keep a model like this secure is to ensure that the CAs themselves are actually trustworthy and one of the more likely ways that will happen is to enforce violations of that trust in the manner Google is suggesting.
Among other things, if what Symantec is saying is true, this means there is zero financial incentive for Symantec to behave. The largest certificate customers in the world have all voluntarily made it infeasible to choose a Symantec competitor, so as long as Symantec has any product at all of any quality, the market will keep buying Symantec.
If what Symantec is saying is true, the mere fact that they went to their largest customers and said "Suppose you had to buy certs from one of our competitors, what would you do" and they replied "We've never thought about that before" is itself a major victory.
If you have too many trusted-third parties in a trust model like this, you run into difficulties with enforcement due to scaling problems, but what we're seeing is the converse of that. Symantec went out and bought up other CAs (Verisign is the only one I can remember for some reason) and they've become so large that them losing CA status would be prohibitively damaging to the internet-at-large. This affects not only enforcement (we don't want to move on a bad actor because the consequences of enforcement would be more severe than the consequences of a rule violation), it also negatively affects the trust model -- now if I'm a bad actor, Symantec is my only target -- even though I could attack the others and yield some of the same benefits of attacking Symantec, if I breach Symantec, the rewards are larger by several orders of magnitude depending on what they store and how they store it.
To digress a little ... today ... I'm not all that surprised that Symantec has reached sort-of monopoly status in this area. Not because Symantec is a well managed company (I'm making no statement for or against that despite my personal opinions), but just because so much of The Internet has turned out this way. If you would have told me in 1998 that for the very vast majority of US Internet users, there'd be one search engine, one e-commerce company, one (expensive) ISP, one e-mail provider, all of it would be secured by services provided by one company's CA and all of that wouldn't matter much because the typical Internet experience would basically be a web-enabled, bluer version of AOL[0], I'd have called you some unfriendly things, but that's where we are. It's a centralized system operating on decentralized technology.
[0] Oh, and that the "one e-commerce company" was that bookstore web-site and would be responsible for hosting such a large part of the rest of the Internet that when they had a major failure and people had that handful of rare clicks off of FAOLBook, they'd have about a 50% chance that whatever their target was wouldn't load.
This is how we make sure issuing certificates for gmail.com, etc will never happen again. An external audit every 3 months is not going to fix anything.
I see it as a polite way of saying "You kids".
I've got personal experience with the first two (embedded devices and mobile apps) having pinned certificates (or limited sets of supported CAs)
There's millions of internet-connected set top boxes, tvs, and dvd/bluray players deployed around the world that run with a limited set of supported CAs. (e: Also, don't forget the tons of 'smart' phones out there running ancient versions of whatever)
If you're someone who supports applications that run on those, then your choices for what CAs you offer on your API/Media/etc endpoints is extremely limited. Some of these decisions were made more than a decade ago, and migrating off those is going to take another decade (i.e as old hardware dies).
What exactly is the issue? If an embedded device trusts Symantec CA, then connections from the device to its central servers will continue working without interruption.
In this example, the device trusts the Symantec CA, and the server presents a certificate signed by the Symantec CA. Everything's hunky-dory. What's the problem?
So now if Google go ahead and distrust Symantec CAs, I need to duplicate that infrstructure (or at least the configuration at load balancers, etc) - this leads to all sorts of complexity.
Well yes if your "embedded" device has a user-facing webbrowser you will need the public CA system, but then I'd wager if you shipped such a device and don't have ongoing updates you have much much bigger problems than an outdated set of root CAs.
But there is zero reason for your mobile app communicating exclusively with your API endpoint to use the public CA system. You're paying for a less secure system with all the same hassles.
Ideally you'd have some way of tying it into some internal authorization database instead of relying on HTTP challenges, but HTTP challenges would work too.
It's a bunch of command line scripts because if you are using it any different way you are probably doing it wrong.
An API is required for you to have proper infrastructure as code and leaving any bit to human input other than another validation step in the automated process is waiting for errors to happen (invalid or incorrectly spelled cert info, missing cert chains, wrong cert chains, etc.).
Symantec even calls out how much work these companies will have to do because so few of them have a proper certificate management system in place that's come back to bite them.
This is the exact issue with the public CA system, their automated processes have been found lacking and now what they certify can not be trusted.
And you're missing the point I raised as well - if you have hosts coming up automatically, you either need automatic certificates, or shared keys. As long as you rely on SSL, there's no other way to go.
(For example of authentication: public CA verifies your domain, internal CA can ask the hypervisor to verify a token on your instance - not remotely in the same class)
But if the net effect is that one bug in a frontend can cause the server to tell the cert server to tell the secure enclave to tell the wire mesh protected crypto coprocessor to sign a bogus certificate then what are we really doing here.
We looked at various systems to manage intermediate ca issuance and reminders for renewal and the cost was incredible... most were >$200k!
Using scripts to automatically generate certificates and using the ugly but effective Windows CA web interface for "vanity" stuff does 90% of what those solutions do. I think we had an intern write a monitoring plugin to open tickets for soon to expire certificates and a few other things that covered the remaining 10%.