Linux Certificate Authority root stores have a too simple view of 'trust'
utcc.utoronto.ca
utcc.utoronto.ca
Not to mention stuff like this: https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+b..., where Ubuntu just unilaterally reverted Mozilla’s removal of a cert in their package, because it was breaking nuget… Note that this was early 2021 — Mozilla removed Symantec from their trust store in October 2018!
In general it just seems like a bit of a mess.
Mozilla actually removed the certs from their trust store in February 2021: https://hg.mozilla.org/projects/nss/rev/9718a34c84429b1e5dc6...
Debian and Ubuntu had jumped the gun by a few weeks and there were certificates still being used that had not been renewed yet, so we had to revert temporarily.
Mozilla had used the CKA_NSS_SERVER_DISTRUST_AFTER tag with a date to specify newer certs issued by that CA were not valid, but as the article above states, the crypto libraries being used in Linux don't support that kind of thing.
I can’t edit my comment now, but hopefully your correction here gets upvoted and is visible to people.
It wasn't unilateral. Ubuntu followed Debian here, who did the same revert: https://tracker.debian.org/media/packages/c/ca-certificates/...
So no, Ubuntu didn't do anything wrong or unreasonable here.
Edit:
Here's the Debian revert that Ubuntu followed:
ca-certificates (20200601~deb10u2) buster; urgency=medium ... Revert Symantec CA blacklist (#911289). Closes: #962596, #968002.
From https://tracker.debian.org/media/packages/c/ca-certificates/...
So yes shipping a CA known to have intentionally issued false certificates is very on-brand for them.
Did TrustCor turn out to have done that? The last time I checked in on that, the distrust was mainly founded on some not-very-trustworthy behavior involving spyware in a related company within the same corporate umbrella.
EDIT: Link to the rationale for distrust from Mozilla ... https://groups.google.com/a/mozilla.org/g/dev-security-polic...
excerpt: "There is no evidence of TrustCor mis-issuing TLS or SMIME certificates."
TrustCor met this requirement and got removed.
Other CAs removed in the past also showed they are unworthy of trust, but in even more blatant ways.
Trustcor allegedly spread spyware, which I do think makes anyone untrustworthy. But it should be checked if allegations can be confirmed.
I haven't heard that before. Care to elaborate what we do that makes you believe we don't take security seriously?
As for engineering decisions, let's just say better stick with debian on anything non-desktop.
Love it for my servers but it is not very convenient for the desktop.
I don't think it's as bad as most people think it is, nowadays.
That is the big feature of Ubuntu for me. Can set in my calendar when it's time to migrate services to the next version.
But that just tells us where the root certs are, not much more. In the API, I load the root CAs and check validity of the cert, nothing beyond that.
And I can do that in multiple different TLS engines.
Given that there is no API for this, all the TLS engines need to re-implement this logic. Hell, in most cases, even things like validity range checks are handled by the applicative logic.
On Windows, there is the builtin TLS API `schannel` which means that you have a lot more structure / order in this. Including the ability to express those sort of details.
There is another aspect, as well. You need to be able to express those issues as errors, otherwise the user is going to be left with a broken box and no idea what is going on.
The various TLS libraries already agree on at least one thing: what certificates look like on disk (e.g. PEM and DER formats). There's no reason why some committee couldn't sit down and standardize a format for ancillary data such as the trust date cutoffs mentioned here. Then the TLS libraries would have to implement it, and return appropriate errors from their validation functions. This could possibly be done in a way without needing applications to also be updated, though that depends on how each TLS library reports validation errors.
But even if a particular TLS library doesn't have a way to express these errors properly, I think it's better to have a user confused as to why they can't connect to a site, than silently connect to a site "protected" by a certificate that shouldn't be trusted.
Java would like a word with you :(
Java actually solves the complaint in this thread, and has done for a long time. The JDK has its own root store program run by Sun and now Oracle, TLS APIs use the bundled root store automatically, it provides tools to manage that root store, exceptions are easily rendered to users in a generic way, and it provides a mix of high and low level APIs.
What you're probably thinking of is the JKS format. Java defined a way to represent a root store on disk that's Java specific, I think because there was no standard at the time. But it migrated to the PKCS#12 format in recent releases.
What everyone's realized in recent years is that trust is context-dependent, and that you don't want to use the same roots of trust (or trust anchors, or CAs, though each of those terms has a slightly different meaning) everywhere, nor do you want to co-mingle them. It doesn't make sense to have one file that specifies "Here are all the CAs I trust, and here are rules for each of them"; It makes a lot more sense to have configuration per use-case of "Here are all the CAs for this usecase", and to rely much more on code within the program to disambigutate. Having one file with all those CA certificates and your private keys is... Backwards. Keytool and their custom format at least kept a clear distinction between "these are my private keys" and "these are keys I trust". PKCS#12 has no such distinction, and the tooling around it (Particularly Java's!) makes it hard to disambiguate safebags, so you're left with one giant mess.
It's actually the same problem as Global Variables[1]; Security and trust isn't one object. It's many. One program may act as itself to many providers, but it might also act as three or four different roles within it's scope to the same provider.
In many use cases you could use a store with only a self-signed certificate. That works, the APIs make it straightforward enough. It has many advantages when possible, e.g. you can set the expiry time to so far in the future it never expires, you don't have to muck about with LetsEncrypt and CAs. You do have to control the client and the server. If your frontend is a web app that's a problem. If it's a mobile or desktop app that's easy. Control over the client has other advantages too like not needing HTTP load balancers anymore (in many cases), reducing costs.
But most modern apps don't just connect to the internet, they connect to internal stuff like databases using TLS as well. It's actually a valid strategy to just use the web trust pool for that as well - I personally use it in places, with the certificate managed by CertBot reused for the webserver and the postgres server.
It's not a solution for the modern enterprise, where mutual TLS is sometimes a requirement and is attractive if not. You will want to have at least one non-web CA pool (Trust Bundle) for your enterprise, and sometimes several more for other departments, or other companies you collaborate with.
PKCS12 keystores work fine, however.
* https://openjdk.org/jeps/229
They already have (https://www.rfc-editor.org/rfc/rfc5280.html, https://www.rfc-editor.org/rfc/rfc6960.html) but CRL/OCSP functionality not implemented in most libraries because it's hard and thankless work.
Or at least, I'm certainly not aware of any mechanism in the CRL standard to "partially revoke" a CA certificate based on complex conditions like "What's the notBefore date on the leaf certificate we're currently validating?".
The problem is, code that handles these things is safety critical and requires real and sustained investment to be reliable. Given this overhead (and the gaps in CRL/OCSP support that I mentioned), it's unclear if this use case will be common enough to commit to supporting.
> There is another aspect, as well. You need to be able to express those issues as errors, otherwise the user is going to be left with a broken box and no idea what is going on.
TLS libraries are terrible at this. Part of it is C: the "integer is the only error type you'll ever need" cannot convey the necessary context, such as which cert is the problem.
Having a library might also help with path building bugs. I've seen bugs in both OpenSSL and GnuTLS in building a correct path, and both just from when the ISRG cross-sign expired.
… and maybe a more complicated data structure in /etc/certs (or whatever the path is) will keep prying vendors out. I've seen a lot of people make out of band changes there, and I would suspect those qualify as "undefined behavior", but OpenSSL's docs don't seem to answer the question of "what happens if vendors do random stuff?" (Like drop non-CA certs into the list of CAs, or don't update the symlinks that seem to form an index…)
Not really; I mean, sure, the validation function will probably rely on returning an integer error code, but there's no reason that the API couldn't also provide a function to retrieve the certificate chain at hand, where an application could walk the chain backwards to the root and figure out the last one (er, or first one, depending on how you look at it) that isn't trusted.
Whether an application would choose to go to the trouble to make use of these APIs is another matter, of course. At any rate, from perspective of the average user, TLS errors are already not so easy to understand, so giving detailed information in this case might not be all that helpful anyway. Just a simple "we don't trust this website, so you shouldn't either" is about as much as most users would understand, probably.
We might need something more - but I wonder i trust-stores could be modified so that:
- every device get a "host" CA (like a ssh key)
- this CA is the only one trusted
- in turn, this CA signs issuer/top-level CA (or even cross-signs the intermediary certs directly) (eg with - new/-force_pubkey)
Additional logic could be applied in the signing step - say rules to set the validity, or rules to sign/not-sign. One might set a 48h lifetime, and run a crown job every night - allowing for "dynamic" "revocation" (cert expiry).
Not sure if this would work out of the box though - I have not looked that deep into ca and ca trust.
The reason the Linux root stores don't have a notion of 'trust until' like the article wants is simple - nothing stops an untrusted CA just issuing certificates that claim to have been issued before the cutoff, so it seems pointless. Browsers have ways to tackle that like Certificate Transparency or crawling the web to try and identify every extant cert, but most apps don't use that.
However, now CAs can't really do this anymore, because - as you say - they'd risk immediate exclusion from browsers if this is detected via CT analysis. So Linux distros can actually benefit from CT and browser's impact in the CA space in an indirect manner.
There is definitely a power imbalance though. E.g., even if a distro implemented "trust until", they could not realistically make their own rules that are stricter than what browsers do: CAs could backdate certs so they get accepted by the distro, but if browsers consider the CA fully trusted, they might not care about the backdated certificates.
(I'm the author of the linked-to entry.)
(b) is a pretty tough problem!
I started to think (but not check specs): Is it even defined what domain name should be checked? MX records can be different from server names. Maybe certificate checking is just not defined for SMTP over TLS?
The decision of whether or not to believe the authenticity of the
other party in a TLS negotiation is a local matter. However, some
general rules for the decisions are:
- A SMTP client would probably only want to authenticate an SMTP
server whose server certificate has a domain name that is the
domain name that the client thought it was connecting to.
P.S. As you probably guessed in my previous comment "3 months later" should of course have been "3 months earlier". Sorry about the mistake.
I suppose you could write up a way to specify this in DNS but it’d take decades to be implemented and you would have to deal with pushback from the snoops who I’m sure don’t mind they can peep into the email you receive.
So having an arbitrary cutoff date in software seems unnecessary. If you still trust the CA to behave honestly for now, then you can simply instruct them not to issue any more certificates.
If you don't trust the CA to act honestly for now, then you need to remove them from the trust store entirely (and maybe use a whitelist of previously issued certificates if you believe they were trustworthy in the past).
Dare I say it, but isn't this the exact problem solved by a chain of published certificate hashes?
This is pretty much how certificate transparency lists (CTL) work too. It's like a blockchain without any proof of work or stake, just a merkle tree of hashes that ensures you cannot retroactively insert one with an older date.
> ... (and maybe use a whitelist of previously issued certificates if you believe they were trustworthy in the past).
Which I guess would be only trusting CTL published certificates up until the last point of trust.
Just because something has a set of hashes in a roughly linear timeline doesn't make it a blockchain. For example, git isn't a blockchain either.
> A blockchain is a type of distributed ledger technology (DLT) that consists of growing lists of records, called blocks, that are securely linked together using cryptography. [...] Since each block contains information about the previous block, they effectively form a chain, with each additional block linking to the ones before it. Consequently, blockchain transactions are irreversible in that, once they are recorded, the data in any given block cannot be altered retroactively without altering all subsequent blocks.
So far this is exactly what's going on in the Certificate Transparency scheme.
> Blockchains are typically managed by a peer-to-peer (P2P) computer network for use as a public distributed ledger, where nodes collectively adhere to a consensus algorithm protocol to add and validate new transaction blocks.
This part deviates from CT (no validation is going on in real time, browsers usually just have a few hardcoded CT logs and a custom rule on when a certificate is trusted). But note the word “typically”: this doesn't mean CT can't be called a blockchain just because it doesn't use PoW, PoC or any other consensus algorithms conventional in the cryptocurrency world.
Calling Git a blockchain is arguably more of a stretch because the purpose of keeping the old records intact is not what we use Git for. Maybe you can build blockchain on Git though? (edit: OF COURSE YOU CAN https://github.com/CouleeApps/git-power)
---
[1]: /s
Given you like Wikipedia so much, here's their definition of what a ledger is (https://en.wikipedia.org/wiki/Ledger)
> A ledger is a book or collection of accounts in which account transactions are recorded.
You'll find similar definitions on non-Wikipedia sources as well.
Thus, if it doesn't have accounts and transactions between them, it isn't a ledger and thus isn't a blockchain.
Using TrustCor as an example there is no hard evidence they have done anything wrong. Is that enough to justify cutting off their clients?
What if one of those clients is a major antivirus vendor etc. with pinned certs in their software? Or road signs etc.?
After all, backdating certs is really obvious in a world with cert transpatency, and you can full distrust immediately if they do that.
Netflix for example uses this feature internally and built a test suite: https://netflixtechblog.com/bettertls-c9915cd255c0
I agree wholeheartedly that this feature should be used more widely to restrict CAs where practical, obviously limiting government CAs to their respective ccTLD(s) seems like an easy one. Personally I'd also like to see this extended to allow for a domain owner to get a private CA certificate issued for their domain(s) which can then be used to issue individual certificates within that/those domain(s) as a more secure alternative to wildcard certificates.
There is no substantial technical reason this couldn't be done, just a lot of older software that wouldn't understand the restrictions and could either reject the certs entirely or consider them valid even if they shouldn't be.
Also, is this additional information even published by Mozilla in a standardized format or if it's just put somewhere into Firefox source code?
Also note that, given how Firefox doesn't implement SCT requirements, TrustCor could also just back-time certificates. So Firefox has similar vulnerability properties as users of distros that haven't removed TrustCor as a CA.
So ideally you would also make it possible to distinguish between OS provided and custom configured certificates, to allow libraries to check for SCTs at all. Otherwise you haven't gained much from the additional information of limiting a CAs ability to issue new certificates.
Not really a single format, but they do write it down:
How does this work? If TrustCor is no longer trusted, what keeps them from creating certificates which claim to be issued before December 1st, even after that date?
See https://groups.google.com/a/mozilla.org/g/dev-security-polic... for the actions proposed depending on how the TrustCor situation plays out:
> If there is reason to believe that the CA has mis-used certificates or the CA backdates certificates to bypass the distrust-after settings, then remove the root certificates from Mozilla’s root store in an expedited timeline, without waiting for the end-entity certificates to expire.
Right now, they're being slowly removed for poor behaviour in general, but there's no direct evidence of abuse of CA powers. If any clear evidence of that appears in future, including backdating certificates, then they'll be completely removed from the trust store immediately.
(But it's the only browser that doesn't support CT)
Moreover, Chrome removed the browser extension API for TLS certificate details, so it is not even possible to do CT log verification via extensions.
Only way to do CT log verification would be by customising an existing TLS MITM software. As far as I’m aware no such solution exists at the moment.
Certificate system is actually rather insecure, and although solutions are possible to develop, nobody has taken time to do it.
Yes they do. Only firefox doesn't.
The SCT contains a signature over some log related information (which is also included in the SCT itself) and everything in the pre-certificate except the signature and poison (which means everything in the real cert, except the SCTs and signature). This means the browser can reconstruct the signed data and verify the signature.
Thus the only way to have a valid SCT and not have at least the pre-certificate show up in the log (after the merge delay) is if the log operator/software messed up. For transparency purposes, a pre-certificate is basically as good as a full certificate, although if I recall correctly the CAs are also supposed to submit the full certificates too.
The idea that you need the approval of one of these companies to publish to the web without a wall of alerts is just absurd.
There are a whole variety of other problems with a DNS PKI, and with DNSSEC in particular, but when we're talking about issues like this thread, revocability is the the big thing.
I also think you may be underestimating the amount of leverage browser developers have over DNS, should they choose to exercise it. If Google, Apple, Microsoft, and Mozilla all agreed tomorrow that they wanted to migrate the web to a new set of DNS root servers they could probably do it. (Though yes, it would be a huge ordeal for everyone involved.) The real reason we haven't gotten DANE transparency (or anything similar) is simply because it hasn't been a priority for browsers. (Why pressure DNS roots to improve a system that you aren't even using in the first place?)
Still, I can't deny what you're saying. The CA system is probably a lot more flexible than a naively designed DNS-based system would be. I'll have to think about that a bit more. Maybe it would make sense to have something like SCTs in the certificate transparency system, where a number of different independently operated organizations all have to sign off on the validity of each cert _in addition_ to the DNS operator that actually issued the certificate. That way you'd get the benefits of diversified trust anchors without the downside of having hundreds of distributed single points of failure.
Browsers always had full control over CA root programs, it's just a question of how willing they were to actually assert themselves in that fashion. I agree they've become more bold about that over the years, but what Mozilla just did to TrustCo was always an option for the major browser vendors, both legally and technically (or at least since automatic updates became a thing). DNS is also effectively a browser feature these days, as evidenced by all the browsers suddenly supporting things like DNS-over-HTTPS despite host operating systems lacking support. Given a sufficiently smooth transition path for users and website operators, browser vendors could collectively decide to alter their DNS implementations in pretty much any way they want.
However, the current status quo is sort of ignoring the big DNS-shaped elephant in the room. You can build all the validation and transparency solutions you want on top of the CA system, but it's still fundamentally dependent on the security of a DNS system that currently requires no cryptographic assurances that the records the CAs are validating against are actually correct.
They're already pushing DNS over HTTP.
https://news.ycombinator.com/item?id=33810755 ("Mozilla, Microsoft yank TrustCor's root certificate authority", 5 days ago, > 350 comments)
A decade ago, probably. Today, with CT logging being mandatory in most browsers? Much less likely.
I don't think using curl and openssl directly check CT, though....
Trusted CAs should be one of the most scrutinized and controversial aspects of system configuration, and OSes should support a variety of trust models.
I've run this by a few security "experts" and the response has always been that it's a UX issue and the goal is to have websites load correctly without the user seeing and complaining about opaque security errors.
It's quite absurd. The rewards for compromising a single CA are so great that surely most state actors have succeeded in doing it at least once.
That approach is nice for ops (don't have to worry about commands to add/remove certs, just drop files into dir) and relatively performant (just one read on cert's fingerprint file name in modern distros).
I think simplest one would be just adding meta file with a bunch of conditions ?
While we're at it, allow the certs to be imported only for certain domain, so ops can, for example, import internal CA of their partner but limit it to only partner's domains
I hate to say it but... I want a systemd-certd that TLS libraries can call into saying 'verify this please' and then the logic for verifying trust path, validity, revocation status etc can be done in one place, consistently and correctly.
Ok. Which CA should i trust ? And why ?
The problem is very broad though. CA/B forum is very effective for what it does, but equivalents for other uses are less so.
Most places crib and slightly rework the Mozilla CA pack with highly variable levels of maintenance and accuracy.
macOS and windows have richer models, but integrations are often poor. For many years homebrew had code that seeded an OpenSSL compatible pem with explicitly untrusted certs on macOS.
We could do with something as effective as CA/B that covers more use cases. Along with it should come some amount of reference implementation material and regularly updated test vectors.
Essentially we need the client side of let’s encrypt, and it can’t be let’s encrypt that does it.
And even if, all I'd get is a "this site is not trusted" that I can manually override if required.
ca-certificates package rolls in Java and Mongo CAs as well as Mozilla.
Their tools to do that is also not very clear on how they are put together the ca-certs either from 32 different subdirectories.
A (ahem) short write up that also includes suggestion of improvements. Debian bug has been filed but the Debian maintainer does not seem to be interested in moving this along.
https://egbert.net/blog/articles/ca-certificates-rebuild-on-...
Someone mentioned earlier that "At least we all agree that they're PEM and DER Formatted", to which someone mentioned that Java had it's own world, and ugh... That's a whole thing. Java has moved backwards in recent years; Their old Keystore format was useful for a single context, in that you could say "Here's the keystore for communicating with Google, it's got a list of public keys to trust and list of private keys that we are"; And that more or less worked with the tooling. The move to PKCS#12 moves to one unified bag with all your private and public keys mixed in and you have to specify which are which, which is theoretically better but in practice represents a configuration nightmare over just specifying separate files with the trusted CAs, our public keys, and our private keys. You still can't deliver a single PKCS12 file to all of your servers as a whole, because SafeBags aren't; They're DES encrypted, which is to say: Not at all[3], but at least it's clear that they aren't.
The right answer is basically what Cloudflare[4] and then Google Cloud[5] tried to do and failed: Create a single file-format for "Public Key and accompanying Private Key", and make it easy to extract the public key from many of those to form a list of CAs.
[1] https://github.com/kubernetes/enhancements/pull/3258 [2] https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/ [3] The actual Java recommendation is to use a single well-know DES key for all PKCS12 operations, which obviates all of the stated and even best-case-steelman arguments for why Java made this change, to the point that anyone defending the decision should probably not be trusted and the proposers were more likely malicious than stupid. That's a rant for a different post. [4] https://github.com/cloudflare/cfssl [5] https://cloud.google.com/docs/authentication/application-def...
"Hey, this page uses a cert signed by $THIS_CA, would you like to trust it?"
Then instead of shipping whitelists, you could ship blacklists, which is presumably simpler and more future proof.
Personally I rely on a TLS forward proxy that uses the certificates in /etc/ssl/certs, as represented by ca-certificates.srt. Thus, I can remove certificates, regenerate ca-certificates.crt and effectively deny certificates that the advertising-supported browser company chooses to accept.
Web browsers from advertising companies, or organisations that subsist from advertising company profits, like Mozilla, ignore /etc/ssl/certs. They do not allow computer owners to remove and delete CA certificates. They refuse to allow the computer owner to add their own choices for CA certficates, e.g., certificates generated by the owners themselves, without forcing annoyances on the owner to try to discourage such owner control.
The "CA system" should be accountable to computer users, not advertising companies, or organisations that depend on advertising company profits, not "tech" companies. As it stands, computer users are locked out of the decision-making process with respect to CA certificates. The CA system has become a tool of "tech" companies that seek to commercialise every aspect of the internet. The fact is, much of the data "protected" by TLS is data belonging to or about computer users that is being sent to "tech" companies without the user's informed consent. In effect, TLS is used to stop the computer owner from seeing what data is leaving their computers and networks.
We need a CA system that is governed by computer users, not "tech" companies. All computer owners should not be discouraged from trusting themselves. The CA certificates computer owners generate should not be discriminated against in favour of ones approved by "tech" companies. Someone is no doubt going to reply to this with something about LetsEncrypt. But that is not letting users trust themselves. It requires users to rent domain names and ask to be trusted by LE. It is part of the existing CA system where users must (cf. may) let third parties decide who is trustworthy.
curl -4o 1.csv https://ccadb-public.secure.force.com/ccadb/AllCertificateRecordsCSVFormat
grep TrustCor 1.csv