Preparing to Issue 200M Certificates in 24 Hours
letsencrypt.org
letsencrypt.org
Unfortunately, a yearly tax has a margin, so the vendors have been dragging their feet. DigiCert and GoDaddy have bribed the Azure team with kickbacks and wholesale discounts, so Azure is now refusing to implement anything else. Why would they? They get a margin on every issued certificate!
So please: If you have a presence in AWS or Azure, call your sales representative and pressure them. Submit requests through the feedback portal, and vote up existing requests.
Raise this regularly, or it will never happen by itself. The incentives just aren't there!
Pressure must be applied by a large fraction of the customers.
You are that customer.
Apply pressure.
As far as I know Azure's App Service has no direct competition other than AWS and GCP.
Initially these certificates were only supported for load balancer and cloud front but with the Secure Enclave you can use the certificate also on an ec2 instance.
Does not support wildcard certificates.
Does not support naked domains.
Is not exportable.
Is not supported on App Service Environment (ASE)
Does not support A records. For example, automatic renewal doesn't work with A records.
Not to mention that this doesn't cover other very common scenarios, such as Application Gateway, VM scale sets, API Gateway, or... anything else.
Each Azure team seems to be operating under the model that HTTPS with a custom domain is some sort of bolt-on that's unique and special to their service. The verification and enrolment is distinct for every service, with gaps and weird and wonderful limitations.
It's like they've been told, recently, that HTTPS is something they should do, so they've all gone and done "something" to tick that checkbox. Some free. Some not free. Some with restrictions. Some without. Some automatically renewing. Some not. Some with ECC, some without. HSTS on some, not others. Etc...
It's a shit show.
At least newer versions of Certbot, and I believe some other ACME clients, will also try to discern if the certificate is Revoked when considering it. So if you have a daily cron job running Certbot, and your certificate with 60 days left on it has been revoked since yesterday, Certbot ought to notice that and attempt to replace it as if it had expired.
If you are doing OCSP stapling, and if your stapling implementation is good (sadly last I looked neither Apache nor nginx were) this ought to be enough to make a mass revocation event survivable for you. Your server will notice the latest OCSP answers now say it's revoked and continue to hand out the last GOOD answer it knew, some time later before that OCSP answer expires your Certbot should replace the certificate with a good one. Seamless.
The new ACME feature is welcome, not least because there are a tremendous number of those bad Apache servers out there, but (unless I misunderstand) I think it's already possible to survive this sort of catastrophe without service interruption.
It explicitly mentions two big flaws with the old one, but not the one most relevant here - out of the box Apache's old OCSP stapling would merrily staple BAD answers simply because they're newer, which makes no sense. I assume that's corrected, but if you know this'd be a good place to say.
And then 60 days later, every single client tries to renew that certificate. That's another 200 million certs in 24 hours. And that'll repeat every 60 days.
So the ACME draft is also about being able to pro-actively smooth out that renewal spike. Some clients would be told to renew again immediately, less than 24 hours after their replacement. Others would be told to wait the whole 60 days. And then after a couple months of managing that, things would be back to normal.
I hope everyone realises that Let's Encrypt is by now an essential part of the Internet, somewhat like DNS is, just massively centralised.
What exactly is the issue with centralization here?
I couldn't name a second, which means they're probably not getting a huge %.
You can both block certs that do not appear in the logs, and decide which certs not to trust ("everything after Friday the 13th at midnight is not trusted"), once you know the date/time of the intrusion.
but the issuance time isn't relevant, they can easily backdate the cert
You can start reading more info here: https://developer.mozilla.org/en-US/docs/Web/Security/Certif...
(HPKP is considered obsolete, if by cert pinning you meant that. If you're confusing HSTS with HPKP, just know that HSTS makes it much harder to mistakenly access the site via http, while HPKP [now deprecated] is the practice of ensuring some hashes is found in the cert that the server sends to mitigate some attacks)
Oh, that’s a neat one and would be very valuable for me. We have poor internet access at home and would be cool to reduce traffic going out to the net.
Thanks for the reply, appreciate it!
https://community.letsencrypt.org/t/certificate-failure-due-...
Pinning has uses, but if I'm running an app, I'm not doing pinning unless I have to.
Pinning to the issuing or root is much safer from an availability standpoint.
https://docs.aws.amazon.com/enclaves/latest/user/nitro-encla...
Of course, the other failure points didn't completely go away yet. But I do expect their number to reduce a lot in the future.
In an emergency.
It would take quite some coordination but I suspect the current pandemic might provide some models and examples as to the way to do things at scale with JiT decision making from wonks who normally drag feet by default.
I'd love to see, say, another charitable CA set up in a different jurisdiction[0] that can hold a similar reputation to the ISRG.
[0] Probably somewhere in Europe, whether that's in the EU or elsewhere like Switzerland or the UK. What matters most is that it's somewhere with strong commitment to the Rule of Law and not beholden to the US.
This is Linux right ? would this be the largest deployment of ZFS-on-Linux then ?
(edit: units)
Your age is showing :-)
Every few years, I gotta get used to the 100s MBs is big!! -> 100s GBs is big!! -> 100s TBs is big!!
Seems like we're entering the age of PBs, and then we stopped caring about capacity and more about the speed of our TB+ sized archives.
I mean, if you enjoy sorting through going "OK that's photos of my kids, that goes in pile A, but these are materials from the re-mortgage application and go in pile B" then knock yourself out, but I have other things I want to do with my life, leave it to the machine to store stuff.
If you lost everything that's because you lacked backups and (repeat after me) RAID is not a backup. Everybody should get into the habit of doing backups. Like any survivalist learns, two is one and one is none.
The "can't read from any other machine" is handled by making sure (this includes testing) that the volumes are readable with dmraid. At least that's for SAS/SATA applications. I'm not sure about NVMe, as it uses different paths in the IO subsystem.
Is this not easily mitigated with a smart UPS? (i.e., one that will notify the host when the battery is low so it can shut down cleanly)
Not really. In a single socket setup, an EPYC gives you 128 lanes. In a dual socket setup, 64 lanes from each CPU are repurposed to connect them together instead of doing PCIe. So just like single socket, you end up with 128 lanes total.
For one thing, "revoking" a root CA isn't a thing, the root's signature is mostly a convenience (it's self-signed) and perhaps in another world roots would exist as distinct documents not as X509 certificates signed by themselves. So there isn't anybody to "revoke" it like other certificates. You can distrust them though.
In the real world lots of systems would never become aware of the revocation/ distrust at all, and there are gatekeepers for many other systems that could become aware (e.g your copy of Chrome or Firefox can learn that a root CA is no longer trusted but it would not do so without humans at Google or Mozilla deciding this was a necessary course of action).
It's necessary to the plot of Rainbows End that this happens unrealistically quickly, Rabbit must be disabled or it will certainly interfere with subsequent events, and it seems Vinge (unlike me) isn't sure exactly what Rabbit is, so this vague technical intervention seems like an effective way to stall Rabbit without thinking too hard about that question.
Revoking a signing cert, breaking the cert chain, would indeed make a mess for everyone using those certs.
How often do you suppose most systems examine that CRL?
If you guessed anything other than "never" you're wrong. Almost nothing you use will ever notice.
But that's not enough anyway - R3 is also trusted via a cross signature from Identrust's DST Root CA X3. So their organisation also needs to be woken and persuaded to perform an unprecedented middle-of-the-night revocation of some certificate they've seen no evidence is a problem.
Now, you've disturbed all these nice people, with your very urgent problem of... you want to stop something you can't discuss happening that involves some confidential things and a bunch of other confidential things and they must never speak of it to anybody - in the morning they're going to be doorstopped by a thousand tech journalists wondering why they broke everything - or they could just hang up. But we'll suppose you did that, as I said it doesn't have any effect. Oops. Rabbit will notice, no doubt, but it isn't disabled.
OK, so what can you do that will actually have some impact? Well, you can get Microsoft, Google and Mozilla to use their out-of-band "kill switch" functionality. Each works differently. Microsoft's has the advantage that it's entirely a closed door process, if you can get the ear of the right person you can make any change you want. Unfortunately the latency is one calendar month. Rabbit is causing you a problem today? By April we can fix that. Oh, you need something sooner? Too bad.
Google are more promising, they can tell most Chrome installs to distrust R3 without any independent confirmation and the updates will typically take only a day or so to have effect. Rabbit will be out of your hair before tomorrow's evening news can run the story about your resignation or arrest. They will probably tell everybody why though...
Mozilla likewise can react quickly, perhaps in just a single day. However unfortunately Mozilla deliberately makes these decisions in public. You're going to need to tell a bunch of random people who don't even work for Mozilla about your urgent need to shut down this issuing CA. They're going to have questions. You probably don't even want to read the questions, never mind answer them. Oh dear.
If a countersigned certificate is revoked by one signer, is it still safe to use? I hope your answer is 'no'. Otherwise, as you say, I can revoke the certificate in Europe but have to wait for the Americans to get out of bed.
Why did they including both certs in the chain if everyone trusts the same CA? They are including them because people cherry-pick. My app is likely not checking both roots, but picking one as the one I trust and seeing what it says, in many if not all implementations. Because I'm looking for a chain that I understand, then seeing if it's valid, and one of the chains I don't understand.
DOS attacks are not an all or nothing thing. Often it serves the attacker's motives if only most of the traffic is blocked, or specific traffic is unlikely to succeed. Taking out a particular person or "just" 75% of users could be desirable.
Unlike "your code" and that of your co-worker who ever that might be, I can tell you exactly what browsers do and you're wrong, though I haven't any idea who (if anybody) "told" you that they check OCSP to "check the cert chain".
Chrome doesn't have OCSP fetching code. It uses CRLsets, controlled by Google, which I described previously.
Safari does do OCSP... if Apple's backend tells the client this certificate was revoked. Why then? Ask Apple, makes very little sense to me, but that's their policy. Again you'll need to liase with Apple HQ to figure out how to get their backend to report all these certificates revoked or nothing happens.
So in the modern world that leaves Firefox, which does have OCSP fetching code, but it's default off and they recommend you leave it that way. Why? Because it's privacy infringing. OCSP sends the CA information about which subscriber certificates you relied on recently. Why were you relying on the certificate for Porn Hub? Exactly.
--
> If a countersigned certificate is revoked by one signer, is it still safe to use? I hope your answer is 'no'.
There's no such thing as a "countersigned certificate" in X.509 thus PKIX and the Web PKI. There can be multiple CA certificates with the same subject, which is what we call a "cross signed" CA.
There's no reason you should treat a certificate differently based on the existence of other certificates with the same subject which have been revoked, and indeed the software you use does not (on the whole) treat them differently. It's likely that you've used such trust chains especially if you're an American because a public CA (not Let's Encrypt) signed the Federal Bridge CA mistakenly at one point, which implicitly imports this huge hierarchy of other signature chains, some of which are revoked.
> Why did they including both certs in the chain if everyone trusts the same CA?
I guess you're talking about the existence of R3 signed by DST Root CA X3 and R3 signed by ISRG Root X1. But those aren't a chain, and so they aren't "including both certs in the chain". They're two certificates for the same entity, and so there are two completely different chains offered by Let's Encrypt.
Depends. The encryption is the same and only dependent on your client/server. Could have been solved by DNS ... maybe.
Self signed certs don't validate that you at least own the domain.
So it's not nothing, but it isn't very much without the Certificate Authority role.
* man-in-the-middling Let's Encrypt and a particular domain (or DNS, depending on the domain validation challenge)
* requesting a new certificate for that domain
* spoofing the HTTP resource response (or DNS response, if applicable)
Let's Encrypt issues multiple verification requests from multiple servers in different locations, both physically and in the network topology. If you can MITM that, you've pretty much taken over the domain and the ability to get a certificate isn't the worst of the operators problems.
That assumes a lot about the operators goals and values. It may very well be their worst problem. Eg. a journalist in a dictatorial area will very likely prefer not to have a cloud service than to upload his data into some compromised service.
It's just that, if it is their worst problem, TLS is patently insufficient, so they must think about it when setting the system up.
But what tempers my concern over that finding is that we found this by looking at cases where there's clearly a DNS takeover - and actually it was rare to do certificate issuance. In most cases it seems if you can MitM say, an Arab's country's army headquarters group mail servers, you can just offer no encryption or serve a self-signed certificate and the users will accept that. So while the Ten Blessed Methods are, as expected, not enough in the face of a resourceful adversary (in this case perhaps the Mossad, NSA or similar) they're also a padlock on a fence with a huge gaping hole in it anyway, our first attention should be on fixing the hole in the fence.
For instance, you can subvert the internet infra around some ccTLD if you're that country pretty easily but then who really owns the domain? Probably you, the country, since you can do anything with the DNS and then anything with the traffic.
In contrast, incumbent low-budget CAs have often been a bit of a wild west of automated validation methods, often based on email, that can and do fall to much simpler attacks than a large-scale on-path attack. While CA/B, Mozilla, and others have worked to improve on that situation by requiring CAs to implement more restrictive policies on how domains can be validated, ACME still represents a much better validated, higher-quality validation process than that offered by a number of major CAs for DV certificates.
One approach to decentralized or at least compromised-CA-tolerant TLS is something called "perspectives" (also implemented as "convergence"). The basic concept is that it's a common attack for someone to intercept your traffic, but it's very difficult for someone to intercept many people's traffic on the internet. So, if the TLS certificate and key you receive from a website is the same as the one that many other people have received, it is most likely genuine. If it's different, that's an indicator of a potential on-path attack. This can be implemented by establishing trust between your computer and various "notaries" which basically just check the certificate from their perspective and confirm that it matches yours.
I bring this up, because if you squint just right you can view ACME as being a method of bolting the same concept onto the existing CA infrastructure: before you connect to a website, LetsEncrypt acts as a notary by connecting to the domain from multiple perspectives and ensuring that the same person evidently controls it from all of them. While not perfect, this is a strong indicator that the person requesting the cert is legitimate.
The on-path attack risk is almost, but not always, on a late stage of the network path to the user (e.g. their local network). The big weakness of the ACME approach is an interception of a late stage of the network path to the server. This tends to be much better secured, but hey, it's still something to worry about. There is obviously also a reliance on DNS, but I would say that DNS has basically always been the most critical single link in on-path attack protection.
They verify domain control, not ownership. Verifying ownership is a much harder problem, and one I doubt could be done in an automated fashion.
This stuff is much more complicated in isolation from the rest - full picture easiest.
To be sure, PKI and certificates in particular have a lot of room for improvement in the UX department. Especially on Windows, where one frequently has to deal with not just .pem files but .cer, .pfx (with or without private keys), and more.
What do you think about connecting to a network, and your phone sends a user/password to a server but instead it's being intercepted?
And the user has no idea.