Let's Encrypt ACMEv2 and Wildcard Launch Delay | Hacker News Reader

Let's Encrypt ACMEv2 and Wildcard Launch Delay

community.letsencrypt.org

140 pointscm218774 comments

While HTTPS is being burned into the architecture of the Web, I have a humble question to ask: what is the dear HN community's next-best-option in a scenario if letsencrypt dies/gets sued/goes under/servers not accessible anymore?

I ponder on this question seeing how Google is planning to increasingly depreciate HTTP. While in general, I welcome the security upgrade, the lack of next-best-free-options makes me vary of this config change locking us into a $$$paid-only web publishing system.

ACME is a protocol - I imagine most of us are hoping that, should LE fade off, one or more others take the reins and implement it.

With PKI,a good protocol isn't the problem but rather making the cost of infrastructure keep-up worthwhile. Anyways, non-pki protocols are in the works.

> non-pki protocols are in the works.

Could you kindly link to relevant threads / RFCs / orgs / undertakings currently working on that, please?

DANE might provide a solution. See specifically the section https://tools.ietf.org/html/rfc6698#section-2.1.1, certificate usage 3:

3 -- Certificate usage 3 is used to specify a certificate, or the public key of such a certificate, that MUST match the end entity certificate given by the server in TLS. This certificate usage is sometimes referred to as "domain-issued certificate" because it allows for a domain name administrator to issue certificates for a domain without involving a third-party CA.

This would need DNS to be secured though and DNSSEC is still a bit of a mess last I checked and not enabled for a majority of DNS traffic.

DNSSEC support is increasing each year, but that's just one issue. DANE would also need to be implemented by browsers for full adoption, not just as a plugin to specific browsers.

Then again, I was responding to the question about an RFC or other standard, not whether it was feasible today. ;-)

It would be feasible if DNSSEC wasn't a total mess, tbh. The support for it is still abysmal and a lot of resolvers (including the one in my router) can't handle DNSSEC responses at all.

I think using DNS over HTTPS in conjunction with signing the response is going to be more viable since you don't have 200 ways a middle box will break it.

If you want to sign over the security of the Web PKI to the United States Government, DANE is a pretty effective way to accomplish that. It is, essentially, an Internet key escrow scheme.

This is one I personally have grown fond of: https://github.com/DeDiS/cothority

I've heard of blockchain based ideas as well but I'll leave that to your google-fu

But it would have to be somebody who already have their public root keys in all the browsers.

But Let's encrypt is not just the ACME protocol, it is ACME + free certificates.

With Lets's Encrypt, consensus hopefully strengthens that the cert system is not a good situation. This may eventually web browsers to support TLS PKI methods other than X.509.

Free is certainly nice, and I'd probably go back to self-signed for a few private subdomains, but you can get a paid cert for under $3/year. The only people really screwed would be those running a non-profit project that used many (sub)domains.

Where are these $3/year certs? Cheapest I've found is about $9.

From a quick search, looks like Comodo's PositiveSSL is being resold for $3.88 for 1 year and $2.88/yr for 2 years.

ssls.com (Namecheap). They're on sale, but even the regular price is only $7 (if you buy two years).

The thing about certificate authorities is the whole system is as insecure as the worst of them, so you may as well go find the cheapest.

I've even managed to get wildcards for $30/year from an alphassl reseller.

My automation tools are slowly migrating towards a proper PKI set up that supports ACME, self signed certs and external certs (vendor signed certs from anywhere).

I first saw this strategy used in the DebOps project's PKI Ansible role at https://docs.debops.org/en/latest/ansible/roles/debops.pki/.

Basically if LE goes down, all I have to do is make 1 line of YAML changes, run an Ansible script and my servers are secured by a different provider's certificate.

It's hard to complain about a slight deadline miss from an incredible project offering free stuff.

I am a little worried about the monoculture LE is creating.

I don't have any bright and practical ideas to fix that, however. Short of some other consortium making an equivalent, but with different tech underneath.

For TLS certificate authorities, a monoculture is way better than the alternative.

I'm thinking of some critical flaw in ACME, or CA outage, bad cert push, etc. I get your point, but I think there's a legit risk here.

Precisely the problem. Real companies mostly prefer to use paid solutions because they have comfort and confidence in knowing there is a business and backing behind it. At least I know I do. I never got the whole developer complex wanting and even expecting everything for free. Seems hypocritical and counter-entrepreneur.

Also for what's worth, I've been deploying Let's Encrypt into production recently using Caddy and (on-demand) TLS and while it works, the rate limits being reset weekly is really scary and VERY VERY easy to hit. I know myself and my clients would be willing to pay for Let's Encrypt Pro (higher rate limits, etc). Take our money.

That's not Let's Encrypt's role to play though. As a user, I'd be much more skeptical of an LE cert'd site than a Verisign cert'd site. That's not a knock on LE, that's just recognizing a higher barrier to entry. Let's Encrypt's goal is to get everyone encrypted, not necessarily to ensure everyone is who they say they are.

And as a fellow person who does real business at times, I often prefer paid products because I know I can get support from the company backing it. I don't think Let's Encrypt will replace the entire CA industry, and I don't think it should. (As tyingq says, monocultures are bad.)

> not necessarily to ensure everyone is who they say they are

How does Verizon (to take your example) ensures this? They have the same zero verification process for the non-EV cert.

At bare minimum, getting a cert almost anywhere else requires a credit card.

This is not something which helps to decide whether the target website is what they claim to be.

I can use my CC to buy faccebook.com, and I will get it (again, outside of EV).

I may then be tracked by Interpol for fraud, but to the end user this does not change anything: Verizon has issued a certificate for my faccebook.com

IYAM, any major web property such as google, facebook, credit card issuers, etc. should buy up all the most likely typo-ed versions of their domains so that this doesn't happen.

The average user is going to blame Facebook anyway if they get duped into logging in at faccebook.com, and it goes to protecting their brand (trademarks are weakened if the owner doesn't defend them).

They do, and if they don't, they end up like the White House a few years ago when whitehouse.<I forgot the TLD> was bought and an almost exact replica was created, according to Rule 34.

My point is that you can buy any site from a CA and no verification is done, exactly as with Let's Encrypt.

Even with an EV there is no guarantee that the requester is genuine ("best effort" to check)

With EV the checks basically care about:

1. Does this entity exist, e.g. it's in an official government business directory and the address details match what was specified 2. Use a third party directory (e.g. Dunn & Bradstreet) to look up this entity, and phone them. Ask to talk to some specific role at the company e.g. "Head of the IT department" and then confirm the details with that person.

This involves actual humans, albeit basically call centre employees, looking at paperwork, making telephone calls, that sort of thing. I guess you could label it "best effort" but to me that signifies much less.

For Americans a surprise problem with EV is that your country doesn't _have_ a central business directory. Each State runs its own directory, most businesses are registered in Delaware, regardless of their actual home state.

https://stripe.ian.sh/ has a completely genuine Stripe, Inc. EV certificate, issued to a Stripe, Inc. just not the one that's famous.

I can get a prepaid VISA card at any convenience store

With no billing address, though, right?

So, Verisign is an interesting example.

Verisign sold their certificate business a long time ago to Symantec.

Symantec had a contract with a Korean company "CrossCert" in which Symantec would issue certificates (including "Verisign" certificates) based on CrossCert's validation.

We don't actually know what CrossCert's validation practices were, because during investigation of problematic certificates Symantec found that CrossCert wouldn't give them the paperwork they should have been keeping to track this work so that it could see what they'd done. It had never asked to see that paperwork previously (over years of the relationship) and had simply accepted a Korean auditor's finding that everything was fine without really inspecting anything itself or letting anybody know about this arrangement with CrossCert.

Did you know about? When you see "Verisign" do you think "Or, you know, maybe some Korean company, but that's fine?"

Never mind, this scandal eventually resulted in Symantec exiting the CA business late in 2017, and today "Verisign" certificates are issued by DigiCert, another company you've probably never heard of.

For any medium or large company, the $200-ish cost of a commercial wildcard certificate is a non problem. I think let's encrypt is mostly useful for small businesses and individuals, who can live with these limits.

I think it's not the cost the issue, but the peace of mind brought by the automatisation of the process of certificate renewal.

This where LE is for me different from usual cert providers.

True but you can buy a 7y commercial wildcard cert. That also offers some peace of mind. You still have to worry about renewing it in many years but on the other side you are not exposed to a job breaking or an api changing.

Nope.

Currently (today, 21 February 2018) the limit is 39 months, starting 1 March 2018 the limit becomes 825 days. These limits are set down in the Baseline Requirements, which are rules agreed between the Certificate Authorities and Browser vendors (in practice mostly OS vendors) at their standing meeting. Chrome at least actually implements these in software, ie leaf certificates which seem to have a longer lifetime are simply invalid and don't work.

In practice commercial CAs tend to sell products as 1 year, 2 year or 3 year, and then they use the extra days in the BRs to let you "keep" the extra days if you renew early, so that you don't feel the need to wait until the last minute to renew to get maximum value.

If you're in business, you need a process so that you are not likely to forget dates. Tax filing deadlines, other regulatory reporting, etc. Certificate expiration can just be part of that.

It's not really that hard. I manage a dozen+ certs manually, I get reminder emails 30 days before they expire, and it takes literally a few minutes to submit a new CSR. I could autmate it even more than I do, but https://xkcd.com/1205/

Me too, the rate limit is preventing me to keep on using LE.

this is the second delay

Am I the only one who thinks the way we deal with certificates is just completely backwards? Why should there even be a CA at all for DV certs? We already have to trust the registrar as ultimately they can get whatever DV certs they want so why not just limit our trust to them instead of adding hundreds of additional organizations that can compromise our security? And we pay them for the privilege of doing this??

I get CAs for EV and OV certs, of course you need an additional trusted party there, but the vast majority of websites out there do not use EV or OV certs and to be fair users by and large don't even notice the difference between a DV and an EV certificate in the first place.

We already have DNS, we already trust DNS to issue certificates to people, why don't we just cut out the middle man since they serve no additional purpose?

This is the idea behind DANE :)

https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Na...

Effectively, you’d serve a self-signed certificate, then use DNSSEC to state that the origin can use the self-signed cert. Then clients can use that without showing warnings.

However, it does rely on DNSSEC not being a total failure, so not likely to end up in widespread use anytime soon unfortunately.

Also, given the more recent growth of alternative solutions to the problems DANE solves, like Let’s Encrypt and CAA DNS records, I’m not sure we’re likely to see much more work being done on it sadly.

The thing is that "control of the DNS" can mean different things.

I have control of the DNS on my network, and at the local coffee shop. I can use that to attack users, but I cannot use that to obtain a certificate.

I still agree with you in a sense, an alternate solution would be great. But we that middleman is the best we have right now.

> The thing is that "control of the DNS" can mean different things.

In every way this discussion is related, "control of the DNS" should mean control of the one authoritative DNS source, in a way which means that if a service across the internet requests a DNS record, it won't be affected by whatever DNS-trick your neighbourhood cafe's wifi may apply.

Trying to muddle the discussion of a fairly straight-forward, reliable scenario with consumer-space DNS resolution is IMO not very productive.

But that's the point - the CA is needed to make a reliable verification of the data coming from the authoritative DNS source, since the consumer can't make that check themselves in an untrusted network.

The discussion directly relates to a user typing google.com and knowing, via https, that they are talking to Google.

The CA preventing the local DNS trick is entirely relevant.

So if I fix my DNS to 8.8.8.8 (or other well-known public DNS) does that not largely squash the coffee-shop attack vector?

I could spoof the gateway and forward 8.8.8.8 to my own machine with my own rules.

I mean strictly in the sense that we already have root servers pointing to TLDs pointing to individual domains all the way down. I'm not suggesting DANE in particular here I'm suggesting trust one "Super CA" that signs a "sub CA" for a TLD that signs another sub CA for the domain which can keep doing the same thing. Basically just chain each CA together but all of the sub CAs would be further and further constrained such that .ly couldn't sign certs for .com and ycombinator.com could only sign certs for subdomains of ycombinator.com.

In essence, we already trust the registrars to link a name to an entity, DV certs are just going to defer the judgement to the registrar so why bother trusting an additional entity to issue certs for every domain instead of just having the registrar sign your CA that can only issue for your domain and only until your domain expires.

Right now AFAIK I can technically get a SSL cert for a domain I control and have it expire well past when the domain expired and was transferred to another party. I now have a valid certificate for a domain that I don't own.

> but the vast majority of websites out there do not use EV or OV certs and to be fair users by and large don't even notice the difference between a DV and an EV certificate in the first place.

You could also say the same for HTTP vs HTTPS (except HTTPS being considerably slower on unreliable networks).

So what's the value add in forcing everyone (even those who don't want to and have no need) to use HTTPS?

The "no need" case is pretty thin.

People seriously underestimate how little you are promised in the plaintext HTTP scenario over a potentially hostile network (ie almost any network)

Everything you say, or the site says, can be read and changed by any third party between you on the network and there'd be no sign they did so, no trace left.

That's almost never what you actually wanted, so, why should we keep allowing indeed arguably encouraging this practice?

> Am I the only one who thinks the way we deal with certificates is just completely backwards?

No. https://www.tedunangst.com/flak/post/moving-to-https

I've been using LE from the start and it's been awesome (especially with clients like acme-tiny), but wildcards changes everything.

How many of you are really going to set up a custom DNS updating script?

Maybe I misread something but I remember hearing wildcard certs will only work with DNS based challenges.

Wouldn't that mean the verification tool would need to support every single registrar's / DNS provider's API?

For example, I host my domains (and DNS) with namesilo.com. Is a challenge script really going to know how to interface with their API to add / remove the TXT records?

What about the 50 other popular DNS providers?

pfSense has a dynamic DNS updater built in with support for a huge number of services - https://doc.pfsense.org/index.php/Dynamic_DNS . There's also an ACME client and HAProxy packages and a built in CA and cert handler.

Its all open source, so grab their scripts ...

If only there was an existing standardized way to transfer zone updates to an existing DNS server...

Yes, but there's already scripts handling many of the most popular.

Check out acme.sh and remember, changing your DNS provider or hosting it yourself really isn't that bad.

https://github.com/Neilpang/acme.sh

Nice, it looks like they support namesilo and ~40 others.

So my suspicion seems accurate. The script will need to know how to interface with each provider and you (as an end user) will also need to be responsible for setting up your own API keys for whatever provider you use.

If you're up for the task, I've wanted to write a script (POSIX /bin/sh certainly) for interaction with DNS providers. Basically lower the entry barrier, and avoid the web interface:

    $ foo mydomain.org --init ovh
    $ foo mydomain.org my.sub A 1.2.3.4
    $ foo mydomain.org my.sub AAAA 1234::abcd
    $ foo mydomain.org --remove my.sub A [1.2.3.4]

etc.

I have yet to find anything that handles any DNS changes in a simple, stupid way. Most API clients I've seen only handle one kind of request: acme.sh only does TXT challenges. DynDNS endpoint only does A records. Etc.

It'd be cool to have and it's relatively straightforward from a software engineering perspective, but it's so much boring grunt work that finding someone who wants to implement it might be challenging.

If this is something you need to do a regular basis, you might consider hosting your own DNS and using BIND's nsupdate tool.

> hosting your own DNS

meh. There are some things that I'd gladly host, but DNS clerly isn't among them. Too many ways to screw up, probable breakage ahead (only 1 dedicated @ OVH), so many rate-limits to implement; DNS servers a re filled with RCEs... (bind being the worst).

Clearly, nope. Not doing this ;)

Crap. I had certs set to expire on 2/28! I was going G to be saved by wildcard. Off to make a bunch of DNS entries....

That's ok you still have almost 26 months ;-p

slow clap :D

Does your ACME client already support ACMEv2 and wildcards? I think only acme.sh has so far claimed to (happy to be corrected if another client has finished its implementation).

I use certbot. I'm not sure as I was just planning on figuring it out for the night of expiration.

I'm part of the Certbot team and I can tell you that the ACMEv2 support is still being implemented by my colleagues now; even if it had been included in a release this week or next, you would have been cutting it pretty close! Many of our users don't have the most current release because distribution packagers don't ship it right away (or even for months or years, depending on circumstances).

Certbot's own autorenewals (which I originally implemented and which I hope are currently working for you -- I'm happy to try to find a way to get them to work in your deployment if they're not!) try by default to renew certificates 30 days before expiry in order to give people a lot of time to notice and intervene if something goes wrong. Consistent with that, I'd encourage all of our users not to rely on anything happening less than 30 days before expiry as part of their plan for keeping certificates current on their sites.

Thank you for all your work. My renewal went through fine. I had commented it out of crontab, but went fine once I ran it. I was pleasantly surprised when tls-sni worked flawlessly.

Great!

One thing to know about the TLS-SNI situation is that TLS-SNI-01 has been disabled for new issuance and only works for renewals. The selection of authentication methods is pretty transparent because the ACME protocol allows the CA to state which methods it will accept for a particular domain authorization (so for the renewals it can state that TLS-SNI-01 is permitted, while for new issuance it can state that it's not, and a client can respond accordingly).

Apparently the ACME WG is working on a TLS-SNI-03 method that will be safer and that might be supported by Let's Encrypt (and ACME clients) at some point. But it's also possible that TLS-SNI-01 could be disabled in the future, even for renewals, even before TLS-SNI-03. So, if you happen to have anything in your setup that prevents the use of HTTP-01 on port 80 for authentication, you should be aware that renewals could potentially stop working at some point (although this isn't imminent), and that you might also have difficulty adding new domain names to your certificates.

https://community.letsencrypt.org/t/important-what-you-need-...

This unfortunate situation is a nice showcase of the ACME protocol's flexibility, because the CA has a way to tell the client very specifically what authentication methods to attempt or not to attempt for each domain (and if new methods are specified in the future, they can be rolled out incrementally without breaking compatibility).

One thing that'd be good here for outside observers is for Let's Encrypt to bake a list of the Blessed Methods used into certificates as OIDs in the Policy section.

Right now a researcher with 5 million Let's Encrypt certificates can see which ones use public keys with unusual patterns, how many are for names in the French .fr TLD, how many have longer key lengths, and so on, but they can't see whether the validation was done with Method 10 (which had this unexpected flaw) or some other method.

Another CA (can't remember which one) was looking at doing this, and it seems like it'd be nice for Let's Encrypt too.

I like this kind of thing in principle, but certificates are already getting kind of large and there are a number of other things that one might put in them via OIDs. But I realize ASN.1 is pretty concise; do you know about how much this proposal would increase the size of a certificate by?

Another concern might be that users might not want this to be disclosed because it might give attackers suggestions about how to attack their renewal processes (although of course CAs have been willing to publish information, like issued certs, that some users wouldn't prefer to make public).

It’s not super hard to implement yourself, particularly given it is a variation of the first ACME protocol for which there is lots of code available, and the staging API for v2 is already up and running.

The main challenge is that wildcards require DNS authorization, so a full automation is DNS api specific. My DNS (OVH) has a neat rest API. But given the variety of DNS providers and APIs it is going to be challenging to rely only on a reference implementation without wrtiting a little bit of code yourself.

It's not mentioned, but I'm assuming this feature with the same original schedule:

Embed SCT receipts in certificates

Is also delayed? I think this has been quite underrated.

I've been following progress in their GitHub issue for it: https://github.com/letsencrypt/boulder/issues/2244

Most recent comment:

> We're working hard on it, but will probably not land this in production before the end of February. Still, we are very much aware of the upcoming deadline and committed to meeting it. Thanks for the enthusiasm everyone! :-)

Reply on news.ycombinator.com