How We Deliver Global SSL with Let's Encrypt | Hacker News Reader

How We Deliver Global SSL with Let's Encrypt

fly.io

177 pointsjnunemaker65 comments

Would greatly benefit from explaining why they couldn't do dns-01 challenges successfully.

This an issue all the way from individuals with crappy or misconfigured DNS locally to whole TLDs (fortunately no really major ones) with non-working DNS. But it's often a teachable moment, something others can learn from.

I'd like to see that too, especially since I've been doing fully-automated Let's Encrypt with dns verification for hosted domains for well over a year now. I might be able to offer a solution to whatever issue they've had.

would you consider writing up your system somewhere?

DNS challenges weren't broken, they technically worked just fine. But they added an additional step to our setup process, and we were losing customers during signup because of it. We can still do the DNS challenge it's just not how we guide people through our setup.

Basically with the DNS challenge, we had to send people away to make DNS changes, have them come back and confirm they were approved, and then we sent them away _again_ to make another DNS change. It was kind of brutal.

(edit: we also tweaked the article to not imply "broken"!)

So typical user engagement/onboarding complexity funnel issues not something inherent to the DNS/LE/Fly.

Why would you need to send them away twice for DNS changes?!

I'm assuming mrkurt meant that you send your users once to create a dns record (in case you haven't already) - you can add any hostname you want on Fly, it doesn't have to exist, yet. Then you go and create a DNS challenge for Let's Encrypt. Obviously, these 2 steps are orthogonal, but this is the reality of user onboarding.

But why would that need to be in two steps?

Well, it's two DNS entries. You could do them both at the same time, but people were getting that TXT record wrong pretty frequently, which would have meant https connections getting an invalid certificate had they changed their actual hostname at the same time.

Since we control the http response once DNS changes, we don't have that problem. And it's simpler for people to create CNAME/ALIAS records.

Erm ... you had people create TXT records? Why would you do that? Wouldn't people then have to manually update the TXT record on each certificate renew?!

Why not have them create a DNAME, a delegation, or just two CNAMEs?

txt records are the only way to do the dns-01 challenge with certbot/Let's Encrypt.

The http-01 challenge is simpler, we can get people setup with one CNAME/A-record.

Once we're serving traffic, we can do all renewals with an http challenge and they don't need to change DNS ever again.

This is an often overlooked option, but you can indeed solve the DNS challenge by having someone create a CNAME record for _acme-challenge.example.com that points to a domain under your control, and then serve the TXT record needed to verify the challenge from that domain. acme-dns[1] provides a nice implementation of this idea, as well as a more detailed explanation.

[1]: https://github.com/joohoi/acme-dns

Whoa really? I didn't know this option existed. This would make these types of configurations a lot better.

Sorry, not trying to mock you, but I'm really curious how you (or anyone else on this thread who seems to be surprised by the possibility) managed to not know.

I mean, assuming you do know that CNAMEs are a thing, this reads to me like "well, yes, HTTP can do redirects, but I didn't know you could also redirect requests for images!" ...

I guess what I am wondering is: What is/was your mental model of CNAMEs that they could redirect DNS resolution, but then it's somehow surprising that they can redirect DNS resolution!?!

I don't think we'd considered a CNAME to redirect the specific txt records that Lets Encrypt requires. It's not intuitively obvious that they support redirects for those lookups.

But how is it not?! I mean, apart from the fact that you never know whether software is buggy, of course.

CNAMEs are not a feature of application software, but rather a mandatory part of the DNS specification that any standards-compliant resolver has to implement. As they claim to be using DNS, that obviously implies the claim that they implement what the DNS specification defines, and that would necessarily include following CNAMEs.

Part of it was that I'm not very familiar with the internals of the challenges and I didn't know that `_acme-challenge` was the record name for the dns-01 challenge. (I probably could have put 2+2 together if I realized this much.)

Another part was that I've never read the dns rfc and honestly I've only ever spent enough time messing with dns until it worked. And I've never seen a CNAME for a record that's resolves to anything but A (or AAAA or another CNAME). So I suppose it was a combination of a failure of imagination, lack of knowledge, and lack of experience. Or maybe, "I dunno I never thought about it".

But now I know! CNAMEs work for all records regardless of type, forwarding the entire lookup elsewhere, which I guess is why they require no other records exist with that name. Pretty cool.

I suppose I could have learned that with a bit less incredulity but hey, it got me here in the end. I guess a link to the RFC might be relevant here [CNAME 1]. I'll probably look through it later, maybe there's some other cool features I didn't know about.

[1]: https://tools.ietf.org/html/rfc1034

You could also have them delegate the _acme-challenge with an NS record

And A and AAAA records are the only way to do HTTP with web browsers ... so?!

What makes you think that CNAME or DNAME records are specific to A or AAAA records?!

I'm really struggling to understand what you're recommending. Will you explain how a DNAME works with the Lets Encrypt auth process in a way that makes it as simple as a single CNAME/a-record?

Well, you have them create a DNAME that maps theirhostname.theirdomain to yourhostidorsomething.yourdomain, and then you have in your nameserver A and AAAA records for yourhostidorsomething.yourdomain and a TXT record for _acme-challenge.yourhostidorsomething.yourdomain!?

(Obviously doesn't work if they are to keep control over other subdomains of theirhostname.theirdomain, and also, I haven't tried it with DNAMEs, so I don't know whether their implementation is buggy.)

I am guessing first time is for verification and second is for actual configuration.

Also my experience with DNS validation and let's encrypt is that propagation times can be variable. You do not know from where in the world the let's encrypt servers will contact your DNS, so you may see the new entries in your DNS, but it may not have been propagated yet to where let's encrypt query them. And then the validation failed and you need to start again from scratch. So you need to introduce some large delays to be safe.

Having DNS validation as an option is very useful, as not all certificates are used for http servers (think smtp) but not trivial to implement.

Also wildcard certs will require DNS validation.

Interesting product - fly.io

It’s essentially a reverse proxy as a service. Sort of like Cloudflare but with what looks like a nicer API, and custom domain name SSL eg. for multi tenant apps.

It was a bit pricy when I plugged in how many requests we get per month on our servers, easily in 5-figure per month. Having said that, internally we built most of their selling points already.

I agree with the pricy bit. I've made a calculation, and it would cost me 1200$ a month to run what Cloudflare does for free.

Now, I get that free is not an option. 50/100$ a month is probably OK. But that's beyond reasonable. And prices change depending on where people visit you from (!?)

I'm not sure what kind of app you're running, but our product/pricing are targeted almost entirely at SaaS apps that need multiple hostnames — which is $2500/mo min on CF: https://fly.io/mix/custom-hostnames

It's really expensive to do that on top of CloudFlare, and most apps like that aren't using a tremendous amount of bandwidth.

We're not _yet_ a great option as a general purpose CDN replacement (both pricing and featureset) but we're getting there, and are pushing prices down continuously. It's more expensive to buy bandwidth and power in certain cities, too, hence the different pricing per region.

If you're willing, I'd love to know what your traffic breakdown is and how you got to that $1200/mo number. You can email me if you'd like! (kurt.mackey@fly.io)

Agree it is not a trivial task to build out the infrastructure for custom SSL. We’ve done it ourselves and it’s just now another service component we have to manage. ie. more time sink for something that isn’t our business core feature

For others who are reading this, the whole process involves reading and storing private keys using asymmetric ciphers and all that. Definitely more stuff than we’d like to be responsible for.

Will definitely keep an eye on fly.io for new features! Thanks

We do custom domains for customers with SSL through netlify - though only an option if you have a static site.

Hey, I work at Fly, too. We have a built-in Netlify integration... If you configure your domain with Fly, you can use it as a custom domain for a Netlify, but also serve any other content on arbitrary sub paths. Here's more about the Netlify integration :) https://fly.io/articles/fly-netlify-explosive-static/

Custom domain name SSL is also provided by Cloudflare, it's called "SSL for SaaS".

I’m almost certain I heard it was only available on their enterprise plan so you’re looking at a couple grand plus per month minimum

Custom domains for multi tenant apps is our biggest value right now. You can get it elsewhere, but there's typically a huge fixed monthly cost just to start.

There are a bunch of companies who've done most of what we've exposed in house (us included in past lives!), once we're a bit older and more fully developed I think we'll have compelling features even for you though! We do have deep volume discounts, fwiw, so if you get tired of running all that stuff let us know. :)

Our API for doing this is quite simple:

  POST https://api.cloudflare.com/client/v4/zones/:zone_id/custom_hostnames
  {
    "hostname":"app.example.com",
    "ssl": {
      "method":"http",
      "type":"dv"
    }
  }

In addition to "http" (described in this thread) we also support "cname" and "email" methods for validation. Some of our customers prefer these other methods as it lets us issue the certificates before /their/ customer points the CNAME over.

Once we receive the call we generate multiple certificates for each hostname: one SHA-2/ECDSA that's presented to modern browsers and one SHA-2/RSA that we serve to legacy ones that don't support ECC. (Let's Encrypt doesn't do SHA-2/ECDSA signing yet but plans to sometime next year.)

We also support uploading your own custom certificates for those that, e.g. see value in Extended Validation (EV):

  POST https://api.cloudflare.com/client/v4/zones/:zone_id/custom_hostnames
  {
    "hostname":"app.example.com",
    "ssl": { 
      "custom_certificate": "$MYCERT",
      "custom_key": "$MYKEY"
    }
  }

Where $MYCERT and $MYKEY are the PEM payloads with newlines removed.

You didn't mention price. :)

LE is awesome.

Clients want an SSL server and it lets me set it up in 10 minutes without additional costs.

Yes this is a delight compared to the old way of having to hand over $10 or whatever to someone for a certificate that one had to remember to renew in a timely fashion versus just using a cron job now with LE.

Right?! "Sure I won't think about this for a year, then I'll remember it 30 days before I need to have it fixed, it'll be great".

Or "I won't have to think about this for three years, and everything will keep working during that time, and I don't have to worry about a cron job failing"

I don't mind offering 90 day certificates, but requiring it goes against the offered reason of LE to expand HTTPS.

Why use a cron job? I have certbot running in one Docker container, the .well-known route directing to the one dedicated Docker host running certbot, and whenever certbot renews the certificate, it writes the cert to the NAS share that is only mounted on the LBs and then triggers a SIGKILL to the haproxy docker containers (which causes haproxy to re-read the cert file and configuration).

Fully automated, no cronjobs to fail. In addition, I have a monitoring on all LBs that verifies the expiration time - it has never hit so far. That cronjob/monitoring is something everyone should have anyway because a well working monitoring system can alert you to so many other possible failure states...

You just described a far more complicated setup then a simple cron job.

Thank you for explaining the details of an alternative method. I have not advanced that far in my deployment skills, yet!

Well, depending on how you install certbot, it sets up a cron job automatically for you (on Debian, anyway). It's no big deal - it just creates the cron file /etc/cron.d/certbot.

I'm curious as to how you get certbot to renew automatically without a cron job or some sort of script in the Docker container that periodically runs 'certbot renew'.

> I have certbot running in one Docker container, the .well-known route directing to the one dedicated Docker host running certbot, and whenever certbot renews the certificate, it writes the cert to the NAS share that is only mounted on the LBs and then triggers a SIGKILL to the haproxy docker containers (which causes haproxy to re-read the cert file and configuration).

Is this really more stable than a simple cronjob? It is far more complicated, for sure, so there should be significant benefits to make up for that.

On a side-note, ManageEngine Key Manager Plus can automate certificate management (request, acquire, deploy, track and renew) for public facing websites.

Video link: https://www.youtube.com/watch?v=oYelZided-E https://www.manageengine.com/key-manager/

Disclaimer: *I work for ManageEngine

It’s a x.509 certificate, not ssl. Also, the ssl protocol has been deprecated in favour of tls.

I’ll let tls certificate slide but ssl certificate just triggers me too much.

What's the point of this nitpicking?

The title is perfectly fine for its purpose.

First, thanks to the terminology used by almost all commercial CAs, "SSL Certificates" is still the most commonly known term for that. Like it or not, but if you want to reach a wide audience, you need to say SSL in addition to TLS.

Second, even fewer people know the term X.509, insisting on that is like insisting on saying RFC-7540 instead of HTTP/2.

Third, in the very first sentence the article sets everything straight:

> We've been hard at work making Let's Encrypt TLS certificates as simple and safe as possible for developers and creators of all kinds

So ... is this an instance of commenting before actually having had just a tiny look at the article?

Because it is important that everyone understand that SSL is dead and TLS is the replacement. SSL/TLS is forgivable, saying SSL is sort of like writing a headline “How this company uses VHS” when you actually meant Betamax. It’s misleading.

I still don't get it. This is clarified in the very first sentence of TFA. Why do you think it needs to be stuffed into the title?

We wouldn't need to have this conversation if the title said TLS.

You're correct, but you're fighting an uphill battle: https://certsimple.com/blog/ssl-or-tls

When talking to management, I try to avoid the acronyms (SSL, TLS, HTTPS, etc.), and in this case, I've coined a phrase that I find be equally understood by management, devops, and sysadmins alike. I tell them that we need to "encrypt server X with a secure certificate" (or "secure cert" depending who I'm talking to, which is as short as the shortest acronym syllable-wise). It's vague enough to cover the spectrum of acronyms in a timeless (for the foreseeable future) way, but unambiguous enough that everyone understands.

Sorry a little off topic, but any idea when Let's Encrypt will (if ever) support wildcard SSL certs? We provision (https://commando.io) a subdomain for each of our accounts (thus thousands of subdomains) using a wildcard DNS A record. Willing to pay, as long as it is less than the $99 a year currently paying through NameCheap.

LE still says Jan '18 - https://letsencrypt.org/upcoming-features/

for the time being, you could try to issue SAN certs with LE (from their website):

"If you have a lot of subdomains, you may want to combine them into a single certificate, up to a limit of 100 Names per Certificate. Combined with the above limit, that means you can issue certificates containing up to 2,000 unique subdomains per week."

We can handle wildcards now, we just issue new certs when we see new subdomains (we have a 2500/wk rate limit from LE and will implement SANs if you need them). Feel free to email I'd you want to set it up. :)

Interesting, and you support DNS A record wildcard? Currently have around 6,300 subdomains.

Yeah A/CNAME to us and then you can use our API to issue certs of all subdomains or if you have a wildcard cert already you can upload that with us. Either way works :)

You could actually install the wildcard cert, then we can slowly backfill with LE certs. And when LE does wildcards we'll just switch to one of those.

Completely unrelated, what is the blog framework / cms you are using the the fly.io blog? It is amazing!

Thanks! We're using ghost.org and of course all our illustrations are done by Annie (https://twitter.com/annieruygt) :)

Oh, and we use the basic Ghost theme, Casper, with some design tweaks.

Thanks for the insight! Really appreciate it!

Clearalias.com is another service that provides SSL multi-tenant apps and SaaSes that want to provide secured custom domains for their customers.

There's only one step required by the customer which is to point their DNS to Clearalias, no API calls or extra set up by the app or the customers.

Ohai! Yeah, you can do the same thing with Fly (which is what most of our customers do - just point DNS at Fly). I cannot stress enough how nice it is being able to build on top of Let's Encrypt! bows

The API is mainly for people who'd like to automate the process, because they want to provide custom domains for their customers (so more for B2B use-cases) ;)

The grandparent post means that the customer hostnsmes also work without any API calls if configured properly.

Reply on news.ycombinator.com