Let's Automate Let's Encrypt
linuxjournal.com
linuxjournal.com
Put "certbot-auto renew --quiet --no-self-upgrade" in crontab and it does its own thing.
(sidenote: Would be interested to hear if there's actually any security issues with putting the certbot client on your machine)
Previously I had a monthly crontab script that updates my LE cert, but now it seems that certbot has been backported to my server's OS (Debian Jessie) which makes my script redundant.
I'll be glad to retire my script. :)
sudo service nginx start
And nginx gets/renews certificates automatically.
It's not unobtainable to make using certs MUCH easier and automated, their work should be supported by us all.
Caddy is mostly just unproven at high scale; theoretically, nginx should be able to handle greater loads, but I wonder what Caddy's limits are and how big the difference would be. Haven't found time to benchmark them head-to-head.
I think we're all just waiting for people to really test its performance under heavy loads and give it a chance to prove itself (or not).
"sudo service nginx certify" would be nice though.
What if the cert doesn't expire in a month -- but only, say, 7 days? 7 minutes? Where do you draw the line? Would you rather have your site go red in browsers or just not have the server start? (Remember, it wasn't started already anyway so it's not like your sites are incurring new downtime.)
nginx -t && sudo service nginx reloaddisclosure: I work with OCCRP and have submitted pull requests on this.
-> We do DNS verification through Dehydrated (https://github.com/lukas2511/dehydrated). Because we use in-house DNS running PowerDNS with a MySQL backend, updates to records are a cinch.
-> Dehydrated has "hooks" that allows you to call your own scripts for each step of the verification process. We have an on-prem sysadmin box that does remote administration of the servers (so that none of our servers have direct access to any other server, just in case of a breach), so we just had to update a few commands to work with Dehydrated's hooks.
-> Each remote server has its own directory within the Dehydrated environment, and there's an "update" script that runs from cron that: a) asks that particular server which hostnames it currently has, b) updates the hostname list in that server's Dehydrated directory, and c) invokes Dehydrated.
Because we host some client sites, it felt skeezy to me to be updating files in their web directories on a regular basis even if it's really temporary. DNS updates however are just fine.
All in all Letsencrypt has been a huge, huge win for us. I'm really grateful for both them and for Dehydrated.
1) Why did it take so long for an open source version to come into existence ? (there must have been some serious hurtles Let's Encrypted had to overcome to do this)
2) Given that you can get LE certs for free and use certbot for auto-renewal, why is anyone purchasing certs anymore ?
0. The cost of certificates is pretty insignificant to most corps.
1. They like to have someone to shout at when cert issuance fails. They happily pay money for this ( see point 0 ).
2. LE's cert renewal policies aren't compatible with most corporate change-management processes.
Having someone assigned to perform rolling cert changes every 60 days on every server just isn't feasible or desirable.
When I worked at "Fortune 50 Corp", an annual ( or even biannual / triannual ) planned cert update was a big deal and we had to stage the change across groups of servers in accordance with an algorithm that ensured that if they all inadvertently screwed-up that there enough servers with the old certs to keep production and pre-pro running.
3. Customers, particularly of financial insitutions, like to see the reassuring green EV indicator in their browser.
Let's Encrypt only offers Domain Validation certificates, not Organization Validation or Extended Validation (green bar) certificates [1]. The certificates themselves are only valid for 90 days, which might pose a problem for organizations where their infrastructure makes changing certificates difficult or time consuming. LE does not offer wildcard certificates [1], and there is a rate limit of 20 certificates per week per registered domain [2], so organizations with too many domains won't be able to use LE for all their certs.
[1]: https://letsencrypt.org/docs/faq/ [2]: https://letsencrypt.org/docs/rate-limits/
Open source tools to run a CA / PKI have existed for a good long time, they just haven't been easy to use.
Automated certificate issuance software has also existed for awhile, but it hasn't been terribly awesome.
The bigger issue is trust - launching a new CA is quite expensive since you need to invest and build all the technology and processes, then get them all audited, and then present the results to the various organizations who control who control which CAs are trusted (see https://cabforum.org/)
The motivation for running a CA has typically been financial - even vendors that provide free certificates (StartSSL, Comodo, etc) have done so as path to moving customers to paid services.
Lets Encrypt/ISRG was kicked off by a group of folks who care deeply about security and had the technical chops and inside knowledge (some (all?) of the founding team) were from Mozilla and had both guidance and experience dealing with the CAB.
In short, there is no good reason not to have done this, and independent of this, another team within Mozilla had started to think about how we could tackle this before we found out the LE work was in flight :)
To slightly expand, getting trusted by the key trust stores (laughably pretending to each represent a "Web browser" at CA/B although nobody would mistake Apple or Microsoft for mere web browser vendors) takes far, far too long to be practical as a first step. Mozilla was fastest and they quote 12-24 months typically, whereas Apple and Oracle are both famously slow to react, and might not sign off on Let's Encrypt for years yet to come.
Well, Let's Encrypt works today, so how did they do that? There's a way to sidestep the trust stores. An existing trusted Certificate Authority can sign a certificate which instead of saying "This is a web server" says basically "This is another Certificate Authority". Let's Encrypt got theirs signed by Identrust. This is called "cross signing". But of course to get an existing CA to sign your CA certificate they need to be sure you're not going to screw up, because it's their reputation on the line as well as yours. So although this is faster than talking to all the CA/B "browser" members for years, it still needed a lot of hard work by ISRG's team.
I recommend the nginx as ingress backend ( https://github.com/jetstack/kube-lego/tree/master/examples/n... ), as GCE had problems talking over https. Also wait for a bit (e.g. 30 minutes) after updating your domain records and enabling kube-lego (i.e. just before this section: https://github.com/jetstack/kube-lego/tree/master/examples/n... ).
Alas, I don't really want to use the official client. It seems messy and complex, and docker kind of contributes to that impression. I wish there was a tool that was simple, clean and elegant.
I tend to recommend lego[1] to anyone needing a high-quality ACME client without certbot's complex auto-configuration features. Might be what you're looking for. It's written in go and can be installed by fetching a single binary.
Ten-ply-crest can run as a standalone microservice or on top of any existing express app. It supports pluggable backends for storage, and doesn't rely on the local filesystem.
It's really short on docs, but "release early and often!". It has been running all my sites since the beginning of the year, Feel free to open issues with any questions and I'll help you get running.
If you need a complete client lib for the LetsEncrypt API in JS, that's included too [1].
[0]: https://github.com/nextorigin/ten-ply-crest
[1]: https://github.com/nextorigin/ten-ply-crest/blob/master/src/...
This caused Let's Encrypt to fail.
I worked around this by temporarily allowing outbound connections, but I wonder whether this needs to be fixed more generally? It seems reasonable to me to block all outbound connections on a web server that isn't expected to need any. But Let's Encrypt's (perfectly reasonable) need to renew certificates breaks this assumption. Unfortunately, it doesn't seem easy to set up an exception on an IP-based firewall to allow Let's Encrypt outbound connections only. This makes automated renewals hard.
[1] - https://www.frozentux.net/iptables-tutorial/iptables-tutoria...
It's a good question. I'm not sure. It is, however, a caveat of the Let's Encrypt experience for me - whether or not it is possible to fix.
> You need to make the renewal request somehow, and that means there has to be an outbound request at some point.
Agreed. It would be nice if there were some way to set up an (external) IP-based firewall such that the Let's Encrypt API calls can go through, but nothing else.
One way might be to declare publish destination IPs as static (rather than dynamically through DNS), but of course this would be rather brittle for them from a service perspective.
Another way might be to get a port number assigned for the ACME protocol, rather than overloading HTTPS. ACME services could operate on both an official ACME port number as well as the HTTPS port for backwards compatibility. The protocol could remain HTTPS-based. Then firewalls could explicitly allow ACME through, while blocking outbound HTTPS for anything else that may be an attempt to exploit a general HTTPS server rather than being known to be ACME-specific.
This is what I'd prefer, I think. The thing is, part of the appeal of Let's Encrypt is the readily available ACME client that JFDI. Such an easy-to-use client doesn't currently exist for this use case.
However I dont think this will work for my setup. I run Arch Linux on a Raspberry PI, which serves OwnCloud via Apache. I have always had to go the manual route with lets encrypt because I dont run Debian.
The problem is these docker images are probably built with ubuntu or debian and assume an intel architecture. So I would have to recreate these images for arm - which means I would have to start with a different image.
Not hard to build myself, but I would basically need the original Dockerfile and then recompile from there.
[0] https://www.digitalocean.com/community/tutorials/how-to-secu...
Maybe still a good idea to sweat the permissions - the way it's set up right now, any process on the system can delete the certificates from this folder, thus breaking your site until such time as you renew them.
Reminds me of the "fair dice": https://xkcd.com/221/
The hack you mention for cron doesn't seem particularly bad though.
The key is doing a dns-01 challenge, instead of http-01, so you don't have to put challenge-response garbage in your repo.