A Unix shell script implementing ACME client protocol | Hacker News Reader

A Unix shell script implementing ACME client protocol

github.com

125 pointsausjke52 comments

Launching those binaries (openssl, curl) seems like a point of failure, not to mention the cough unique challenges of doing simple things like managing an array of strings in a language like bash.

My idea of the simplest letsencrypt client is lego. One binary to rule them all.

https://github.com/xenolf/lego

At the top of the readme

> This is a work in progress. Please do NOT run this on a production server and please report any bugs you find!

And it's been in development for at least two years per the dates on the files - with hundreds of commits made at a reasonably continuous pace judging by the code frequency graph.

Either that's a ridiculously conservative warning, or I think there is a problem with something in this approach.

It's a rediculously conservative warning. :)

Lego + NGINX + Kubernetes works very well for most projects.

"One binary to rule them all."

But if one is using a shell to launch that binary, then IMO there are two userland binaries: the shell and the binary being launched. In that case, the second binary depends on the first.

Lets assume one really wants to use only a single binary for whatever reason. Could she use busybox? Does busybox have an openssl-like function? I am not sure.

However I can confirm that the BSD equivalent of busybox can easily be compiled, statically, to include an http client, openssl and the other utilities needed for these shell scripts. I use such a binary for daily work.

Note I am not a LetsEncrypt user and have no comment on ACME or these shell scripts or other programs. I am only commenting as an avid shell scripter and user of static, "multi-call" binaries.

I use a shell script ACME client on FreeBSD (called letsencrypt.sh; different from the one linked in this submission and is available in FreeBSD's repos) and have been for a couple of years now. It's worked flawlessly in that time and was an absolute doddle to use. In fact easier than the other ACME clients in FreeBSD's repos at that time.

I do hear your criticism of shell scripting - I've written enough things in Bash and Bourne to be intimately familiar with the pitfalls myself. However the string handling required here is pretty basic (strings are encoded anyway as part of the ACME protocol so no escaping required there; and domain names are just ASCII characters without whitespace (even unicode domains just get compiled down to ASCII) so it's all pretty basic stuff). Plus curl and openssl are both solid tools and pretty standard on most systems (given openssl is used in the vast majority of web servers; you'd probably need that just to run your SSL certs anyway).

I don't have anything against lego either. In fact I'm quite an advocate of Go - having written a number of projects in that language myself. But I do think this is one situation that shell scripts are a technically valid option as well.

Ion shell is alpha but it has arrays, simple types and methods to process strings/arrays. I'm not the author but find it quite promising, since the shell ties so much of my userland together.

On the other hand, we have stuff like this: https://github.com/Neilpang/acme.sh/blob/master/dnsapi/dns_a...

Duuuude! There's an awscli out there that does all that for you, is maintained by Amazon, covers all the edge cases and is forward compatible cause Amazon takes care of that. Instead we have hundreds of lines of gnarly shell script code to compute hashes and prepare HTTP requests :(

I don't know about awscli, but s3cmd has a number of times been backwards and forwards incompatible with itself making any kind of script using it have to check the s3cmd version or punt everything into a docker image.

Meanwhile, the underlying API have been fine...

s3cmd seems to be an unofficial tool, from what I see.

The awscli, on the other hand, is the official CLI from Amazon.

"This is a work in progress. Please do NOT run this on a production server"

Not sure if "one binary to rule them all" really applies.

Why is launching those binaries a point of failure?

I use acme.sh in production on several servers.

When I decided to use it, I was looking for something I could read and understand in less than a day. The one file approach helped this. Lots of code bases are designed for easy change, which often means many small files, so you find the right line easier if you know which file you have to go to.

Because of historic reasons, I use it mainly with Apache, which I configure manually. I know there are options for automatic update of the config, but these weren't there when I read the script first. For me personally there are two problems - maybe these are even addressed in new versions of acme.sh, I haven't checked yet.

1. Apache won't load the vhost config for https if it can't find the key and cert files. Which now means, I have to use a http-only config and another full featured http/https when the challenge is done. 2. My typos in (sub-)domains are one of the main sources of confusion when I try to get a new certificate. I think using dig or host and curl it should be possible to warn the user in this circumstances.

Maybe you could write a small script that checks the parameters using your rules, then calls acme.sh passing these parameters?

Wow 6k lines of bash script with 1.6k tests file [1].

Looks a bit nightmarish to maintain.

1 : https://github.com/Neilpang/acmetest/blob/master/letest.sh

The test file seems fun, but overall it's not that bad. Bash is a fairly simple to parse syntax-wise, so as long as you know your coreutils and you know what you're looking for, making simple changes is fairly easy.

It doesn't look so simple, but definitely looks quirky.

Take a look at how the tests are invoked :

    for t in $(grep ^le_test_  $FILE_NAME | cut -d '(' -f 1) 
    do
      if [ -z "$CASE" ] ; then
        __green "Progress: "
        [ "$_ret" = "0" ] && __green "$_ret" || __red "$_ret"
        __green "/$num/$total"

        printf "\n"
        num=$(_math $num + 1)
      fi
    ...

It basically parses the very same file and checks for functions with the name `le_test`. This means that a random comment containing `le_test_` will break the whole script. You need lots of discipline in order to make things work.

Bash works perfectly with "Write programs that do one thing and do it well". Wouldn't be so surprised if this was a bunch of small test cases separated by file.

Looks like it's only grepping for lines that start with le_test_. A comment would start with #

^ is beginning of line, so you'd need:

    le_test_xxx # <- this is not a comment anyway
    # le_test_xxx does something nice. # <- doesn't match ^le_test_

I've just looked over the code and it's not that long. I could change the style and remove a few features and have it down to 2k lines.

The measure should be whether you can get your head around the whole script at once. I'm fairly sure I can do that. It's a clean example of getopt, makes full use of coreutils (and in an idiomatic way). It handles various OS and shell compatibility issues (and shows it has experience with where these arise). It does logging in a straightforward yet full-featured way.

As usual almost every comment here on HN is superficially critical.

Hmm, I deal with many files and maybe 100k lines of code in one project, 6k is not that much...

Edit: In case people don't know about Certbot from the EFF.

https://certbot.eff.org/

Acme clients are a space where there are a lot of bad choices. My favorite acme client is this: https://kristaps.bsd.lv/acme-client/

It is:

- Small

- Written in C

- Can be statically linked against LibreSSL and curl

No nonsense, does what it says on the tin, works on many operating systems.

I have written a small plugin for the acme.sh dnsapi that works with my DNS setup.

I run acme.sh in a FreeBSD jail (acme-client). It writes files that are picked up by another jail (acme-dns) that runs nsd. This jail is NOT one of my main authoritative name servers. It only runs _acme.mydomain.tld containing records like:

    _acme-challenge.test IN    TXT    XXXXXXXXXXXXXXX

These records are pointed to from my main name servers with records like:

    _acme-challenge.test  NS acme-dns.mydomain.tld.

(CNAME seems to work too, I will probably switch to that)

All this gets me the following benefits:

a) Everything runs restricted by jails that only run for about 10 seconds when issuing or renewing certificates.

b) No need for the service to be publicly accessible. (no issues with firewalls, no need for public IP's)

c) No need for the service to be some kind of web-server (think smtp, imap, irc, xmpp, etc.)

d) The service does not need a public "A" record.

e) No risk of me or the script messing up any live/production configuration.

f) The only thing that needs to accept inbound connections is the "acme-dns" jail on port 53 for about 10 seconds when it is running.

There are still some things I need to find a good solution for. Like easier distribution of certs, keys, etc. I also want to generate the private keys elsewhere and only give the CSR's to the acme-client jail. (If this is possible with ACME. I think it is.)

This setup is not yet complete and I am still experimenting, but it seems to work well.

That sounds like there's a lot of moving parts but it's interesting. Is there any part of it you can share?

Maybe when it is a bit more complete. Currently too much is hardcoded for it to be useful outside my setup. But if you look at one of the existing dnsapi plugins you will notice that only two functions need to be implemented. The rest is described above.

BTW: Remember to use letsencrypt-staging for testing.

Also, have a look at:

https://blog.crashed.org/letsencrypt-in-freebsd-org/

It was this that inspired me in the first place. I just added the separate subdomain and separate nameserver concept.

It's the seperate nameserver and subdomain that I can't quite get my head around - are you saying you can reply to a challenge for x.y.z.org from a nameserver at a.b.c.org?

I probably wouldn't trust implementing important network speaking services as a shell script, potentially running as root.

OpenBSD's acme-client [0] is a fork of Kristaps Dzonsons' project (formerly letskencrypt), it's a properly privilege separated ACME v1 client, written in C, using pledge(2) on OpenBSD, libseccomp on Linux.

https://kristaps.bsd.lv/acme-client/

[0] https://man.openbsd.org/acme-client

acme-client is great and I use it for RSA certs, preferring it over acme-tiny [0]. However I prefer ECDSA (until we get EdDSA), so for those certs I use acme-tiny.

https://github.com/diafygi/acme-tiny

A simple (6000 line) shell script.

"With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea." (hat tip to the authors of RFC1925)

https://tools.ietf.org/html/rfc1925

I just read through the top 1000 or so lines. Yes it's shell script. No it's not grokable if something goes wrong with it.

E.g.:

_supported_vtypes="$(echo "$response" | _egrep_o "\"challenges\":\[[^]]*]" | tr '{' "\n" | grep type | cut -d '"' -f 4 | tr "\n" ' ')"

+ extensions, see the DNS stuff.

The list of DNS APIs supported is impressive. This is the main problem with automating wildcard certificates (DNS provider specific APIs that makes using a single client challenging).

I'm curious to know if anyone else has looked specifically at the stateless mode this script uses. I've pondered it a bit, and it seems reasonable to me superficially, but I haven't invested much time into deeply understanding ACME. I'm wondering if using stateless mode opens one up to any sort of risk that is not present in the other modes.

EDIT: clarification; typo

Yes, this adds a risk which you might judge worth taking.

ACME http-01 validations involve asking an HTTP server for a resource in the .well-known/ reserved URI space with an arbitrary token name, and expecting a reply which contains the token AND a magic value associated with an ACME account.

Ordinarily one configures the server manually each time to respond to requests for a token you know will be used for a single ACME validation you want to succeed.

"Stateless" mode configures the web server to always reply saying the validation is OK for your ACME account, to any request.

Bad Guys can't just use this stateless configuration to get certificates because they don't own your ACME account, if they try to use _their_ ACME account, the validations fail because "stateless" is configured for a single account.

However, if bad guys get your ACME account private key or trick you into configuring one they know, with "stateless" mode they can request certificates at any time and your server will validate the requests automatically.

what about https://github.com/diafygi/acme-tiny ?

Yep. Works great and has a limit of 200 lines of code. Support for ACME2 will be a different project (possibly with a limit of 256 lines of code from what I've heard)

I thought I new Bash pretty well, but the leading underlines in function names is throwing me for a loop.

Is this a unique stylistic thing to the author, or is _foo() versus foo() as a function name following an established convention?

I asked someone when I saw them doing similar things. They adopted the style after reading Google's style guide for shell scripts. Not sure if that's where this author got it from, but it's one possible source.

I think this is the perl-style "private" convention. Those functions are helpers that aren't intended to be used on their own so he prefixes them with _.

I am not going to hate on this tool for being big, because it's portable, that kind of comes with the territory. However, if you wanted something similar that was more portable with less dependencies, you could write it in Perl. There are pure perl libraries for whatever you need, including crypto, and Perl runs on systems like Windows without the need for an extra subsystem or environment.

Still wondering what is this adds over a script like this, running weekly using cron:

    ./letsencrypt-auto --renew
    nginx reload

You mean certbot, the official client? It's not supported on Amazon Linux, for example.

What if your service is not HTTP? I use LE certs with nginx, Prosody (XMPP) and Mumble (voice chat).

ACME seems so overcomplicated.

How about just making the domain owner publish which certs are valid on a url like /certificates.txt

Then LetsEncrypt could periodically check if the certs they issued are still endorsed by the domain owner. And if not revoke them.

>And if not revoke them.

Certificate revocation is not guaranteed. Yes, there's CRLs and OSCP, but there are plenty of clients that don't do either of those. Not to mention how big CRLs will get if certificates can be issued and revoked for free.

Access to this /certificates.txt would need to be done over a TLS connection as an attacker could insert their own certs here.

ACME avoids this by associating a specific CSR to a response, so an attacker could not insert their own certificate in the middle of the process and get it signed.

But it does have dependencies (as any shell script tends to). e.g. it requires openssl for computing hmacs; and it needs either curl or wget for doing http requests.

Also, Why would someone prefer this over https://github.com/lukas2511/dehydrated ?

+1 for dehydrate - been using it for a long time

Indeed. For a long time I was using my own fork of simp_le, but the code wasn't easy to split the file writing logic into per domain directories for certificates, etc.

L1: #!/usr/bin/env bash

Bash is GPL Licensed and does not ship with most UNIX-like systems. *BSDs don't ship it, macOS is stuck in the past, etc. As a BSD user myself, I install curl(1), but not bash(1) on my machines (or jails).

OpenSSL (an alias of LibreSSL when applicable - e.g. OpenBSD) is standard and available on virtually all systems.

Reply on news.ycombinator.com