Dehydrated: Letsencrypt/acme client implemented as a shell-script | Hacker News Reader

Dehydrated: Letsencrypt/acme client implemented as a shell-script

github.com

129 pointsthunderbong107 comments

I used this years ago to implement SSL for a bunch of customer vanity domains, and as far as I know it's still operating.

The best thing is it's declarative: you provide a text file with domains you want in each certificate, and it makes it happen.

Certbot has discrete commands you have to run to add, remove, or modify certificates, so you have to keep track of state. We had a management UI to control these domains, so it didn't work well with something like terraform (which I think otherwise solves the state problem for certbot). Instead we just wrote out the domain file based on db config changing + daily, and ran dehydrated. If there were no certificate changes or renewals needed, nothing would happen.

Also the hooks are great, iirc we had a pre-check in place for our own http-01 validation, so as not to cause failures on let's encrypt. (Mostly customers would cname domains to us, but lots could go wrong). We were also using s3 to store the validation files (again, super easy with bash hooks) to make it work across a pool of load balancers.

This is a great example of a tool where a focus on simplicity (config, hooks, and the way it stores everything) creates an incredibly easy to use, flexible (and thus powerful) system.

Apparently the author recently sold the project to a company called apilayer: https://lukas.im/2020/01/30/selling-dehydrated/index.html

They plan to keep the project open source and employ Lukas to continue maintaining it.

APILayer = ZeroSSL

ZeroSSL are the only ones providing certificates for IP addresses and free year-long certificates. Cudos to them for disrupting this market, almost monopolized by letsencrypt. Don't have high hopes, though, big players probably will kill them as they killed other free certificate issuers. For some reason letsencrypt status-quo as the only free certificate issuer benefits big players.

The certificate for zerossl.com is also issued by Let's Encrypt.

https://crt.sh/?q=zerossl.com

letsencrypt is hardly the only free cert issuer. even google offers them https://security.googleblog.com/2023/05/google-trust-service...

Which ‘big players’?

Jesus, not only do people seriously use this but somebody bought it? The world is insane.

Parsing json in shell? Easy!

filter="$(printf 's/.\[%s\][[:space:]]$[^"]*$/\\1/p' "$(json_path "${1:-}" "${2:-}")")"

Please, no! How can you know this is safe?

Why not just use Python? It is installed pretty much everywhere, supports json without such hacks.

A very relevant question. Acme.sh, a similar shell script ACME client, had a remote code execution problem last year.

https://github.com/acmesh-official/acme.sh/issues/4668

> It is installed pretty much everywhere

Yes, but which version?

Hase the json module api chnged in the past decades?

https://news.ycombinator.com/item?id=40100587

Not the modules, but Python itself.

3.6 is pretty much universally available at this point.

RHEL 7 (which still has 4 years of extended lifecycle support left) ships with Python 2.7.5

MacOS Sonoma (the latest version) doesn't include Python at all; older versions of MacOS which are still supported ship with Python 2.x

Even python2 is muchly preferable to bash/awk, thanks.

I don't think anyone using RHEL 7 is in the target audience for this software. Be realistic.

There are still machines on 2.6 kernels where I work.

> I don't think

> target audience

You != people

And do those machines have web servers exposed to the world?

Yes, why wouldn't they? There are multiple vendors who still provide security updates for CentOS 6

Do you ask that question to bash me for having an old, unsupported OSes open to the world?

Then address this to the owners of those machines. And get a load of cash ready to give it to them, so they could migrate to somehitng newer.

I don't think you really have any idea what you're talking about, to be honest - there are millions of RHEL 7 and CentOS 7 boxes in production right now, and many of them are serving websites and could benefit from SSL certificates.

I know of a couple large organizations that have tens of thousands of RHEL 7 machines that use letsencrypt for customer websites, and I sent this project to sysadmins on their teams, so it's very realistic to expect that businesses are still using supported software.

Not everyone can just upgrade to the latest version of Ubuntu every 6 months for their production workloads like you might expect.

In my experience Python software is not really nice when it comes to distribution. Hate the venv stuff and pip install hassles. Besides, it is slow. Honestly, if runtime interpretation is a requirement, PHP would be better then in these respects.

You can do letsencrypt stuff using python standard library, without touching pip and virtualenv. This is my own weekly cron script, unchanged since 2020, with openssl(which is usually preinstalled) and acme_tiny.py with system python. You can easily check what the script is doing by adding set -x and acme_tiny.py is quite readable too. Unlike the monstrosity above.

#!/bin/sh

set -eu

set -o pipefail

cd /home/letsencrypt

openssl verify -CAfile lets-encrypt-x3-cross-signed.pem -attime `date -d 'next month' '+%s'` mah_domain.crt | grep expired || exit 0

DT=`date "+%Y%m%d"`

python3 acme-tiny/acme_tiny.py --account-key account.key --csr mah_domain.csr --acme-dir /srv/www/mah_domain/htdocs/.well-known/acme-challenge/ > signed_${DT}.crt

curl -sO https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.p...

cat signed_${DT}.crt lets-encrypt-x3-cross-signed.pem > mah_domain_chained_${DT}.pem

mv mah_domain_chained_${DT}.pem mah_domain_chained.pem

mv signed_${DT}.crt mah_domain.crt

set +e

nginx -c /srv/nginx/conf/nginx.conf -s reload

The point of using '#!/usr/bin/env python3' is that you don't have to mess with virtual environments.

Using this exact shebang line signals "This script is intended to be executed directly (not a python module) and it only needs the python3 standard library."

Surely it could just include jq.

https://jqlang.github.io/jq/

python hasn't been installed by default in almost any env i've worked in for the past ~5 years

maybe for dev environments, but certainly not for servers

Interesting how that was achieved. Most popular Linux distributions are very hard to install without python3 an perl5. LSB https://refspecs.linuxfoundation.org/lsb.shtml defines them as standard.

Maybe not so hard as you might think. On my openSUSE desktop, uninstalling python3 would remove only calibre and virt-manager. Most of my server/container/vm installs don't have any python version. Perl isn't much harder to get rid of depending on the use case, for example some development stuff uses it.

I'm also not sure distros generally care about LSB. Debian dropped it years ago and did any distro even bother to implement it correctly?

I don't get all these arguments. You would really prefer to use hard-to-check shell script to simply installing system python package if it isn't installed already?

Why would I want a system level python install?

Because it allows for scripting infrastructure with far fewer obfuscated hacks than bash/awk. And if you use system python, vulnerabilities are more likely to get patched by distribution vendor. Your homebrewn json parsing regexes won't.

You said distros can't install without python and perl, my argument is the opposite. Shell scripts aren't in the picture.

I would not trust that script. If Python is such a hassle then there is also Lego: https://github.com/go-acme/lego which supports tons of DNS providers and is very well documented.

Dehydrated has been my ACME client of choice for many years now. Nice and simple and easy to use.

Works fine. My only complaint is that it’s a bit slow. Granted, my use case is a bit different than most.

  $ wc --lines /etc/dehydrated/domains.txt
  5161 /etc/dehydrated/domains.txt

At that size I would recommend to dynamiclay generate the domains.txt and only place domains in there that require renewal.

Any reason to prefer it over acme.sh ? Seems pretty similar. I've been using acme.sh for years, usually in nginx mode, sometimes in DNS mode.

What's the motivation behind closing file descriptors 3 and 4?

   # Close weird external file descriptors
   exec 3>&-
   exec 4>&-

From this commit:

https://github.com/dehydrated-io/dehydrated/commit/b116e6bc2...

When are those ever used? Is it to customize output for when you’re sending to a file vs terminal?

systemd socket activation opens extra fds starting from fd 3 (SD_LISTEN_FDS_START). Though I can't think of a scenario where this may be relevant for dehydrated. Maybe some weird systemd unit somewhere that invoke dehydrated does this?

Using it for years in DNS mode, works like a charm

Why would you use this for anything else than a joke given that Certbot exists, and is even in most repositories? (asking seriously)

Certbot is a massive pile of dependencies that you need to care and feed and update. You should not need 100s of megs of nonsense for something that can be done with a shell script.

People are always talking like shell scripts are without dependencies. Shell scripting ist mostly piping program outputs to different programs, that have dependencies on libraries themselves. Each of them have to be kept up to date to stay secure. Just because they are mostly preinstalled on the system doesn't make them not a dependency.

This, together with the fact that most shell scripting is bash based which is, in my opinion, not a very good language, makes it less secure to me than a python tool.

If it is just a few curl and openssl commands, why make a user install hundreds of megs of python deps just to ultimately call mostly openssl commands anyway

One of the biggest risks today is supply chain attacks. The more dependencies you have, the more people you are giving the ability to tamper with your critical code paths.

Wouldn't this bash script be just as susceptible to supply chain attacks? What dependencies does openssl, cURL, sed, grep, awk, mktemp pull in, and could it be vulnerable to attack like we saw with xz?

I think the point is to minimize the total attack surface on a system. If the bash script only depends on programs like those you list, which Linux distros package as default anyway, the shell script has not introduced any new dependencies on your system.

Absolutely there is still supply chain risk there. But less risk in 3 dependencies than 300.

3? You think you underestimate the number of libraries your software uses.

I actually full source bootstrapped a Linux distribution from zero recently. Most distros have a lot of dependencies you do not need.

In this case you only actually need curl, openssl, and busybox. Those depend on at least a small libc implementation and a kernel but those are certainly already present.

Coreutils or busybox is probably already installed too. Still, I will grant the requirement of musl and a linux kernel since we are being pedantic or maybe talking about an embedded linux use case, so 5 deps total to boot from metal and get a cert.

To be fair I would never actually ship openssl or busybox in a real embedded project. Would probably write a simple standalone binary using the standard library of Go or something.

How is that in any way realistic? Most people don't bootstrap Linux systems. And just because it is pre-installed doesn't make it safe. Most software on a modern Linux distribution is dynamically linked to several libraries which might depend on other libraries.

> Wouldn't this bash script be just as susceptible to supply chain attacks? What dependencies does openssl, cURL, sed, grep, awk, mktemp pull in, and could it be vulnerable to attack like we saw with xz?

All the packages listed are already probably installed on your system, so you have to worry about their integrity already (your system package manager (RPM, Deb) probably leverages them).

Something like Certbot pulls in dependencies on top of what your system already has, whereas Dehydrated or Acme.sh use tools that you already have to worry about anyway because they're part of the base OS.

I’m pretty sure every open source repository ever created is vulnerable to “an attack like we saw with xz,” and by that I mean a labyrinthine, multi-year effort involving highly sophisticated exploits combined with psychological manipulation, coordinated sock-puppets, global actors, and deep intrigue—to the extent that it’s absurd to cite as a threat that “dependency management” can meaningfully prevent.

I saw the following programs whil skimming the code:

awk, grep, curl, getent, sudo, mkdir, mv, ln, cat, rm, openssl, touch

Most of these have around 10 dependencies on libs.

Don't misunderstand me. I have nothing against the script. I just don't like the argument that bash is better because it has no dependencies.

busybox, openssl, curl. Done. :)

The OS itself depends on many libs ;-)

The only things that are a bit heavy in the list are openssl and curl. Still, they are relatively self-contained tools, nothing to do with the dependency hell of certbot.

It doesn't even work well in my experience. ACME.sh is a lot more hassle free, I've always had issues with Certbot's DNS verification.

Basically I'm not avoiding Certbot to make a point, I just think it's inferior for my specific use cases. I don't know about Dehydrated but I also expect it to be BS free.

Yup, I gave up on certbot after trying to understand why I couldn't get the Digital Ocean DNS plugin to work, which is maintained as part of the Certbot repo. (This was a few years ago, so details may have changed.)

I discovered the issue was that the plugin does some pretty broad-brush guesswork about which domain in your DNS hosting it should actually populate with the response value. If you own a bunch of similar domain names (as many orgs do), the plugin may guess wrong.

Much happier to be using dehydrated, and I don't regard it as a major impediment that I had to spend 10 minutes hand writing the necessary API call to the DNS provider.

Exactly. I've been using dehydrated on two servers since it was named acme.sh (pre-2015) and updated it once. It just works and no dependecies.

Dehydrated and acme.sh seems different. Is one of them fork?

https://github.com/acmesh-official/acme.sh

https://github.com/dehydrated-io/dehydrated

Brainfart from my side. When I first installed dehydrated.sh it was originally named letsencrypt.sh (not acme.sh) and later renamed to dehydrated. That was quite a while ago.

With that rationale, why not use https://github.com/go-acme/lego which is a self contained lets encrypt client in Go?

Static binary, sure, but hardly a tiny supply chain attack surface: https://github.com/go-acme/lego/blob/master/go.sum

Also their official builds are built with Alpine which is a hobby distro that does not even do signed code or packages.

Are you saying that any use of Alpine is, by definition, a supply-chain security problem?

I am, yes. Alpine is not full-source-bootstrapped, often imports and trusts external binaries blindly, has no signed commits, no signed reviews, no signed packages, and is not reproducible. It is one phished git account away from a major supply chain attack any day now.

Alpine chooses low security for low contribution friction. It is the Wikipedia of Linux distros, which granted it a huge package repository fantastic for experimental use and reference, but it is not something sane to blindly trust the latest packages of in production.

It is one of the reasons why I made stagex, which in most cases is a near drop-in replacement.

https://codeberg.org/stagex/stagex

Thanks for the detailed response!

EDIT: Also, stagex looks pretty compelling; I hope it catches on!

The only joke here is Certbot, a tool intended for use on servers but distributed on a channel intended for consumer use - Snap. The only package manager that will update on its own, when it feels like it, without a killswitch, runs as a daemon intertwined with the system.

Icing on the cake is that the Certbot team advertises alternative install methods, of which none work and all of the lengthy guides for them recommend to use Snap instead. It’s an insult for professionals.

> the Certbot team advertises alternative install methods, of which none work

I've only been using it for a little while, but the pip + venv method seems to work decently well: https://certbot.eff.org/instructions?ws=other&os=pip

Did you see that bold disclaimer on top?

> Partial support > The Certbot team supports this installation method on a best effort basis. If you are on a more obscure or heavily customized system, these instructions may not work and the Certbot team may be unable to help you resolve the problem.

This is an instant no-go for any professional environment I ever worked in.

Yep, this.

So ridiculous to have to install yet another package manager, snapd, to get certbot installed "the easy way".

The alternative to get around snapd works fine (IMHO, for my situation, a lightsail instance running amazon linux 2023 where installing snapd is a pain in the ass), but you still have to jump through some hoops.

I'm a simple man, I see a comment complaining about snap and I automatically upvote.

It's 2024. If you're still distributing snaps as the preferred method of install, you're alienating your users. AppImage, Flatpak, and regular containers are all far better deployment options.

Same. I have a couple of comments as to why snap is awful.

> If you're still distributing snaps as the preferred method of install, you're alienating your users.

Dropped lxd because of it. Now I hear they're a proper debian package, but that decision was so weird (especially the auto-update part) I don't trust them anymore.

I wish I could upvote this more... snaps are the worst!

I wish everything in linux land just had an AUR equivalent.

I am doing docker run -it --rm certbot/certbot

Is there a problem with that?

I wouldn’t call it a problem, but let’s hear what the Certbot docs have to say:

> this mode of operation is unable to install certificates or configure your webserver, because our installer plugins cannot reach your webserver from inside the Docker container. > > Most users should use the instructions at certbot.eff.org. You should only use Docker if you are sure you know what you are doing and have a good reason to do so.

These problems are solvable if you know what you do, but the whole premise of ACME was making it easier to obtain certificates; plus, I shouldn’t need to decide between an autonomous and hostile package manager or keeping a container environment running, secure, and configured - to set up bloody TLS certificates for a Webserver. That said, good for you if it works :)

I mount webserver docroot and /etc/letsencrypt directories into the certbot container, /docroot and /etc/letsencrypt mount points respectively, it is totally prepared for this.

How did you implement reloading the modified TLS certificates after renewal tho?

it's a two line script running periodically. the first line is docker the second is nginx -s reload. As /etc/letsencrypt dir is mounted from nginx into certbot, certbot upgrades that, nginx reload picks them up. Easy as pie.

so much this. Certbot is a monster with incredible amount of dependencies.

The EFF themselves recently explored the question whether Certbot should just be replaced with servers like Caddy. https://www.eff.org/es/deeplinks/2024/03/should-caddy-and-tr...

One of the biggest benefits of dehydrated is that it doesn't try to integrate with a DNS provider on its own. It just connects to ACME server and calls a hook, which can be implemented with a simple shell script[1]. The most popular third-party integration is lexicon[2], though you're not required to use Lexicon. (e.g. you're free to use awscli, gcloud, linode-cli, etc. to do the actual DNS record manipulation)[3]

This means its dependencies footprint is much smaller, and allows you to do things that can be a nightmare to configure with Certbot or other alternatives. For example, at one of the scenarios I had to set up was that we had to query a credential via HashiCorp Vault, which is then used to cURL into an API endpoint. The shell script in total was pretty short (~200 LOC) and it worked extremely well. The fact the shell script is so simple that I could test adding/removing records without ever invoking ACME process is also a huge benefit.

[1]: https://github.com/dehydrated-io/dehydrated/blob/master/docs...

[2]: https://github.com/AnalogJ/lexicon

[3]: https://github.com/dehydrated-io/dehydrated/wiki

Using it with Hetzner DNS API that way.

Does DNS mode work to issue for internal servers that don't expose 80/443 to the internet?

Yes.

That's one the main reasons to use it, another being wildcard certificate.

I created a k8s deployment which spins up nginx in a job to update TLS secrets with dehydrated because I couldn’t get an admin to install cert-manager. It works great, especially because I had scheduled rebuilds of the base container off of latest images. I left the job 3 years ago, and it’s still running fine and staying ahead of the vuln scanners.

I had also used a pre-shared private key, which I put on a F5 BigIp, and just scheduled a job on the F5 to pull the updated certs daily.

I think the amount of alternative ACME clients is a testament to the absolute mess that certbot is.

Constant breakage unless you're using snaps on the most popular distributions. The CLI is absolutely idiotic and convoluted and the plugins do a lot of guess work without informing you, which results in some fun debugging. It's also a large pile of dependencies that take up space.

How does dehydrated compare to nginx-proxy with acme-companion from an operational perspective? I've been looking at other options and the best so far seems to be Caddy.

I prefer this acme client written in pure C : https://github.com/ndilieto/uacme

Was my default way of updating my certs until I switched to using an Nginx proxy docker image that handled it for me automatically. Still highly recommend this tool.

As a bash script actually, not a (implied POSIX) shell script. Do not mix these two.

Seems to work in zsh too. Both bash and zsh are shells like c-shell ksh etc. Nowhere is it implied that shell-scripts are written for Bourne shell.

both bash and zsh are orders of magnitude more complex than sh or even ksh

I am not familiar with zsh, but is it really interpreted by zsh? Because the script has #!/usr/bin/env bash in its shebang, isn't it executed by bash on your system, even if launched from zsh?

It seems to be very source-able in both bash and zsh. If you just executed it, it will run in bash.

They seem to have gone through some effort for this to be true.

Bash is a shell script. Posix is not implied.

This script is explicitly a Bash script and it is not executable by every other shell present on modern unix-like systems. Examples are Korn shell, Almquist shell. Hence the distinction: if one states that the script is a shell-script, it implies that it can be interpreted by any modern shell, for which there is only one common denominator, POSIX. This script is explicitly only Bash shell compatible, not any-shell compatible.

> (implied POSIX) shell script

Why would POSIX be implied in any way? I mainly use windows, should I have been upset because I thought cmd or powershell was implied?

I don’t get why people are saying it’s hard to install Certbot. I just do:

  sudo apt install certbot python3-certbot-nginx

And it “just works” on Ubuntu. The whole thing is super easy and takes around 1 minute to get a cert installed and configured with nginx.

Not everyone's happy to allow certbot to change nginx configs. I never tried it and I'll never try it.

Also not everyone wants to install few dozens of packages just for this little thing.

All you need is certbot and a post-hook to restart nginx.

There’s this misconception that you need extra packages to get certificates working with different software.

It has a mode where it doesn't touch your nginx config. That's what I use because indeed, I don't want it to mess with my nginx config.

For the rest, I'll indeed look into lighter alternatives some day. But the setup works on its own so I'm a bit lazy.

> And it “just works” on Ubuntu.

Sure. Now look at the dependencies that are installed compared to dependencies that are installed for dehydrated (or acme.sh, etc) which generally are: bash, OpenSSL, cURL. This is very handy for more appliance-like system (I ran dehydrated on (RH-based) F5s for years before ACME was put into the GUI; also ran it on (FreeBSD-based) Isilons.)

Also, if you want to do an audit of the code, how many lines of Python need to be examined (including dependencies) compared to how many lines of Bash? (Both would have common dependencies like (Open)SSL and HTTP/cURL libraries.) As we saw with the recent XZ kerfuffle, 'software supply chains' are becoming important.

Personally I find it much easier to understand / configure dehydrated:

    $ git clone https://github.com/dehydrated-io/dehydrated.git
    $ sudo cp dehydrated/dehydrated /usr/bin/
    
    # cat > /etc/dehydrated/config
    WELLKNOWN="/var/www/htdocs/.well-known/acme-challenge"
    CONTACT_EMAIL="you@example.com"
    # cat > /etc/dehydrated/domains.txt
    example.com www.example.com
    # mkdir -p /var/www/htdocs/.well-known/acme-challenge

    $ sudo dehydrated -c

(Ubuntu/Debian has packages for it as well.)

Certbot just works? It has a ton of bugs.

Reply on news.ycombinator.com