Hacker News Reader

Comment by andrewstuart | Hacker News Reader

I used nginx for a very, very long time but abandoned it.

I switched to Caddy because I just got so sick of the constant pain of certificates. Caddy does certificates automatically.

I also was sick of the complexity of nginx configuration. I could get the things done that I needed to do but it was always painful and time consuming.

I've worked out how to do in Caddy everything that I did in nginx though that took alot of time and effort, but I tend to create quite sophisticated web server configs.

At this stage there's nothing really that nginx could offer that would bring me back from Caddy.

View on HN

> I also was sick of the complexity of nginx configuration

I'm old enough to remember the same complaints about Apache, and nginx being the simpler alternative.

Something about systems evolving to a level of complexity/sophistication until they collapse under its own weight...

I wonder if there's something about changing uses cases. If you design for x and y use cases then you end up with a nice, simple design for x and y. People then start doing a lot of z so you adapt to support z, but it adds some complexity because it's not exactly what you had in mind when you laid the foundations.

A competitor comes along who's designed for y + z, and everyone loves how nice and simple it is. Then use case u comes along...

Nah, probably use case x. Rinse repeat.

If we are to trust a certain philosophy based on simplicity, one of the _very hard_ things you have to do as a head maintainer of software is to have the ability to say no when that case arises.

That’s the trick - sometimes, if you don’t add the features, you lose share because the market has shifted and now a different set of features are what is valued. So you can either patch and lose focus, or not patch and lose customers.

This seems very much like the browser/frontend space, where Webpack was once cool and slim, but is now just as bloated and slow as the systems it set out to replace.

Was Webpack ever slim? I always used browserify when I wanted a slim option.

glad someone mentioned it, browserify is way simpler to use and bundle. But now we have esBuild!

Webpack was never slim. It just integrated things nicely in a more declarative way than Grunt/Gulp.

>I'm old enough to remember the same complaints about Apache, and nginx being the simpler alternative.

Was about to comment the same !

What is that saying:

"You either die early, or live long enough to see yourself become the villain" ?

> Something about systems evolving to a level of complexity/sophistication until they collapse under its own weight...

To my knowledge the nginx config hasn't gotten more complex.

I don't know caddy but I suspect it's just simpler.

Yeah, what's happened here is that Caddy is optimized for the most common use cases and gives them a "happy path." This simplicity is only possible in a world where something like nginx (and apache httpd before it) exists and people can figure out what the "happy path" should even be.

Having used Apache, Caddy, Nginx and more, I can tell you it't not about "sophistication" or features necessitating complexity.

It's about bad configuration file design, little thought put to usabillity and ergonomics, and, actually, a total lack of knowledge in that domains (usability) on behalf of the devs...

If you look at those three specific examples, you'll see decades of progress in the field between them. I don't think it's fair to say bad config file design, but maybe more fair to say that we've just gotten better at it as an industry over the last 30 years, and the ability to write better config files has gotten easier as well.

I’m old enough to remember Nginx and it’s selling point was never easy of configuration. It was always about the number of connections it could handle which made it great as a reverse proxy for web2.0

> I'm old enough to remember the same complaints about Apache, and nginx being the simpler alternative.

Nginx is still far simpler than Apache.

It just so happens that subsequent projects picked specific usecases to work on their happy paths to make them far simpler. Certificates is one of them.

> Something about systems evolving to a level of complexity/sophistication until they collapse under its own weight...

This is nothing of the sort. It's a matter of newer and better options popping up, not that old options got worse.

I came fro Apache through Nginx to Traefik and shortly tried Caddy. Went back to nginx for a lot of stuff because it's easier to understand and debug than Traefik.

I think a shell script with certbot and "nginx reload" is much easier than some closed service I can't take a look at.

I'm using all three, Apache, Caddy and nginx almost daily. For the trivial, Caddy is amazing. When things get more complex alongside needs themselves, nginx is amazingly powerful. Apache at the same point becomes overly verbose, anomalous and painful. It does take some discipline though, to comment and structure your configuration files, but that applies to anything that isn't trivial. return 444; alone is a great feature I can't do without, not to mention all the modules one can use.

The thing with certificates is that most tutorials are terrible. A very seamless way is to use the webroot method. Create a nginx configuration file that places LE's challenges to a specified folder on your system, include that config into your server blocks. Then you instruct certbot to use that folder. Really quite easy.

> Really quite easy.

“Your ideas are intriguing to me, and I wish to subscribe to your newsletter.”

Is there a detailed writeup on this? This sounds great, but I don't want to faff about with it.

In its most simple form, it works as follows.

/etc/nginx/letsencrypt.conf:

    location /.well-known/acme-challenge {
        default_type "text/plain";
        try_files    $uri $uri/ =404;
        root /tmp/letsencrypt-auto;
    }

/etc/nginx/sites-enabled/default:

    server {
        [...]

        include letsencrypt.conf;

        [...]
    }

Reload nginx.

Invoke certbot for the new domain once with:

    certbot certonly --webroot -w /tmp/letsencrypt-auto/ [--must-staple] -d example.com

Wait, how does that makes nginx use the certificates? For me it doesn't have any cerbot/letsencrytpt specific configuration as I use DNS-01. I just point ssl_certificate and ssl_certificate_key to the correct path where cerbot saves the certificates and be done with it.

Making use is easy as well, like you described. Just a matter of specifying ssl_certificate(_key) path. It doesn't support variables so it can't be based on the server block's domain/variables unfortunately.

DNS challenges are a bit more seamless, but I personally don't like giving access to entire zones to a single machine. Like most DNS APIs force you to.

The LetsEncrypt tutorial showed me exactly this configuration.

What's keeping you from using Caddy exclusively?

For us non-specialists, is 444 a shorthand "terminate with extreme prejudice" upon suspected hacking?

https://serverfault.com/questions/822484/nginx-444-error-eqi...

It's great if you want to not even bother replying to some requests. I like to use it to not reply to requests that come without a valid SNI/Host.

> I just got so sick of the constant pain of certificates.

I've honestly never had any sort of "constant pain" by using nginx and acme.sh. Certbot is an actual abomination.

I have a script just as a shortcut for these two commands:

    acme.sh --issue --dns dns_cf -d "$1" -d "*.$1"
    acme.sh --installcert -d "$1" -d "*.$1" --certpath /foo/$1/cert --keypath /foo/$1/key --fullchainpath /foo/$1/fullchain --reloadcmd "/usr/local/bin/docker-compose -f /foo/docker-compose.yml exec -T nginx nginx -s reload"

You just have to run those commands once per domain and it'll keep that wildcard certificate valid forever, acme.sh sets up a cronjob to renew the cert when needed and will automatically reload my nginx container after.

And if you use caddy you never need to know what any of that means, since caddy completely automates it. Maybe it's not a pain to you since you're used to it, but it's a major obstacle to most people looking to set up a website, particularly for the first time.

Even though I've set up nginx and certbot before, I'm happy I don't need to think about that stuff with caddy. Total waste of mental resources. I just want to get stuff on the Web.

I want more control over the cert request however, and to use the certs for SMTPS, IMAPS etc. But for simple HTTPS only caddy is great.

I’m the same. I’m able to host unlimited subdomains using wildcard TLS and unlimited apex domains using TLS on-demand, all from one Caddy config. It’s amazing.

No disrespect to the good people behind nginx. Caddy is simply part of a new generation of web servers.

> Caddy is simply part of a new generation of web servers.

I feel old now, as I continue to deploy Apache :-)

It's cool as long as you compile in mod_perl, use --with-mpm=prefork, set LC_ALL=C, and use /bin/csh. ;P

> I feel old now, as I continue to deploy Apache :-)

we wrote bash scripts backends. don't worry about being old. keep going ;)

some even had their cgi code running in kernel.. all the more reason to be merrier.

One pain point with caddy for me was cloudflare dns validation. I had to rebuild caddy from scratch, build my own container image. This process is slow as hell and very RAM-intensive, I had to add swap just for it. I wish they publish images with popular plugins built-in.

They do provide pre built stuff. It's just really unclear on their website that this is exactly what they do.

I had the same problem as you. Here is me whining about it on the Caddy frum:

https://caddy.community/t/wildcard-certificates-building-fro...

But in the end I managed to get the pre built stuff.

Their download section is extremely unclear about whether or not you need to build Caddy. You don't.

With this docker image you can just download the binary and add it as a volume!

https://hotio.dev/containers/caddy/

Hm, i found this to be pretty straight forward through Docker:

e.g:

    FROM caddy:2-builder AS builder
    RUN xcaddy build \
        --with github.com/greenpau/caddy-security

    FROM caddy:2
    COPY --from=builder /usr/bin/caddy /usr/bin/caddy

You can keep your container image, and just do the "COPY --from=builder" step in there. We do this in CI/CD on every build.

I did something similar. It puts my server to knees and takes something like a hour to complete, requiring all services to stop before I can run it. This is extremely unergonomic for me.

what's the server spec?

VPS with 1 core and 512 MB RAM. I just checked, not really a hour, it spent 25 minutes and then failed with "no space left on device". Right now there's 4.9GB free, so it takes at least around 5 GB disk space which is an issue as well.

Building on separate computer will work, of course, but it makes everything more complex.

Easy way around that is to use a CI service like GH Actions. Build it there, push it to dockerhub, and you're done.

Have it run on a schedule to keep it updated, if necessary. Alternatively, GH Actions allows you to set a manual trigger so you can rebuild whenever.

You can just create the image on some other powerful server and the just pull the final image from the registry.

You go to the download page. Click the checkbox for cloudflare dns and hit download. The binary you download will have the module built in. It's very simple but of course being nonstandard it makes updates unnecessarily painful. It would be nice if you could wget it with a special hash that represents the modules you need or something like that.

I use docker.

Some other user explained how you can use caddy to automate it below.

We have the caddy add-package command that may be of interest to you!

The configuration format is easily the worst aspect of nginx. There was a stretch where I got heavily into using it because it really is capable of quite a lot, but I could never totally grasp the config, despite how I've managed to learn pretty much any programming language given to me. Depending on what you're doing, a certain other thing may be totally moot or out of place, but it's certainly not obvious, and good luck actually debugging your nginx config without resorting to The Google. Once the config is set, nginx can do its job quite well, but it's bad if you're frequently making architectural changes.

every configuration format has it's quirks, and nginx certainly has a couple.

but coming from apache it was the freshest breath of air possible. i still prefer it to any yaml/json alternatives newer products like envoy use. a pure ymmv point.

i don't think that having a manual or reference open while doing more involved setups is a sign of bad design. some things are simply complicated...

That's the thing, it's not a programming language. It's a sequence of declarations. In-spirit very similar to iptables or nftables rules for example.

Documenting things that took time for you to grasp is a great way to learn, in my experience.

I wasn't trying to equate it to a programming language, though. IMO, a config format shouldn't be more difficult to grasp than a Turing-complete language. To me, nginx conf is hard to gain proficiency in because the amount of experience to know whether a declaration is in the appropriate place or won't be ignored is very high. Obviously someone can indeed become proficient, but I've found it to be extraordinarily high for someone who just needs it occasionally; I really don't want to live in nginx land all the time, in part because I'm a software engineer and not really an ops person.

For instance, should `upstream` be defined at the top-level or under `stream`? If you have another server doing a proxy_pass to that upstream, does it also need to be under `stream`? This is in no way obvious by either looking at nginx configs or by reading the docs.

And then what if you have a web server? Does it need to be under `http`, or can it be top-level? It can be either one depending on who's config you're looking at. But if your server you're pointing to is already an HTTP server, why would you want nginx to use its own http serving on top of your HTTP server?

That's just a couple of examples. And yes, there are absolutely logical answers to those, but I do not believe that the nginx config format speaks for itself.

Certificates are only annoying if you use certbot, which has terrible UX, virtually any other tool makes it trivial, e.g. Dehydrated or acme.sh.

> At this stage there's nothing really that nginx could offer that would bring me back from Caddy.

Performance?

My guess is that a massive majority of websites will never notice any performance difference.

Its noticeable whether you need 1 server or 3 servers for the same workload.

It's quite rare that your webserver will be the bottleneck at this point. Your app will be the bottleneck much more often.

I don't think the difference in performance between Nginx and Caddy would mean you need 3 servers versus 1 of the same spec. But of course, you need to run your own benchmarks on your own config to determine which is best _for you_.

Benchmarks cannot be done generally, because the config drives so much of what happens at runtime, and everyone has different needs.

Hint: *asterisks* in stead of _underscores_ --> italics.

I guess I'm too used to markdown :) thanks

> I switched to Caddy because I just got so sick of the constant pain of certificates. Caddy does certificates automatically

That may be fine if you only use those certificates for port 443 but it falters where certificates are used for the remaining 65533 ports. Instead of leaving certificate request and renewal to individual tools I prefer to centralise it in one spot - a small container or VM is sufficient - which deals out ready-to-run certificates to those systems which need them. When dealt with this way nginx actually has no problems whatsoever with certificates, all it takes is those three (or more depending on your protocol needs) lines in the config file:

   listen 443 ssl http2;
   ssl_certificate /etc/letsencrypt/live/site.example.org/fullchain.pem;
   ssl_certificate_key /etc/letsencrypt/live/site.example.org/privkey.pem;

The /etc/letsencrypt/live tree gets pushed to the relevant machines by the certificate handling container/VM with certificate files in the form and shape needed by whatever program is to use them.

So, in short, the Caddy approach may be fine for those who only run web things but it 'fails' (as in 'adds additional complications') for those who venture beyond port 443.

That's not true at all. Caddy can automate certificate issuance with any of the 3 challenge types, either HTTP-01 (requires port 80) TLS-ALPN-01 (requires port 443) or DNS-01 (requires building Caddy with a DNS plugin for your DNS provider).

Also you can cluster Caddy by making sure the filesystems are synced, or using a storage module like Redis. Then any of the Caddy instances can do any step of the issuance phase (one can start it, another can complete it).

You can use these certificates for sites on non-443 ports as well, once you have the cert. Nothing special at all to do there, you just configure Caddy to serve a site with a different port, e.g. `example.com:65533`

You also get numerous other benefits, like OCSP stapling, issuer fallback (if LE is down, it'll try ZeroSSL), etc.

You don't seem to understand what I mean when I say 'port 443' - i.e. HTTPS. I'm talking about the use of certificates in non-web applications, e.g. SMTPS, IMAPS, database servers, etc. I prefer not to run a web server on my database servers, mail servers and what have you.

It's been 5 years since I used any web server as I started using Go to build web applications.

1. HTTP server is production ready.

2. autocert fetches certificate automatically from LetsEncrypt.

3. Graceful restarts (zero downtime) using signals.

I think the the biggest advantage of languages which enable portable applications like Go is not having to mess with web servers anymore, At least for most common use cases.

heh, I'm on the exact opposite side of the fence. Tried caddy, found it confusing and incomplete so I went back to nginx. Buuut then again, I've been writing nginx config for 10-15 years :)

Wouldn't mind hearing what you found confusing, so we can try to close any gaps

Anyone else using Caddy as a forward proxy.

https://github.com/caddyserver/forwardproxy

nginx config is only complicated if you want to take advantage of complex options. The config for a basic website is trivial.

Seems like a poor tradeoff for a slower, less configurable web server.

Unless you want a good HTTPS configuration. The output of https://ssl-config.mozilla.org/#server=nginx is significantly longer than the one for caddy.

FYI that website is outdated and unmaintained. The person who worked on it left Mozilla AFAIK (would love to be proven wrong). See https://twitter.com/CubicleApril/status/1438940565916225545

The correct thing with Caddy is to _not configure anything at all_ for TLS. The defaults are the correct thing to use. If you override the defaults, then you're more at risk of bitrot due to not remembering to update your own config. Let Caddy (and the Go stdlib, really) choose what's secure.

Nginx is still faster and that's what matters for highload. If you are not using the server for highload, no need for Nginx.

> I used nginx for a very, very long time but abandoned it.

I still think that Nginx is pretty good, however there are a few annoying things about it, certificates being just one of them.

Personally, I also found that attempting to use it as a reverse proxy kills the entire instance when there is no DNS record for one of the sites (say, 1 out of 20 that are proxied). I ran into this when running containers that hadn't passed health checks and therefore didn't have any traffic routed to them, which meant that if any of them went down, all of them would be unreachable. Furthermore, the popular suggestion of using a variable for the proxy URL just broke redirects in some apps: https://blog.kronis.dev/everything%20is%20broken/nginx-confi...

Caddy is pretty good, though I couldn't get the equivalents of all the options that are available in other web servers, for example allowing encoded slashes when hosting a Sonatype Nexus instance like what you can do with Apache. Things might have changed, but last I tried, it didn't quite seem to work: https://help.sonatype.com/repomanager3/planning-your-impleme...

Furthermore, there were issues with certificates as well: if the configuration for one of the aforementioned 20 sites was bad and the certificate couldn't be renewed, then the entire instance went down. What does this mean? Well, if I run Caddy as a container, I'll get the "fail fast" approach, which will sadly then mean that I'll get a restart, that will still fail with the bad configuration and eventually will hit Let's Encrypt rate limits. Sure, I can have some alerting (extra work) or have restart backoff/delays (though this will be bad for restart times if the instance were to ever crash for other reasons, e.g. load), but neither seems like an actual solution. Very annoying, especially because everything will once again go down, instead of being built for resiliency.

In the end, I kind of just went with Apache for the time being, because despite being a bit awkward at times, it's still an okay web server for the scales that I work at and is okay to deploy inside of containers. I wrote about it more on my blog, "How and why to use Apache httpd in 2022": https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-...

It even has Let's Encrypt (well, ACME) support built in, thanks to mod_md nowadays: https://httpd.apache.org/docs/trunk/mod/mod_md.html

Of course, I still use Nginx + PHP-FPM with supervisord for containers where I need to run PHP, because of some other weirdness when you try to get it working with Apache, which probably has something to do with how I build container images. That said, I couldn't actually find what the problem was, so it was easier for me to just use Apache on the edge and Nginx for the apps (PHP, or maybe even serving static assets), about which I wrote on yet another blog post, "Containers are broken": https://blog.kronis.dev/everything%20is%20broken/containers-...

In short, it's nice to have a choice of web servers and being able to pick whatever is the best suited for your needs. It's just that there's a lot of weirdness going on with what should be simple configurations.

FYI, I just revamped how Caddy handles encoded slashes: https://github.com/caddyserver/caddy/pull/4948

It's much smarter now.

Caddy also doesn't stop if a certificate can't be obtained. When did you last try? Maybe a big was fixed but it would have been YEARS ago.

Thanks for the response, that's really nice to hear!

I'll probably check out the more recent versions of Caddy as well, admittedly they're most likely a rather painless experience for most folks when compared with Apache/Nginx, though I've also heard good things about Traefik, which is even integrated in K3s as an ingress.

I guess most web servers are workable for a variety of use cases with a bit of effort, though I definitely reminded myself why people weren't exactly the biggest fans of the Apache config syntax just recently.

Would you mind to elaborate a bit more on the sophisticated configs?

automatic certs

multiple back end proxies

authorization sub requests

for multiple domains

CORS

and other stuff I can't recall offhand

Chef + reliable CAs other than Let's Encrypt that have an API = the end.

Invest in:

0. Automation

1. Documentation

2. Take away SSH

3. Except for data stores, ephemeralize servers so they can be blown away and automatically reprovisioned

4. Have tested backups (untested aren't backups)

5. Save time

6. Solve other, bigger problems

> reliable CAs other than Let's Encrypt that have an API

Would you care to give an example or two?

Let's Encrypt's ACME protocol has been adopted by ZeroSSL, Buypass and ssl.com to offer free 90-day certificates.

Free CAs are prone to fraud and abuse because they have no innate funding model and there's no barrier to anyone issuing a cert to any particular domain while attempting to circumvent domain authentication.

> there's no barrier to anyone issuing a cert to any particular domain while attempting to circumvent domain authentication.

Let's Encrypt has a much better track record at preventing people from issuing certs for random domains compared to many other commercial entities.

Certs? Dehydrated

Yeah, I feel certbot is a pain to use for anything which is not a toy. Dehydrated on the other hand works well in real setups.

Nginx bites you in the a**: you start your project, choose the most famous web server (nginx), then you build and build and start needing the pro features. But they don't give you an up-front price! You have to contact them for a custom quote. That's when you start pondering moving to Caddy. But you're too far down the rabbit hole. Luckily, Cloudflare does a decent job of providing the features you need, at a clear price.

Reply on news.ycombinator.com