Next Steps for the Caddy Project Maintainership
caddy.community
caddy.community
As a note, Caddy is one of those tools which hits the 80-90% of functionality with 50% of the complexity.
For both my homelab and hobby projects it just works. Its configuration is sane and well documented.
I highly recommend giving it a try.
Here’s a complete configuration file for a Wordpress site with a LetsEncrypt TLS cert, static files, and PHP executed via FastCGI:
example.com {
root * /var/www/wordpress
php_fastcgi unix//run/php/php-version-fpm.sock
file_server
}
I appreciate Caddy immensely. It doesn’t do anything Nginx can’t do, as far as I know, but it’s a whole lot more ergonomic while doing it.
Which is to say, in N years, will the Caddy defaults be full of some unfortunate decisions?
Caddy and Traefik have been around for a while now, so curious what has prevented the boring technology from essentially flipping some defaults around.
Admittedly there are some decisions we made with Caddy v2.0 that we would like to revisit eventually with a Caddy v3.0 in some future, but we haven't gotten to the point we've felt the need to plan that, most of those issues are minor enough that they haven't been deal-breakers. (And for context, v2.0 being a rewrite and rearchitecture from v0/v1 was necessary to unlock the potential that Caddy has realized today).
And those good defaults matter. If I pin down a set of TLS protocols today, will those still be good choices a couple of years from now? I don’t know. I’ll bet the then-current version of Caddy will still have good default settings. If HTTP/4 comes along, I suspect Caddy will configure it correctly on my site without me doing anything other than upgrading to a newer version, while other servers will want me to explicitly update their configs to enable it.
Similarly, nginx isn't going to be the one picking your SSL config out of the box; it's a bunch of arcane flags and settings (if you're not intimately familiar with SSL anyways) that you look up once how to get the SSLabs A+ score and then you put it in a reusable snippet to not have to think about that stuff ever again for future domains. If you want ACME certificates, you're gonna have to tell it to do that (nginx recently got a module for this I think, which kinda makes it equal in this sense to caddy's automatic LE cert generation) or where to find the certificates.
Caddy automates a lot of this sort of thing; the result is smoother/cleaner config files (it's hard to deny that reverse_proxy is shorter than the several dozens of config lines you'd use for nginx) but you also need to be aware of what Caddy does by default and the assumption that what the Caddy developers think should be the default will also always be what you want, unless you want to poke at manual overrides (in which case it's pretty much nginx again.) It can also do a few common variants on static site hosting, like automatically pulling a git repo, running a page generator on it and serving the resulting folder, which nginx just straight up doesn't do afaik.
In practice, I found Caddy works well on development/toy machines where there's no real expectations for having to think about prod and "not having to spend an hour reconfiguring your setup" when you start a project is beneficial and it's easy to swap things like ports around. Nginx works better on production machines because you basically set it up once and then can safely never have to think about it again, while still allowing you explicitly see why it's (not) doing what you want later down the line.
Traefik I legitimately couldn't see a reason to use however. It's config syntax is atrocious and looking it up online tells me it's mainly popular with non-technical homelab people because it can serve a web UI to manage your reverse proxies in. I guess that's a desirable feature for some?
One thing I did like about Apache back in the day was that it made it really easy to give per served web folder configuration. Nowadays I just toss stuff on a subdomain (cuz there's little reason not to), but if you only have one hostname to put things on (ie. a homelab thatt you can only access by local IP address), that's obviously not an option. .htaccess files were pretty neat for doing that.
Nginx can't really do that as easily, you have to start futzing with different location blocks and it kinda gets messy quickly.
On a not-so-relevant note, I do think Apache has probably the nicest "dirlist" out of all of them. Nginx's is really ugly and Caddy's feels too overdesigned.
Would I use it for a new service today? No.
Not because the configuration itself is complex (as I say, I've been working with it for decades), but because managing the config is complex.
You end up using custom templating systems, testing the new config oob, and then sending signals to the daemon. There's a huge pile of scripting needed to manage all of this in even the most basic way, let alone integrate it with a service mesh or orchestration system.
Even the ASF will push you towards ATS or Dubbo or APISIX rather than old-school Apache Server.
Caddy will get you up and running in seconds, whereas forcing Apache's square peg into the modern environment's round hole is going eat weeks for very little benefit.
As a side note, I see that Apache still describes it as "the most popular web server on the Internet", but that's probably not the case: https://w3techs.com/technologies/overview/web_server
To me, that has shades of XFree86 calling itself "the premier open source X11-based desktop infrastructure." Um...
server {
server_name example.com;
root /var/www/wordpress;
location ~ \.php(/|$) {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/run/php/php-version-fpm.sock;
}
ssl_certificate /etc/ssl/local/service.pem;
ssl_certificate_key /etc/ssl/local/service-key.pem;
}
No professional should care about a handful of extra lines. Anyway, in many real life situations, the config will be long and complex, whatever tool you use.
In the case above, the cert was created offline with the excellent mkcert from mkcert.dev, which is perfect for a developer machine. In other cases, I've had to use a certificate provided by my client. For the remaining cases, cerbot automates all, including nginx's config. Or if one installs the latest nginx, ACME cert retrieval is now included, so the difference to Caddy is shrinking.
I don't deny that Caddy is a worthy tool, but I don't care if it makes me write a few lines less in configuration files that I rarely write or update. Praise should focus on more important features. [edit] The excellent leadership shown in this post seems more important!
That "except" is doing a lot of lifting, in my opinion. Automatic Let's Encrypt is a big part of why I reach for Caddy. Install, run, done. No cert management headaches. It felt like magic the first time I used it, and now that I think of it, it still does.
Also, if I want to add another domain that should be accepted and reverse proxied to my application, in Caddy I just do this:
example.com wp.example.com caddyfreakingrules.example.com {
root * /var/www/wordpress
php_fastcgi unix//run/php/php-version-fpm.sock
file_server
}
Another thing, does nginx with the above configuration automatically load new certs if the ones that were there when the process spawned have since expired? Because not only does Caddy automatically renew the certs, it is handled transparently and there's zero downtime (provided nothing changes about the DNS pointers of course).
Caddy is freaking awesome!
Bonus, if this were your Caddyfile (the entire thing, this is all that's needed!):
{
admin off
auto_https prefer_wildcard
email hostmaster@example.com
cert_issuer acme {
dir https://acme-v02.api.letsencrypt.org/directory
resolvers 1.1.1.1 1.0.0.1
dns cloudflare {env.CLOUDFLARE_API_TOKEN}
}
ocsp_stapling off
}
example.com wp.example.com caddyfreakingrules.example.com {
root * /var/www/wordpress
php_fastcgi unix//run/php/php-version-fpm.sock
file_server
}
# This is simply to trigger generation of the wildcard cert without
# responding with the Wordpress application on all of the domains.
*.example.com {
respond "This is not the app you're looking for" 404
}
The result of which is that Caddy will only generate one cert for the above 3 sites, and Let's Encrypt won't leak the existance of the wp.example.com and caddyfreakingrules.example.com domains via certificate transparency.
But it’s not only the config that makes Caddy nice. I think that’s a good example of the kind of care and consideration that goes into Caddy’s development, though.
Automatic HTTPS, multiple domains, proxying specific routes to local services, etc etc, managed by one extremely legible config file.
I've had literally one service failure over that period, and it was my own error after running upgrades of the droplet's operating system.
Highly recommended.
Congrats to Mike on growing the project to the point where he can responsibly take a hand off the wheel now and then. And thank you!
It's actually Matt :)
(He and I were coworkers for a time.)
Hopefully some Mikes have been involved in the project as well and picked up some stray well wishes.
Anyway, I'm hoping this will help the project scale better on the development side. The community has shown that it can be responsible. Thanks for being a great part of it over the last 10+ years.
Caddy has been great!
I had in mind somehow requiring or rewarding everyone to pay a small amount. Like, anyone who uses software from a platform such as GitHub should pay a subscription fee for maintenance depending on usage. It could be small so that it doesn’t interfere with usage. Considering huge number of users, that could pay for maintainers.
The point is, there are large number of users, if each user pays a small amount to the platform, they won’t notice it, yet it accumulates to a maintenance fee.
Paying a maintenance fee makes sense, regardless of how you label it. The entirely free software has problems and may not be sustainable.
Sponsoring right away.
1. This may have been a misread on my part, on second read, gp's comment says nothing about mandatory payment. The reason I'm wary of mandatory payments is that you'd have to quantify who deserves what, and expend more time on the administration - likey setting up a foundation or company, or which not every F/OSS project needs. Again, I'm speaking generally - a Caddy Foundation, or Caddy Inc. may be appropriate. "Open source project" is a very broad term, ranging in size from leftpad to the Linux kernel, and the needs will be different for each.
(And this time with a solid option for companies too!)
> The Ultimate Server
> makes your sites more secure, more reliable, and more scalable than any other solution.
Is this an alternative to nginx or something?
A stand-out feature has been ACME support built-in, and it’s a fairly capable reverse proxy. I’ve seen organizations use Caddy to provision certificates for customer domains at scale with very good results.
Personally, I much prefer the way caddy does configuration / plugins (as someone reasonably conversant in how nginx does those things) - comparable to "sysv init scripts vs systemd unit files".
I have not configured lone servers in a long long time
There still remains this simple to reproduce bug where the page doesn't load of you use the full domain name of a site.
Still, only you and one other person with a similar grudge have ever complained about it (we've never had any github issues opened about it in years, neither on our forums) and nobody who cares has attempted to solve it with code changes.
Which is sad, as now I have to reconsider.
I’ve seen people mentioning about dot at the end of the domain a few times this year. Also never knew is a valid domain and should be able to resolve. Some people mentioned before YouTube.com. Won’t load ads. But I think they fixed.
Caddy always worked well and recommended to other people. So I’m a pro caddy user, don’t get me wrong.
We appreciate the recommendations! :)
Looks like https://github.com/caddyserver/caddy/issues/1632 to me?
Oh and now I see you work on the project. Hard pass on Caddy if this is how you respond to mild criticism.
In my own cases of responsibility, Caddy would have eliminated them had it been around. Instead I've learned to be paranoid, though having things like this are far better in terms of easing cognitive burden.
Cheers for all of the hard work by you and other maintainers.
I try to avoid engaging in online flame wars but I will say that the developers - including Francis - have been nothing but helpful and courteous to me personally and I've also learned a lot from their numerous positive contributions to Caddy-related forums.
You seem to think there is a conspiracy against Caddy. That seems doubtful. But I could see disproportionate defensiveness like what is on display here causing some people to not be fans.
From the outside, such a group mechanism is happening here. Your reaction to the issue being brought up is not rational, and from the outside not understandable. Even if it was brought up 18 month before in the same way. And no, the curl post does not match your hostility. And no, the tone did not came off as "pretty snide", even if tribal voting pattern greys out your opposition anyway.
Please reconsider your approach here. If the issue is not fixable in caddy (though Apache seems to handle it fine?) maybe formulate a standard response that says something like "sorry, this is complicated, see the curl explication here" and leave it at that. Have an action plan that recognizes group dynamics and the burnout symptoms existing in the project, as mentioned in the article.
If you want another vote for how annoying the trailing dot issue is in general, hear it from no other than the author of Curl, Daniel Stenberg himself https://daniel.haxx.se/blog/2022/05/12/a-tale-of-a-trailing-...
That is not a grudge. That is not slander. That is not a hill to die on. That is not an attack.
This makes me wonder how many other minor bugs are dismissed by you as a grudge due to you overreacting like this. It makes me a lot less confident in your project.
It’s perfectly fine for you to say ”this is low priority and we have no plans to fix it in the immediate future”. What’s not fine is treating it like a personal attack because they dared mention it twice in 18 months.
I really don't think it's fair for you to make a judgement on me or the project from an interaction like this. At least judge the project on its technical merits. I've been very transparent here. But I can't stop you from having your thoughts. It is what it is.
> we'd just like for it to stop being shoved in our face.
This is the comment you are referring to:
> There still remains this simple to reproduce bug where the page doesn't load of you use the full domain name of a site.
They aren’t asking you to repeat yourself. They aren’t shoving it in your face. This is an open discussion thread with many participants. They weren’t talking to you directly. This is information anybody here can find interesting and relevant. I did.
> I really don't think it's fair for you to make a judgement on me or the project from an interaction like this. At least judge the project on its technical merits.
How you are reacting to this is far more important to me than the original bug.
Remember when 37signals suffered data loss because they were using GET requests to delete things? When people pointed out they had a bug, they were offended and blamed GWA. What happened next? The same thing happened all over again, users suffered more data loss.
Or how about when Naomi Wu reported a problem with Signal, where the common use case of third-party keyboards for Chinese people was rendering all of their security worthless? They dismissed that as somebody with a grudge and ignored her for a year. What happened next? People found out that Chinese keyboards were compromised; she was 100% right, and Signal users were in danger.
I’ve seen what happens when people have this attitude towards inconvenient people reporting inconvenient bugs. It’s a danger to users, and you are making Caddy seem dangerous with this attitude. I was a happy user of Caddy right up until this thread, and even halfway down this thread – even after reading the mention of the bug – but your reaction has flipped that to the opposite because I can’t trust that there aren’t more bugs you are handling this way.
I know you've already made up your mind, but look at our track record of answering support questions on the forums and tickets on GitHub, and you'll see that the picture you've formed in your mind from this thread is not accurate.
Those comparisons are very straw-man and I won't entertain them. As I've already said, IMO there's more risk in introducing a new security bug in trying to fix this issue than there is leaving it as-is (failing fast and hard).
You are still locked into this idea that the sole purpose of bringing it up is for your response. This is an open conversation, not a dialogue between only you and them. It doesn’t matter if you have made your stance clear, them bringing it up gives other people a chance to hear about it and discuss it.
> I know you've already made up your mind, but look at our track record of answering support questions on the forums and tickets on GitHub, and you'll see that the picture you've formed in your mind from this thread is not accurate.
To be clear: my mind was made up that Caddy was a good, reliable choice, and it was your behaviour in this thread that changed my mind, it wasn’t my imagination.
> IMO there's more risk in introducing a new security bug in trying to fix this issue than there is leaving it as-is (failing fast and hard).
I believe that, but I also believe your attitude is a bigger threat to security than either.
> I believe that, but I also believe your attitude is a bigger threat to security than either.
I can't change your belief, nor do I care to, but I think that's absurd. Show me an actual security threat relating to this and I will address it. But this problem as stated is not one.
You keep saying that, but you did change my belief! My opinion is not immutable, I listen to what people say, and that is the reason we have ended up here. Because I listened to you and you convinced me to change my mind about Caddy.
> Show me an actual security threat relating to this and I will address it. But this problem as stated is not one.
“This problem” that I’m concerned with is your attitude not the FQDN bug, and I already gave the Signal example. When you start perceiving people reporting bugs as attacks and grudges, it makes it dangerously easy to dismiss real problems.
If that person found another problem with Caddy, I think they are less likely to report it to you because of this. If they did report it, I would think you are very likely to dismiss it because of who they are, not the contents of the bug report. This is a serious problem for my trust in Caddy.
An HN thread is not the place to report a bug. Nor do I think it's fair to form opinions about project maintenance (which doesn't happen on HN) based on comments in HN.
First, never used Caddy. I have no dog in this fight. I do manage a (closed source) project (which is decades old).
After reading this entire thread and all replies in all branches (I didn't vote on any of the comments), I think it would have been better for you not to reply at all. It would have done less damage to you (the negative emotions it brought out, the perception of others) and no one would remember that top-level comment. It would have been at the bottom of the page, an insignificant utterance. You elevated it, by protesting too much (I won't quote that famous line, but you get the idea).
And I must say you reminded me of my younger self, in the way you wouldn't let go of the issue and wouldn't let others have the last word. I've learned that this behavior is definitely self destructive and unproductive. The trigger was something that lived in me. It was never, I learned, about them. We choose how we respond. I've found one thing to work for me:
When an online discussion makes me emotional, I write a response in a text editor (not in the place where the comment is) and I let it sit for a few hours. Then, I do something completely different. I almost never post that comment, when I return to it, but I sometimes post something much softer. Mostly, I remove all emotion from the comment. Emotions are triggers for others, after all.
Why did I write so many words on this seemingly trivial online dispute? I hope I can help in some way, because I saw myself in your comments. Take them for what they are, me trying to help.
Given they're aware of previous discussion and the stance on the feature request, I don't think they're deterred by the discussion here. Your addition of fuel to fire here is the very thing that's not helping.
> If they did report it, I would think you are very likely to dismiss it because of who they are, not the contents of the bug report.
That's a huge assumption on your behalf.
Anyways, on the feature request, Caddy is not the only software who disagrees with it being valid, and curl had their back-and-forth on it. There's no legitimate bug being dismissed, and you can go through the issue tracker to audit it. Equating this discussion with 37signals or Signal is false equivalence.
Disclaimer: Caddy maintainer
They accused them of a grudge, an attack, slander, and shoving it in their face. For something as mild as this:
> There still remains this simple to reproduce bug where the page doesn't load of you use the full domain name of a site.
That’s a long way beyond exasperation, that’s a massive overreaction.
> That’s a long way beyond exasperation, that’s a massive overreaction.
Your reaction to Francis is _the_ overreaction. Francis simply said to OP to put their money where their mouth is. The "slander" comment comes later as a general statement on why this subject has become annoying.
Stop being hung up on Francis' response. The niche feature was discussed at length multiple times. You're welcome to search the web for all the conversations we had on the subject. Caddy has been around for 11 years. We've seen this subject more than you've seen it brought up. Again, OP referenced the discussion on the issue tracker in one of the earlier times they brought it up. They _admit_ it's niche. What's the point of continuously bringing it up?
This is what you’re referring to, right?
> > I think it's unfair to say that I post this every time when I've only mentioned it twice before, with the previous time being 2 years ago.
> Last year: https://news.ycombinator.com/item?id=39474419 and you also said "I have only brought this up once before on HN and it was over 2 years ago." in that same thread.
Okay, so the last time was 18 months ago not two years. But do you really think that mentioning it three times in 3.5 years can fairly be described as a grudge?
> Stop being hung up on Francis' response.
This is the only thing that matters to me in this thread. The bug itself is not that interesting. It’s a big deal to me that your team seems to take even the mildest mention of a bug as some kind of harassment. I’ve seen that kind of attitude before, and it’s dangerous.
You’ve explained why you won’t fix it (seems that you don’t feel you’ve got sufficient coverage to be confident to do so, or that it’s not possible to cover this type of change) so just put an faq up about it and do not take it personally. You cannot expect zero criticism.
This makes me wonder how many other discussion threads are wasted by this kind of complaints.
Overreaction to (alleged) overreaction never solves the problem!
My 2nd thought: Actually, this is very likely a culture/communication difference whereby both people care (I'm a big fan of Erin Meyer's work here)
My 3rd thought: I wonder what happens if I provide this repo and the chat comments to codex. Outcome: https://github.com/wsimmonds/caddy/pull/1
My 4th though: Perhaps I can make 'enemies' become friends if they both have disdain for AI ;)
Note: I would absolutely not submit this as-is. Caddy's an amazing project though I am not very familiar with its implementation and I'd seek to understand it, conventions etc. and make some obvious improvements to the code which has been generated - but this was a minor bit of fun. I created 4 separate versions and only in one of them did anything with TLS related get amended.
bananas
why is this your hill to die on?