Hacker News Reader

Comment by KRAKRISMOTT | Hacker News Reader

I wish Nginx would improve their developer experience. Their deployment patterns and configuration are horrid compared to Caddy which is amazing to use.

View on HN

Caddy has a lot of great features, but (at least as of last year) Nginx has the edge on documentation. Caddy’s docs were almost useless. Hopefully they’ve improved since I last used it.

Caddy cannot be found in the default repositories of Debian or RHEL. This raises the question of why one would use such a server. Personally, I am hesitant to download a random pre-built executable from Github, even if it is open source. I would much rather use the apt or dnf version, as anything else seems like just another toy server.

Debian's requirements for packaging of Go software is unreasonable. They expect every single dependency to be individually packaged. The total dependency chain of Caddy ends up being massive. We (the Caddy maintainers) don't have time necessary to allocate to a single distribution, to package and maintain every single dependency individually when all we want to do is ship a single static binary (plus some support files).

Instead, we ship with our own debian repo, hosting graciously provided by CloudSmith https://caddyserver.com/docs/install#debian-ubuntu-raspbian. This is packaged via CD with GitHub Actions, and you can verify the authenticity of the build since it's signed by Matt Holt's GPG key.

For RHEL, it's in COPR, and that's the best you'll ever get for similar reasons https://copr.fedorainfracloud.org/coprs/g/caddy/caddy/

Adding to Francis input, the release artifacts (not the .deb packages, which are signed with Matt's key) published on GitHub are authenticated with Sigstore tooling[0]. You can verify the artifacts and the .deb packages were not tampered to the byte! The builds are reproducible and verifiable. FUD doesn't have any room to loiter.

You can also build it from source using the `buildable` source archive artifact that includes all the deps so it can be built in air-gapped machine. Like its sibling artifacts, the source archive is signed, the signature is published, the signing certificate is available, and the checksum is published and also signed. What's so concerning?

[Disclaimer: Affiliated with Caddy]

[0] https://www.sigstore.dev/how-it-works

What's the reasoning behind that packaging requirement on Debian? Thanks for working on caddy by the way! I find it very neat.

Debian only ships free software (in main, but that's a detail).

This is actually enforced and there is processes in place to ensure that it stays that way.

This means that all new software that Debian packages is audited by a group of volunteers, the ftp-masters team, they check copyright, license and stuff like that.

If all binaries in Debian would vendor all of their dependencies, this would cause a lot extra and duplicated work for the ftp-masters, a team that already have a lot to do.

Same with security, if a popular go library needs to be patched to fix a security problem, then it's easier to do that in one place instead of patching it in N different binary packages.

Honestly, I don't understand it fully. I just know the barrier-to-entry is too high for us to spend time on it. We don't have contact with any debian packaging maintainers that would be willing to work with us. But https://go-team.pages.debian.net/packaging.html is one of my main resources for my understanding of their requirements.

And that goes without saying that Debian in general tends to release much slower than we'd be comfortable with. We don't want users running outdated and potentially insecure versions of Caddy. Best if users keep up to date by using a first-party installation method where we have control over the distribution pipeline.

All software in Debian needs to be Free software - the user must be able to modify and run it (ie recompile after modifying). And for software packaged for Debian that means being able to work with "apt-get source" and "apt-get build-deps". This of course includes dependencies.

That creates a bit of a split between Debian packages and language specific packages like rust crates, golang, python eggs or ruby gems.

There's some friction there, but the reasoning makes sense (but it is ok to disagree of course).

Isn't caddy fully free and open source software?

Certainly AFAIK. I didn't mean to imply otherwise. But FOSS as distributed in binary packages by Debian needs to remain possible to inspect, modify and build via the Debian (source) mirrors - hence all dependencies need to be packaged too (as opposed to living their seperate existence somewhere "go get" may be able to retrieve them from - or not ten years from now - and your respirator depends on a certain version of caddy for it's status display...).

Ahh that makes sense!! I get why Debian maintainers would want that, but it does seem quite hard to manage as a developer. I've went down the rabbit hole of how different Linux distros manage their repository packaging after your original comment so thanks for that :).

> Debian's requirements for packaging of Go software is unreasonable. They expect every single dependency to be individually packaged.

Based on the sibling comment that points out a volunteer has packaged caddy for Debian 12 - that work has been done?

>Caddy cannot be found in the default repositories of Debian or RHEL.

Debian 12 (bookworm) will have it: https://packages.debian.org/bookworm/caddy

FWIW, that was created by someone not affiliated with the Caddy project, and looks to no longer be maintained (latest is v2.6.4, but it has v2.6.2). So as a maintainer of Caddy, I cannot recommend using that repo.

This is the official Debian repository. The package versions are frozen in each major Debian release. However, they may backport security and bug fixes.

In practice, in the case of less popular packages, they do this on demand, when someone requests it in the bug tracker.

Well, users should know that if they report issues while using releases from that source, we can't reasonably help them, and that they should use an official release to get bug and security fixes promptly.

I want to emphasize that we have no contact at all with the people maintaining that Debian package, they've never reached out to discuss anything. We're absolutely open to that (and they know where to find us, not hard to contact us either on GitHub, Twitter, our forums, here, etc).

It's exactly the same way tens of thousands of other packages have been shipped for decades, including many other web servers like nginx, httpd, lighttpd. No need to paint so much drama over this.

They will contact you if the need arises. It's the same usual process that has been used since the 90s to great success.

Users will reach out to us first, not to debian, because we're easier to reach for help (via social or our forums). If they tell us they're using an outdated version which doesn't have the fix for what they need, I have no other choice but to tell them to stop using the debian-maintained package, and use our officially maintained package.

> Users will reach out to us first, not to debian, because we're easier to reach for help

Maybe. That is indeed a risk with third party distribution.

But do note that Debian has its own support channels, and infrastructure (like the "reporting" tool: https://packages.debian.org/stable/utils/reportbug ).

Oh please, you do have plenty of other choices.

It's ok to not want to support older versions or downstream packages (even if imo there is value in doing so) but don't be a drama queen and claim you can't.

Choices such as? How else would we get the user to run an updated release with the fixes they require?

Has anyone actually done any research on how good the backporting of security fixes is in frozen distros?

Maybe it's pretty good for very popular packages, but how about the more niche ones (and when it comes to Debian I'm not sure how popular Caddy is in their view)?

Anecdotally, my experience has been okay... but not great -- you can end up with something Frankenstein would create

The versions often feel arbitrary and don't line up. For example... I've been watching this for years:

https://bugs.launchpad.net/ubuntu/+source/firewalld/+bug/183...

This is more on the edge case side of things, too. Not really security patch related -- but a consequence of picking/choosing component levels

With this the firewall can randomly just stop being effective

When things aren't exactly upstream, the knives you're juggling get a little bigger and more unbalanced.

Mostly ok, sometimes terrible:

https://wiki.debian.org/SSLkeys#Technical_Summary

This comment is so out of touch with how Linux distributions work. This is the package most Debian users probably should be using, unless they absolutely require one of the newer versions.

IMO most users do require the newer versions because we made critical changes to how key things work and perform. I cannot in good faith recommend running anything but the latest release.

> we made critical changes

That's exactly why people (including me) tend to like LTS - no critical changes till next release. Upgrades for security with minimal surprises. I go further and often use unattended-upgrades on my Ubuntu fleet. I don't wanna version bumping until I explicitly ask for it as much as possible.

A lot of users require stability and this is how stable software distribution works. Only security fixes get backported, but no functional changes. It is unfortunate that Caddy hasn't adopted a segregated LTS and non-LTS approach, but that's not Debian's fault.

> we made critical changes

In two patch versions? With minor version unchanged?

While it is convenient to have software prebuilt in a trusted repo, these repos are more about providing toolchains. If something isn't in the repo (or the repo, as it often is, ie out of date) use the toolchain to build what you want.

Caddy provides their own yum repo and I'm pretty sure it's in EPEL too.

Just build it from source?

And then watch it like a hawk for vulnerabilities and rebuild as needed. No thanks.

The docs were brilliant for v1, it wouldn't surprise me if they were the spec for a great user experience and the code came second.

Despite v2 supporting a very similar config file, the documentation doesn't emphasise that and tries to steer you towards its API, confusing JSON config syntax etc.

It's still a very good web server for very few lines of config, but I don't relish trying to learn something new from its docs like I used to.

> and tries to steer you towards, its API, confusing JSON config syntax

I disagree. The docs don't do that. What you're probably talking about is the Getting Started guide https://caddyserver.com/docs/getting-started, which is a tour of how Caddy works, so it first shows you the "bare-metal" look at how it works, then it introduces the Caddyfile which allows you to simplify your user experience.

There's even a comparison table between the two https://caddyserver.com/docs/getting-started#json-vs-caddyfi... which explains when you'd want to use JSON (i.e. if you want programmable, API-based usage) or Caddyfile (i.e. for quick-and-simple hand-written config, 95% of users choose this).

I recommend starting from https://caddyserver.com/docs/caddyfile-tutorial or https://caddyserver.com/docs/caddyfile/concepts to get an idea of how the Caddyfile works.

That might be true now, but I spent a lot of time trying to use caddy v2 and not being able to since the only docs where for json and some things where available in caddyfile and some in json.

If it is fixed now then that is great, but it took at least a year (I'm guessing more) after the launch of v2.

Of course it wasn't going to be perfect at the initial release, nothing ever is. Caddy v2 was a complete rewrite from v1, so there was a lot of TODOs to polish it up.

Definitely take another look now, there's been a ton of progress since then, 3 years ago. The initial v2 release was in May 2020, soon after the pandemic hit.

> Of course it wasn't going to be perfect at the initial release, nothing ever is

What I'm trying to say is that on launch v2 was not a good replacement for v1, especially in the docs area. I've seen quite a few major version bumps in OSS, and it feels docs is an area that is usually neglected, and for quite a while (at least a year, I'd say more) the v2 docs where not useful for someone who had not participated in the caddy v2 community discussions.

I'm just trying to describe what led me to go from a avid caddy proponent back to an nginx user.

I'll take another look next time I have a project that needs a http/s server!

> Now you know that the Caddyfile is just converted to JSON for you.

> The Caddyfile seems easier than JSON, but should you always use it? There are pros and cons to each approach. The answer depends on your requirements and use case.

Followed by a table comparing the json and caddyfile approaches. What's the confusion?

> The docs were brilliant for v1, it wouldn't surprise me if they were the spec for a great user experience and the code came second.

I agree with this, v1 was an excellent user experience, I actually ran it on my servers for way too long. There was also the Wedge fork which might have helped with the EOL but sadly it didn't go anywhere: https://github.com/WedgeServer/wedge

> Despite v2 supporting a very similar config file, the documentation doesn't emphasise that and tries to steer you towards its API, confusing JSON config syntax etc.

Others responded to this a bit more, but while I agree that different config types are a confusing experience, at the same time I appreciate that they support something like that in the first place. I might not use it often, but it's nice that you can.

It's almost like the docs were written for someone upgrading from V1 (or familiar with V1) instead of a newcomer who knows nothing (like the V1 docs were written for).

That's not true. The docs are written with the expectation that the user understands how the web works. We can't reasonably teach that in our docs. Instead, users should read MDN for that stuff. If you were coming from v1, the only page that makes that assumption is the upgrade guide https://caddyserver.com/docs/v2-upgrade. Everything else is either a getting started guide, a tutorial, or reference docs for Caddyfile and JSON config.

I didn't say it was true, I said it just seems like it. As an example: https://caddyserver.com/docs/caddyfile/directives/php_fastcg... -- it shows the syntax, but nowhere on the page does it tell you WHERE to put it in the config file. Is it top-level? Do I nest it in something else? Keep in mind, most people are starting with a mostly blank file, and zero context about how Caddy works (whether or not they understand how the web works). This page won't answer the basic questions of where it is allowed, and neither will the getting-started docs, which tells you to make a json file instead of a Caddyfile but the docs for the thing I looked up doesn't look like json (maybe I know this, maybe I don't). It's all very confusing for someone looking in the docs for a solution and instead has to learn how everything works, whether they want to or not.

That's what this page is for: https://caddyserver.com/docs/caddyfile/concepts#structure. You shouldn't be reading directive docs before the concepts page. The directives page https://caddyserver.com/docs/caddyfile/directives at the top also tells you where these go, which is where you must have came from to find that directive's page in the docs. The information is all there, you just need to read it. Read things in order, don't skip steps; don't skip steps in the getting started guide either, that defeats the purpose of the tour it gives you.

Google is not going to take me to that page first when I search for how to do something in Caddy. It's going to take me to the directive, most likely.

To say it again:

> It's all very confusing for someone looking in the docs for a solution and instead has to learn how everything works, whether they want to or not.

How can we fix this without repeating the Concepts page on every page of our docs?

Maybe we should just un-index all docs pages except the intro pages.

I think there are several ways to handle that. Assuming you are using templates for the docs:

1. A simple link like this: https://docs.k3s.io/installation/configuration#:~:text=For%2....

2. An aside at the top that informs the person where to get more information with a link.

3. An example showing the structure of the file.

4. A link to examples/tests in the repo showing how it can be used.

I would think that a simple aside in your template would work wonders. Maybe saying something like:

> See our page on [directives](link) to learn how to best use this in your configuration.

Couldn't agree more, I absolutely LOVE Caddy, but the docs were truly awful the last time I had to look, all forums, etc also referenced v1 a lot which was really frustrating.

I think we've improved them a lot since then. A new website is in the works which should improve them even more.

Personally I love the nginx configuration, I find it intuitive and simple to work with. I for example use it to proxy traffic to linux network namespace or unix sockets. Nginx also have great performance so I use it to offload SSL/TSL connections. It also runs on few resources and I've used it on tiny VPS' with only 128MB of memory.

I like the general syntax but some things are just annoying or at least surprising like add_header replacing all headers added in parent scopes which would otherwise be inherited.

Speaking of documentation, there’s this weird edge case where =404 in try_files causes $args to no longer work(!) and I ended up wasting an entire day on it.

It’s too bad, I only moved away from caddy because of weird disconnection issues when reverse proxying certain apps.

I'm not sure I understand. Please write a bug report if you think you found a problem https://github.com/caddyserver/caddy/issues

Reply on news.ycombinator.com