Hacker News Reader

Comment by nixcraft | Hacker News Reader

Caddy cannot be found in the default repositories of Debian or RHEL. This raises the question of why one would use such a server. Personally, I am hesitant to download a random pre-built executable from Github, even if it is open source. I would much rather use the apt or dnf version, as anything else seems like just another toy server.

View on HN

Debian's requirements for packaging of Go software is unreasonable. They expect every single dependency to be individually packaged. The total dependency chain of Caddy ends up being massive. We (the Caddy maintainers) don't have time necessary to allocate to a single distribution, to package and maintain every single dependency individually when all we want to do is ship a single static binary (plus some support files).

Instead, we ship with our own debian repo, hosting graciously provided by CloudSmith https://caddyserver.com/docs/install#debian-ubuntu-raspbian. This is packaged via CD with GitHub Actions, and you can verify the authenticity of the build since it's signed by Matt Holt's GPG key.

For RHEL, it's in COPR, and that's the best you'll ever get for similar reasons https://copr.fedorainfracloud.org/coprs/g/caddy/caddy/

What's the reasoning behind that packaging requirement on Debian? Thanks for working on caddy by the way! I find it very neat.

Honestly, I don't understand it fully. I just know the barrier-to-entry is too high for us to spend time on it. We don't have contact with any debian packaging maintainers that would be willing to work with us. But https://go-team.pages.debian.net/packaging.html is one of my main resources for my understanding of their requirements.

And that goes without saying that Debian in general tends to release much slower than we'd be comfortable with. We don't want users running outdated and potentially insecure versions of Caddy. Best if users keep up to date by using a first-party installation method where we have control over the distribution pipeline.

All software in Debian needs to be Free software - the user must be able to modify and run it (ie recompile after modifying). And for software packaged for Debian that means being able to work with "apt-get source" and "apt-get build-deps". This of course includes dependencies.

That creates a bit of a split between Debian packages and language specific packages like rust crates, golang, python eggs or ruby gems.

There's some friction there, but the reasoning makes sense (but it is ok to disagree of course).

Isn't caddy fully free and open source software?

Certainly AFAIK. I didn't mean to imply otherwise. But FOSS as distributed in binary packages by Debian needs to remain possible to inspect, modify and build via the Debian (source) mirrors - hence all dependencies need to be packaged too (as opposed to living their seperate existence somewhere "go get" may be able to retrieve them from - or not ten years from now - and your respirator depends on a certain version of caddy for it's status display...).

Ahh that makes sense!! I get why Debian maintainers would want that, but it does seem quite hard to manage as a developer. I've went down the rabbit hole of how different Linux distros manage their repository packaging after your original comment so thanks for that :).

Debian only ships free software (in main, but that's a detail).

This is actually enforced and there is processes in place to ensure that it stays that way.

This means that all new software that Debian packages is audited by a group of volunteers, the ftp-masters team, they check copyright, license and stuff like that.

If all binaries in Debian would vendor all of their dependencies, this would cause a lot extra and duplicated work for the ftp-masters, a team that already have a lot to do.

Same with security, if a popular go library needs to be patched to fix a security problem, then it's easier to do that in one place instead of patching it in N different binary packages.

Adding to Francis input, the release artifacts (not the .deb packages, which are signed with Matt's key) published on GitHub are authenticated with Sigstore tooling[0]. You can verify the artifacts and the .deb packages were not tampered to the byte! The builds are reproducible and verifiable. FUD doesn't have any room to loiter.

You can also build it from source using the `buildable` source archive artifact that includes all the deps so it can be built in air-gapped machine. Like its sibling artifacts, the source archive is signed, the signature is published, the signing certificate is available, and the checksum is published and also signed. What's so concerning?

[Disclaimer: Affiliated with Caddy]

[0] https://www.sigstore.dev/how-it-works

> Debian's requirements for packaging of Go software is unreasonable. They expect every single dependency to be individually packaged.

Based on the sibling comment that points out a volunteer has packaged caddy for Debian 12 - that work has been done?

>Caddy cannot be found in the default repositories of Debian or RHEL.

Debian 12 (bookworm) will have it: https://packages.debian.org/bookworm/caddy

FWIW, that was created by someone not affiliated with the Caddy project, and looks to no longer be maintained (latest is v2.6.4, but it has v2.6.2). So as a maintainer of Caddy, I cannot recommend using that repo.

This is the official Debian repository. The package versions are frozen in each major Debian release. However, they may backport security and bug fixes.

In practice, in the case of less popular packages, they do this on demand, when someone requests it in the bug tracker.

Well, users should know that if they report issues while using releases from that source, we can't reasonably help them, and that they should use an official release to get bug and security fixes promptly.

I want to emphasize that we have no contact at all with the people maintaining that Debian package, they've never reached out to discuss anything. We're absolutely open to that (and they know where to find us, not hard to contact us either on GitHub, Twitter, our forums, here, etc).

It's exactly the same way tens of thousands of other packages have been shipped for decades, including many other web servers like nginx, httpd, lighttpd. No need to paint so much drama over this.

They will contact you if the need arises. It's the same usual process that has been used since the 90s to great success.

Users will reach out to us first, not to debian, because we're easier to reach for help (via social or our forums). If they tell us they're using an outdated version which doesn't have the fix for what they need, I have no other choice but to tell them to stop using the debian-maintained package, and use our officially maintained package.

Oh please, you do have plenty of other choices.

It's ok to not want to support older versions or downstream packages (even if imo there is value in doing so) but don't be a drama queen and claim you can't.

Choices such as? How else would we get the user to run an updated release with the fixes they require?

> Users will reach out to us first, not to debian, because we're easier to reach for help

Maybe. That is indeed a risk with third party distribution.

But do note that Debian has its own support channels, and infrastructure (like the "reporting" tool: https://packages.debian.org/stable/utils/reportbug ).

Has anyone actually done any research on how good the backporting of security fixes is in frozen distros?

Maybe it's pretty good for very popular packages, but how about the more niche ones (and when it comes to Debian I'm not sure how popular Caddy is in their view)?

Anecdotally, my experience has been okay... but not great -- you can end up with something Frankenstein would create

The versions often feel arbitrary and don't line up. For example... I've been watching this for years:

https://bugs.launchpad.net/ubuntu/+source/firewalld/+bug/183...

This is more on the edge case side of things, too. Not really security patch related -- but a consequence of picking/choosing component levels

With this the firewall can randomly just stop being effective

When things aren't exactly upstream, the knives you're juggling get a little bigger and more unbalanced.

Mostly ok, sometimes terrible:

https://wiki.debian.org/SSLkeys#Technical_Summary

This comment is so out of touch with how Linux distributions work. This is the package most Debian users probably should be using, unless they absolutely require one of the newer versions.

IMO most users do require the newer versions because we made critical changes to how key things work and perform. I cannot in good faith recommend running anything but the latest release.

> we made critical changes

That's exactly why people (including me) tend to like LTS - no critical changes till next release. Upgrades for security with minimal surprises. I go further and often use unattended-upgrades on my Ubuntu fleet. I don't wanna version bumping until I explicitly ask for it as much as possible.

A lot of users require stability and this is how stable software distribution works. Only security fixes get backported, but no functional changes. It is unfortunate that Caddy hasn't adopted a segregated LTS and non-LTS approach, but that's not Debian's fault.

> we made critical changes

In two patch versions? With minor version unchanged?

While it is convenient to have software prebuilt in a trusted repo, these repos are more about providing toolchains. If something isn't in the repo (or the repo, as it often is, ie out of date) use the toolchain to build what you want.

Caddy provides their own yum repo and I'm pretty sure it's in EPEL too.

Just build it from source?

And then watch it like a hawk for vulnerabilities and rebuild as needed. No thanks.

Reply on news.ycombinator.com