Hacker News Reader

Comment by mholt | Hacker News Reader

Is this a good opportunity to plug an open source project I've been working on for 5 years that basically solves this problem? https://caddyserver.com will keep your certs renewed for free, automatically, without extra tooling, dependencies, or moving parts. I really hope more sites will use it because this stuff happens.

If you've heard of Caddy but haven't used Caddy 2 yet, we've made some huge improvements with it, and it's capable of managing tens of thousands of certificates at a time. As of next week's beta it can handle certificates with lifetimes as short as a few minutes. It works as a load balancer, in a cluster behind load balancers, and with Docker with its new dynamic config API. There's no technical reasons I know of why sites like TechCrunch can't use it.

View on HN

What does it do that certbot doesn't? Personally, I don't see the benefit of bundling certificate renewals with the HTTPS server.

A quick perusal of the Let's Encrypt forums will reveal a few issues. For one, it's easy to misconfigure. There's a lot of surface area it has to cover in the web server and there's a lot that can go wrong. It can also be tricky to install since it is separate from your web server and has a number of complex depeneldency requirements (like many large Python programs). And because it's an external dependency, there's no way the server can react to errors in CertBot, only CertBot can control the server, so you don't get the benefit of duplexed interactions.

Caddy is written in Go, a memory safe language. I cannot overstate how significant it is that most servers like ngixn, Apache, and HAProxy are written in C and cannot offer the same security guarantees.

Caddy will also staple OCSP, in the most robust way compared to other servers. It has weathered outages that took Apache and NGINX sites down (in Firefox).

We've also seen CertBot consume high amounts of resources at scale, whereas Caddy can handle thousands of certs no problem. That's why some companies have talked to me about why they switched.

Caddy can coordinate cert management in a cluster. This happens automatically when configured with the same storage backend. CertBot doesn't do that. Caddy works great behind reverse proxies, or as the reverse proxy.

Perhaps most importantly, Caddy is the only server to use HTTPS by default without needing any explicit configuration. You simplify your deployment workflows and have less room for things to go wrong.

There's also a values statement here... If you think privacy and security are important, you choose software that enables privacy by default because it aligns with your values.

Sure, any number of solutions can get you TLS and a few even get you TLS via ACME, but Caddy (2) is highly optimized to handle the edge cases and scaling requirements a lot of sites have these days.

I don't meant to pooh-pooh your project, but Go's memory safety isn't some kind of automatic security guarantee... especially when it comes with a giant downside of having to run as root. Every other production-grade server drops privileges as soon as possible, for a very good reason.

Go does bounds checking and doesn't have dangling pointers, so it's inherently safe from buffer overflows.

You don't have to run Caddy as root in production, even to bind to low ports. (None of my sites do.) https://memorysafety.fail for more info on memory unsafety.

Thank you for your condescending reply. I understand Go's memory model quite well (and have used Go quite a bit). News flash: you can have bugs of other kinds besides buffer overflows.

Er, sorry, I wasn't being condescending. I intended to clarify some relevant facts, please don't take it personally.

You had said:

> Go's memory safety isn't some kind of automatic security guarantee...

But in some cases, it is. I'm not claiming more than that, nor am I saying that you can't have "bugs of other kinds" -- on which point I agree with you.

> Er, sorry, I wasn't being condescending. I intended to clarify some relevant facts, please don't take it personally.

Fair enough, it's easy to misunderstand the intended tone in online comments.

Why would a Go server need to run as root?

Asking because I run Go based servers (in production) on Linux, and they're not running any of the Go processes as root.

https://github.com/caddyserver/caddy/issues/528

It seems like they've fixed their systemd config though to run as a non-privileged user, so that shouldn't be an issue. But it seems like Go processes can't drop privileges because of an open issue with the runtime.

Actually, it is because of a long-standing issue with Linux.

* http://jdebp.uk./FGA/linux-thread-problems.html

It is interesting to read https://github.com/caddyserver/caddy/issues/528#issuecomment... , by the way, and see how all of this was later removed because it did not work with older versions of systemd. Notice the reviewer comments at https://github.com/caddyserver/caddy/pull/1127 .

Fail to start if it can't reach the ACME server https://github.com/caddyserver/caddy/issues/1680

That was 3 years ago when ACME was still a new and unstandardized protocol. That was one of the worst days of my life, trying to fix that while finishing my NIPS paper. If you read the issue you'll see that we fixed that and learned from it.

> There's no technical reasons I know of why sites like TechCrunch can't use it.

Of course there is - a web server that gets a certificate via LE is a single screw. A commercial internet-facing site is a high rise. One does not just replace a screw in a high rise -- it requires ensuring that the right tools are in every toolbelt to manipulate the screws. For Caddy to be useful in a non- oh look ma, i'm on the internet projects, it needs to fit into the tooling system.

For example, it should be fetching storing certificates in a database that can be accessed by multiple servers. Rather than doing everything behind the scenes, it should be emitting events indicating its desire to renew a cert and let someone else handle ( maybe a different caddy ) handle the renewals, etc, etc, etc.

Oh I understand that, believe me. That's an operational reason, though, and not as much a technical one.

By the way, Caddy CAN store certs in a database accessed by multiple servers. It coordinates cert management when configured as a cluster, automatically. Caddy is the only server that does this (for free).

> That's an operational reason, though, and not as much a technical one.

Every single operational reason is a technical reason if it touches operation of the technology.

> By the way, Caddy CAN store certs in a database accessed by multiple servers. It coordinates cert management when configured as a cluster, automatically. Caddy is the only server that does this (for free

An edge web server should carry no knowledge of a concept of a cluster. It should simply be able to ask something for a cert. Otherwise it is a screw that insists that in order for it be used in high rise, it needs to be used in a specific pattern that the screw designer envisioned.

If you want to make Caddy easily adoptable in high rises I suggest the following path:

1. Subscribe to events saying "fetch this cert from here"

2. Emit events saying "Need to renew X" when you need to renew.

3. Create ability to forward renew requests to renew requests processor.

4. Make request processor update the cert store and emit the event that the key pair is available.

> An edge web server should carry no knowledge of a concept of a cluster.

Caddy doesn't. That's what's so interesting!

> It should simply be able to ask something for a cert.

It does.

> I suggest the following path:

Oh good! That's pretty close to how it actually works already!

I'm happy to explain more how Caddy actually works before we start armchairing solutions here. Or, feel free to read the code for yourself: https://github.com/mholt/certmagic/

I think the problem that most people have is that they have no way to orchestrate multiple components, so they really want everything built into one binary.

Once you can run multiple processes and have service discovery, the need for this stuff goes away. For example, I use cert-manager to get certificates for my Envoy edge proxy. cert-manager runs continuously and updates the TLS certificates in a Kubernetes secret. Kubernetes injects the secret into the Envoy container as files, and then Envoy reads them without any knowledge of where they come from or how they can be renewed. (It could also be configured to get them at runtime with xDS / Universal Data Plane API). This is all very easy and provides infinite interchangeability; there would be no change in cert provisioning necessary if I switched to Nginx or HAProxy.

But, most people don't have a setup that supports something like this, so they really need the all-in-one thing or they're just going to say "we don't really need TLS". They get by because nothing in their infrastructure ever changes; certs are new because they require operator intervention every 90 days and that is a new experience. It is unfortunate, but that's where most of the world is at right now.

Agreed.

However if an author of the solution does insist on implementing it in one binary with orchestration step being in the code rather than pulled out into the infrastructure management, the author should not be surprised that no one outside tiny single server operations would consider it to be a contender.

Your logic assumes that Caddy's auto-HTTPS cannot be configured and must be self-contained, but that's not accurate. Caddy can be configured with traditional, manually-managed (or externally-managed) TLS certificates just like other web servers too, if you need that.

All we're saying with this project is, "There is a better way going forward, and we're solving that problem."

You are presuming that as a designer of the Caddy you understand how the non-single-VM customers are running their infrastructure.

Here's a typical problem of a distributed entry point system - network partitioning.

* If your key/cert store relies on key servers being always accessible from everywhere, it simply won't work.

* If your solution is to bake a key into a file system, then it is no better than existing nginx implementation.

Basically, if you want your solution to be adopted, it needs to be significantly better than the entrenched ones.

Yeah, I think that's a fine analysis. I don't think Caddy cares that it's not the "right way" if it solves problems for some people. It is certainly easier than certbot and nginx for cases where you have one VM that you play with as though it's a pet. And a lot of people treat their production environments that way, so it does solve a problem.

The next problem that people will run into is that because they can't reliably run multiple "things", they won't have a monitoring stack to tell them that certs aren't renewing. But, maybe nothing bad will happen and it won't be a problem. "Hope is not a strategy" if you're a Google SRE, but people do pretty OK with hope in the real world ;)

> The next problem that people will run into is that because they can't reliably run multiple "things", they won't have a monitoring stack to tell them that certs aren't renewing.

Can you elaborate on this? I'm not sure I follow; there's nothing preventing people from using something like https://ohdear.app/ (or any other monitoring tool) with a self-contained solution like Caddy. In fact, I recommend that people do.

No, something like that is fine. I am just trying to convey that there is a huge industry around fixing these problems very incrementally. ohdear.app today, then DataDog... all because they can't just run Prometheus and Alertmanager.

This isn't how I like it, but I totally see the use case. Caddy is a part of that use case and is probably the reason it exists. I think that's fine!

(Edit: ohdear is really neat though, I think people might want that no matter what. You have to serve your status page from somewhere, might as well get some cert checks thrown in for free!)

Just to clarify, Caddy v1 exports metrics to Prometheus. V2 could easily do the same, it's just as extensible as v1.

https://caddyserver.com/v1/docs/http.prometheus

I am not talking about Caddy, I'm talking about the thought process that leads to needing everything integrated in one process.

> The next problem that people will run into is that because they can't reliably run multiple "things", they won't have a monitoring stack to tell them that certs aren't renewing. But, maybe nothing bad will happen and it won't be a problem. "Hope is not a strategy" if you're a Google SRE, but people do pretty OK with hope in the real world ;)

Sort of agreed with this.

Based on my experience, the reality is a bit more complicated. As soon as one is past the "this VM is a special pet and it is the only one I have" stage in my experience even smaller companies get dozens and soon hundreds of distinct HTTPs entry points:

* Main site/API ( code )

* Ops support

* bizops ( invoicing/quoting )

Which is where the issues of monitoring and observability come up.

I bet techcrunch has close to a hundred HTTP/HTTPS entry points and every time they evaluate a solution to replace their current entry point structure they ask if the new method would at least solve all of their existing problems. Ops people aren't gong to add another partial solution so the mix.

> I bet techcrunch has close to a hundred HTTP/HTTPS entry points

I wouldn't be so sure about that. It's certainly possible that's the case but equally I've personally seen new sites on Techcrunch's scale literally served from a couple of a couple of boxes running Wordpress (I kid you not!) which sits behind a few more boxes running pretty aggressive Varnish profiles and then that is sat behind Akamai's CDN. Most of the content on news sites is static and highly cache-able so there isn't always the need to go down the micro-services route.

I think you are thinking strictly about an origin for techcrunch.com and excluding all the other services that spring up internally to run any reasonably sized property.

For example, does it run RabbitMQ anywhere? Rabbitmq admin front-end is HTTP. Same goes for Kafka and other applications and we aren't even touching bizops.

I'm not going to argue that you're wrong nor that I have much experience of Techcrunch aside the odd link posted on here but it's worth noting that you don't actually need all of that stuff and thus many online new sites don't. Plus sometimes those other resources might be managed by other teams where they follow different deployment models / best practices / etc than those who manage the public news site (team responsibilities are often messy and illogical)

My point isn't to say you're wrong, just that there will be as many exceptions to your point as there are occasions it's accurate.

There's also Traefik if all you want is a load balancer for services run Docker containers. It does Lets Encrypt as well.

Right now I use the official HAProxy container + the official certbot container and a script that renews all my domains and then sends a sig-HUP to HAProxy to reload the certs. It mostly works but there have been a few hickups (HAProxy not starting because of the README file in the live LetsEncrypt folder; however if you send an HUP, it ignores it fine).

Has anyone used Traefik? Is it worth switching to over HAProxy+Certbot?

https://docs.traefik.io/

I use Traefik with Kubernetes clusters via Helm chart. Works amazingly. Just define a ingress resource and boom, everything just works. Note, if doing on-prem Kubernetes and cloud native load balances are not available, highly recommend the MetalLB[1] project. Traefik requires a load balancer resource in front of it.

[1] https://metallb.universe.tf/

Unless I'm overlooking the obvious -- and I apologise in advance if I am -- but you don't clearly state who you are using as your CA.

It's Let's Encrypt. Mentioned in the Features sections: Security and Privacy > Certificates > AUTO OBTAIN

Ah yes, I see it now. That's buried deep though. Definitely not obvious.

It's because most people go to our website already knowing they need to use Let's Encrypt, and they've found it by searching for a server that works with Let's Encrypt. Hence, we don't make that detail our front-and-center. Instead, we're targeting people who don't already know what Caddy is. The information is readily available in the docs.

That's clearly not always the case otherwise we wouldn't be having this conversation to begin with.

Also I'm not suggesting you make it front and centre but as it is currently there's only one mention on your landing page and it's buried so far deep it might as well not be mentioned at all. I did also look through some of the other docs and still missed it. If I missed it you can bet others have too. All I'm suggesting is that it's worth being a little more explicit about that because that will be a key detail for some people.

You mention about it being written in Go. That seems less relevant (in my personal opinion) but that's a great place to add a line saying something like

"Caddy uses the ACME protocol to automatically manage secure certificates issued by Let's Encrypt."

Caddy works with more than just Let's Encrypt though, and we'd like to avoid encouraging a monoculture of automated CAs (as much as I love Let's Encrypt).

> That's clearly not always the case otherwise we wouldn't be having this conversation to begin with.

Well, sometimes we forget that HN is not actually the center of all new tech. ;) This discussion is a bit of an exception. Most of our leads don't come from HN.

But I will see what we can do to make that more obvious once we're out of beta.

> Caddy works with more than just Let's Encrypt though, and we'd like to avoid encouraging a monoculture of automated CAs (as much as I love Let's Encrypt).

Which is good but now we're back to my original question: "What CA's does Caddy support?"

It shouldn't be that hard of a question to find the answer to but it seems even you can't give me a straight answer and that's really off putting.

> Well, sometimes we forget that HN is not actually the center of all new tech. ;) This discussion is a bit of an exception. Most of our leads don't come from HN.

You were the one who posted the link! If the landing page isn't designed for HN audiences then maybe that's not the link you should have posted on HN?

I know it's hard taking what seams like criticisms when it's your own projects (I maintain popular open source projects too so have experienced this first hand many times myself) but sometimes it pays not to default on the defensive. I'm not saying you have to listen to nor agree with all feedback given -- and obviously I don't expect you to entertain people who are just rude or demanding -- but it's an order of magnitude harder for people to raise legitimate constructive comments or genuine questions / concerns if they're back-footed.

> "What CA's does Caddy support?"

From the site: "Any ACME-compatible CA can be used." Relevant config parameter: https://caddyserver.com/docs/json/apps/tls/automation/polici...

> It shouldn't be that hard of a question to find the answer to but it seems even you can't give me a straight answer and that's really off putting.

Um, my bad? I said I would try to improve this when I have time.

> You were the one who posted the link!

And the HN commenters proceed to think they are the central source of a project's vitality and popularity. I can post the link, but that is not where most of Caddy's traffic comes from. A bit self-aggrandizing of the HN crowd to assume that their opinion is the one true way; but this is true of most mob mentalities.

> but sometimes it pays not to default on the defensive.

Ah, so you think I'm being defensive... I see... I guess it is hard these days to have a discussion of facts and ideas without thinking about it in terms of attack/defense.

Anyway... I admit, I digress into lamenting the state of the HN community, which is off-topic.

> You were the one who posted the link! If the landing page isn't designed for HN audiences then maybe that's not the link you should have posted on HN?

The logical extreme of this statement is that @mholt shouldn't post a link to any website unless that link is specifically tailored to the average reader of the site he's posting to. That, or Hacker News is special among all websites @mholt could post to.

I don't think that's fair. I also don't see the defensiveness you see - instead, I see @mholt explaining his website's strategy for the benefit of your understanding (as well as that of any future readers). The alternative to which would be not responding to your feedback at all, as he already has sound reasoning not to incorporate your specific suggestion (which we know because he explained it).

It's important to read into the best possible interpretation of a comment and respond to that, assuming good faith, especially on communities like this one. Otherwise we begin to assume everyone is attacking or defending.

I wasn’t just basing my comment off his responses to me, his tone was similar through this thread and on the Github issue tracker as well (I later saw others have commented on it too). But it’s possible that’s just his writing style and we are reading too much into it and if that’s the case then I’m sorry for doing so.

Regarding the link point. I don’t agree. If you purposely post a promotional link saying “use my tool” to a specific forum then you can’t really backtrack and say “you’re not my intended audience for this page” when people raise questions based on incomplete information published to that link. That’s just bad product advertising. Or at the very least, you should add a disclaimer saying “this is normally a manager link (etc)”.

As it happens, I am actually the target audience for that landing page because I am a tech lead responsible for making architecture-based decisions and the number of HTTP end points we have is few because that’s not the main side of our business (so certs often get forgotten about). That’s why I was asking the questions I was asking.

It's Let's Encrypt. :) https://caddyserver.com/docs/automatic-https

Is this the one that refuses to start if there is a valid cert but it cant renew it?

Nope, Caddy issues certs in the background and will retry with exponential backoff for up to 30 days. It reverts to the staging endpoint during retries so it doesn't count against your rate limits. It also has internal throttling to allow multiple challenges to take place at once (in most cases), so it can obtain several certificates per second as CA rate limits allow, being one of the highest-performing ACME clients available, and one of the most robust, too.

https://caddyserver.com/docs/automatic-https#errors

Also, yes, it's the one that had this issue (presumably this is what plopz had in mind https://github.com/caddyserver/caddy/issues/1680, https://news.ycombinator.com/item?id=14375022)

Caddy is not the only one, though. Traefik had the same problem: https://github.com/containous/traefik/issues/791 So to say Caddy is "the one" that had that issue is a mis-characterization.

Reply on news.ycombinator.com