Comment by notyourday | Hacker News Reader

Comment by notyourday | Hacker News Reader

Parent Full thread

> That's an operational reason, though, and not as much a technical one.

Every single operational reason is a technical reason if it touches operation of the technology.

> By the way, Caddy CAN store certs in a database accessed by multiple servers. It coordinates cert management when configured as a cluster, automatically. Caddy is the only server that does this (for free

An edge web server should carry no knowledge of a concept of a cluster. It should simply be able to ask something for a cert. Otherwise it is a screw that insists that in order for it be used in high rise, it needs to be used in a specific pattern that the screw designer envisioned.

If you want to make Caddy easily adoptable in high rises I suggest the following path:

1. Subscribe to events saying "fetch this cert from here"

2. Emit events saying "Need to renew X" when you need to renew.

3. Create ability to forward renew requests to renew requests processor.

4. Make request processor update the cert store and emit the event that the key pair is available.

> An edge web server should carry no knowledge of a concept of a cluster.

Caddy doesn't. That's what's so interesting!

> It should simply be able to ask something for a cert.

It does.

> I suggest the following path:

Oh good! That's pretty close to how it actually works already!

I'm happy to explain more how Caddy actually works before we start armchairing solutions here. Or, feel free to read the code for yourself: https://github.com/mholt/certmagic/

I think the problem that most people have is that they have no way to orchestrate multiple components, so they really want everything built into one binary.

Once you can run multiple processes and have service discovery, the need for this stuff goes away. For example, I use cert-manager to get certificates for my Envoy edge proxy. cert-manager runs continuously and updates the TLS certificates in a Kubernetes secret. Kubernetes injects the secret into the Envoy container as files, and then Envoy reads them without any knowledge of where they come from or how they can be renewed. (It could also be configured to get them at runtime with xDS / Universal Data Plane API). This is all very easy and provides infinite interchangeability; there would be no change in cert provisioning necessary if I switched to Nginx or HAProxy.

But, most people don't have a setup that supports something like this, so they really need the all-in-one thing or they're just going to say "we don't really need TLS". They get by because nothing in their infrastructure ever changes; certs are new because they require operator intervention every 90 days and that is a new experience. It is unfortunate, but that's where most of the world is at right now.

Agreed.

However if an author of the solution does insist on implementing it in one binary with orchestration step being in the code rather than pulled out into the infrastructure management, the author should not be surprised that no one outside tiny single server operations would consider it to be a contender.

Your logic assumes that Caddy's auto-HTTPS cannot be configured and must be self-contained, but that's not accurate. Caddy can be configured with traditional, manually-managed (or externally-managed) TLS certificates just like other web servers too, if you need that.

All we're saying with this project is, "There is a better way going forward, and we're solving that problem."

You are presuming that as a designer of the Caddy you understand how the non-single-VM customers are running their infrastructure.

Here's a typical problem of a distributed entry point system - network partitioning.

* If your key/cert store relies on key servers being always accessible from everywhere, it simply won't work.

* If your solution is to bake a key into a file system, then it is no better than existing nginx implementation.

Basically, if you want your solution to be adopted, it needs to be significantly better than the entrenched ones.

Yeah, I think that's a fine analysis. I don't think Caddy cares that it's not the "right way" if it solves problems for some people. It is certainly easier than certbot and nginx for cases where you have one VM that you play with as though it's a pet. And a lot of people treat their production environments that way, so it does solve a problem.

The next problem that people will run into is that because they can't reliably run multiple "things", they won't have a monitoring stack to tell them that certs aren't renewing. But, maybe nothing bad will happen and it won't be a problem. "Hope is not a strategy" if you're a Google SRE, but people do pretty OK with hope in the real world ;)

> The next problem that people will run into is that because they can't reliably run multiple "things", they won't have a monitoring stack to tell them that certs aren't renewing.

Can you elaborate on this? I'm not sure I follow; there's nothing preventing people from using something like https://ohdear.app/ (or any other monitoring tool) with a self-contained solution like Caddy. In fact, I recommend that people do.

No, something like that is fine. I am just trying to convey that there is a huge industry around fixing these problems very incrementally. ohdear.app today, then DataDog... all because they can't just run Prometheus and Alertmanager.

This isn't how I like it, but I totally see the use case. Caddy is a part of that use case and is probably the reason it exists. I think that's fine!

(Edit: ohdear is really neat though, I think people might want that no matter what. You have to serve your status page from somewhere, might as well get some cert checks thrown in for free!)

Just to clarify, Caddy v1 exports metrics to Prometheus. V2 could easily do the same, it's just as extensible as v1.

https://caddyserver.com/v1/docs/http.prometheus

I am not talking about Caddy, I'm talking about the thought process that leads to needing everything integrated in one process.

> The next problem that people will run into is that because they can't reliably run multiple "things", they won't have a monitoring stack to tell them that certs aren't renewing. But, maybe nothing bad will happen and it won't be a problem. "Hope is not a strategy" if you're a Google SRE, but people do pretty OK with hope in the real world ;)

Sort of agreed with this.

Based on my experience, the reality is a bit more complicated. As soon as one is past the "this VM is a special pet and it is the only one I have" stage in my experience even smaller companies get dozens and soon hundreds of distinct HTTPs entry points:

* Main site/API ( code )

* Ops support

* bizops ( invoicing/quoting )

Which is where the issues of monitoring and observability come up.

I bet techcrunch has close to a hundred HTTP/HTTPS entry points and every time they evaluate a solution to replace their current entry point structure they ask if the new method would at least solve all of their existing problems. Ops people aren't gong to add another partial solution so the mix.

> I bet techcrunch has close to a hundred HTTP/HTTPS entry points

I wouldn't be so sure about that. It's certainly possible that's the case but equally I've personally seen new sites on Techcrunch's scale literally served from a couple of a couple of boxes running Wordpress (I kid you not!) which sits behind a few more boxes running pretty aggressive Varnish profiles and then that is sat behind Akamai's CDN. Most of the content on news sites is static and highly cache-able so there isn't always the need to go down the micro-services route.

I think you are thinking strictly about an origin for techcrunch.com and excluding all the other services that spring up internally to run any reasonably sized property.

For example, does it run RabbitMQ anywhere? Rabbitmq admin front-end is HTTP. Same goes for Kafka and other applications and we aren't even touching bizops.

I'm not going to argue that you're wrong nor that I have much experience of Techcrunch aside the odd link posted on here but it's worth noting that you don't actually need all of that stuff and thus many online new sites don't. Plus sometimes those other resources might be managed by other teams where they follow different deployment models / best practices / etc than those who manage the public news site (team responsibilities are often messy and illogical)

My point isn't to say you're wrong, just that there will be as many exceptions to your point as there are occasions it's accurate.

Reply on news.ycombinator.com