Caddyhttp: Enable HTTP/3 by Default | Hacker News Reader

Caddyhttp: Enable HTTP/3 by Default

github.com

284 pointsfariszr115 comments

Oh hey HN. This is in the Caddy 2.6 beta already: https://github.com/caddyserver/caddy/releases/tag/v2.6.0-bet... - please try it out!

Thanks to Marten Seemann for maintaining the quic-go library we use. (I still haven't heard whether Go will add HTTP/3 to the standard library.)

Caddy 2.6 should be the first stable release of a general-purpose server to support and enable standardized HTTP/3 by default. HTTP versions can be toggled on or off. (Meaning you can serve only HTTP/3 exclusively if you're hard-core.)

PS. Caddy 2.6 will be our biggest release since 2.0. My draft release notes are about 23 KB. We're looking at huge performance improvements and powerful new features like events, virtual file systems, HTTP 103 Early Hints, and a lot of other enhancements I'm excited to show off on behalf of our collaborators!

Absolutely awesome!

I have recently gave Caddy a shot, and immediately loved it!.

I am a bit worried about the performance deficit compared to apache or Nginx, but none of my servers really have reached the scale where it actually matters.

Its unfortunate that there is no dynamic brotli support, but oh well, a trade off I'm willing to make.

One of things that aren't clear to me are Caddy plugins. How big of an affect do they have on peformance? Are plugins a core part of Caddy's usage? Or are they just exist to cover some edge cases?

I'm thinking of using the caddy-filter and caddy-crowdsec-bouncer plugins.

Glad you liked it!

Caddy's performance is competitive with nginx, but there are too many dimensions to discuss here. I guarantee that for most people, your web server will not be your bottleneck.

Dynamic brotli is expensive, as brotli compression is inefficient. There are no optimized Go implementations of this yet. But Caddy can serve precompressed brotli files without any separate plugins.

What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin. (All plugins are compiled in natively.)

> What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin.

Oh I didn't know that, thanks for the awesome project!

Having plugins installed has no performance incursion at all (because they don't run unless configured), but using them will, of course affect performance. But it depends on the plugin and what it does.

You have to do your own benchmarking to see what effect they have to performance. Everyone has different needs and configs, so it's not possible to give authoritative benchmarks. Just try it out and see.

Regarding brotli, there is a plugin you can try if you'd like https://github.com/ueffel/caddy-brotli, but brotli's on-the-fly performance is not that good. It works well if you pre-compress you assets though, and you can serve those with the file_server directive's precompressed option.

I've just moved over to Caddy and had a really good experience in doing so (thank you to you and all the other Caddy contributors!). One thing that stuck out to me during setup is that it seemed surprising to see gzip encoding not enabled by default. Wouldn't it make sense for a server that enables https by out of the box with zero configuration and soon supports http3 to also gzip by default?

Good question!

When I first announced Caddy in 2015 and the server got busy, it started downloading everything as .gz files. (facepalm) (good ol' days)

But that's just a fun story, not actually the reason we don't enable it by default...

The tricky thing about enabling features by default is that turning them off is awkward in config. And while we like to have "magic" in Caddy, we don't like too much magic. Plus, gzip performance in Go is less good (but more memory-safe) than it is in C and assembly. Klauspost's flate implementation is very fast and that's the one we use. But even with a super-fast implementation, gzip requires memory and CPU that busy servers may be in short supply of.

So it's opt-in for now. We can always change it later; going the other way is harder.

(Caddy also supports serving pre-compressed files, if enabled! And starting with Caddy 2.6, those can be teleported at the speed of electricity to HTTP clients using sendfile. HTTPS also sees faster file serving in 2.6 due to optimized copying.)

Thanks for the response! I'd imagined reasoning along those lines. I'd perhaps expect that people anticipating such high load that gzipping output is problematic would be investing enough time in server setup to review and tweak these options. And for everyone else, compression by default is a good starting point. Moving from Nginx (HTTP2 off by default) to Caddy (HTTP2 on by default) you'd already need to review your setup if running a highly trafficked site to ensure any reverse proxied servers can handle the additional request load due to multiplexing.

But it's obviously a trade-off and there's lots to weigh up...

It is an interesting decision point, and I get your reasoning. But, I have noticed there are a good number of websites that could be better for their users (especially mobile users), because someone just took the defaults.

Not high priority, but if Caddy would offer a report of things that aren't configured but perhaps should be...that could help. Things like Cache-control/Expires/Etag headers, Gzip/Deflate, Caching, and so on.

If gzip is enabled, is the gzipped version of a static file cached in memory, or is it gzipped anew for each request?

Caddy doesn't cache responses by default, but you can enable caching with the cache plugin: https://github.com/caddyserver/cache-handler/

Does this apply to static file serving? Although I suppose relying on the OS cache might be enough.

Why was gzipping everything a facepalm? Performance?

It was making the browser download files with the .gz extension, instead of serving the content in a way that the browser would display. TL;DR it was broken at the time. But it was a very long time ago.

In addition to the reasons that mholt mentioned in response to your question, enabling HTTP/GZIP compression could possibly be less secure for some web server configurations due to things like the BREACH attack. See https://en.wikipedia.org/wiki/HTTP_compression#Security_impl... and https://en.wikipedia.org/wiki/BREACH for more info. I might be wrong but I don't think that current web serving protocols mitigate an attack like this. It might be better to default to safe settings that don't use HTTP/GZIP compression even if it might slow things down for the time being.

Eh, it's not just current web serving protocols. Any protocol where:

- An application uses compression

- An attacker is able to supply chosen data to it

- The application compresses the attacker's data and static secret data together

- The attacker is able to monitor the size of the compressed data

- This can be repeated by the attacker a number of times

will be vulnerable to having its secret data stolen by techniques like BREACH. If you want your secret data to stay secret, don't compress it with attacker chosen plaintext where the resulting size could be monitored.

Does Caddy have the same issue with configuration spaghetti problem with Traefik? How does it differ from Traefik's model?

I'm not super familiar with Traefik, but if you're talking about using container labels for config, no, Caddy's config is given as a JSON document through a REST API. The CLI can let you use config files if you prefer. Or the Caddyfile is a more human-friendly format that Caddy supports by default. You can actually use any config format you want if there's a config adapter for it: https://caddyserver.com/docs/config-adapters

Caddy is vastly vastly better in this regard. Traefik config files drive me insane when rewriting anything.

Planning on trying Caddy 2.6 one of these day to see how much improvement it can make over other web servers, maybe CentmindMod would be interested in it. We all are aware local vs real world benchmark are differ.

Caddy v2.6 is making huge improvements here. See https://github.com/caddyserver/caddy/pull/5022

But also, please don't take benchmarks like this as gospel. Run your own benchmarks, on your own hardware, with your own config. Someone else's config may not be ideal for your usecase.

So you want to benchmark the network?

On a VPS, but no doubt that running Go app directly is more performant, I could use Go for serving static contents as well.

I have done that test. Go is not significantly faster than Caddy for serving static content.

I've been using Caddy in a 3 node HA cluster sharing an anycast bgp address for about 18 months now and it's been fantastic. Certs "just work" across the cluster once consul is wired up. I recently added IPv6 which also "just works."

greenpau/caddy-security is fantastic and "just works" for OIDC sso.

mholt, thanks for recently adding the ability to bind to multiple specific IP addresses by default, this help me conserve precious public IPv4 addresses.

This is a thing? Any docs on this as the top search is this comment.

For which part? The comment you replied to mentions a bunch of different features, and a plugin.

The HA Cluster with an Anycast IP.

They're essentially just saying they run multiple instances of Caddy, and those instances are reached by clients via Anycast. This isn't anything specific to Caddy, that's networking-level stuff, in front of Caddy.

TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance. Most anycast setups I have seen have been geographic which means unless you move you won't get routed to the wrong endpoint.

*http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.

You are right, but it's easier than you think. At my workplace we have anycast without a ton of "tie breakers" so each client may have several backend servers that it could reach. But the router itself does do the IP based hashing when there are two valid routes to the same prefix. So yes, there is something doing hashing, but it's not anything special, it's just the usual "equal-cost multipath" behavior on the router.

> TCP needs a connection to be tracked or it counts as invalid session since the tuple wouldn't existing in the connection table. So there has to be something going on or something in front of Caddy doing ip based hashing to a backend caddy instance.

It's called BGP protocol, as OP mentioned. This is basically how cloudflare works, you have the same IP for dozens of datacenters around the world. If routing changes and suddenly you are routed to different POP then your TCP session is broken but in most cases this is rare. You could "fix" this by routing TCP packets to POP B when detecting on POP A that session origin was POP B but this would incur additional latency for the rest of the session etc. But this has nothing to do with Caddy, you could use any other software same way using anycast IP.

BGP just makes sure your packets get to the right location. If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.

> BGP just makes sure your packets get to the right location.

I think I wasn't clear enough. Yes, BGP will route the packet to the nearest POP for that IP if it was announced in different locations, that's how "anycast IP" works, thanks to how BGP works. OP asked:

> or something in front of Caddy

So I've described what is it and how it works in simple terms.

> If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.

If you are talking only about BGP then of course you are right but I was not talking about solving this using only BGP. You can tunnel the packet to your other POP which was origin, as I mentioned in my answer.

So if you have a session established after 3 way handshake in POP B

client <-> POP B

and suddenly you are routed via POP A then just tunel all packets for that session:

client <-> POP A <-> POP B

or you could use MPTCP to solve this if supported etc.

... and now we're talking about contraptions you'd have to put in front of Caddy to make this work, which was what the commenter above was interested in hearing about. I don't care either way, except that the implication that BGP4 solves this problem set me off.

Obviously, in reality, people mostly rely on geography to deal with this: you're just unlikely to get randomly rerouted from a Singapore rack to a Sydney rack, and so the issue doesn't come up. But if you care about it --- as the previous commenter does --- you want a real answer.

> But if you care about it --- as the previous commenter does --- you want a real answer.

Yeah, I wasn't clear enough in my first comment, my last comment was more detailed so I guess it should match "a real answer" now.

> Obviously, in reality, people mostly rely on geography to deal with this: you're just unlikely to get randomly rerouted from a Singapore rack to a Sydney rack, and so the issue doesn't come up.

Yeah, most of the time it doesn't come up, anecdotally it's also more specific to some regions, in Europe for example I've seen this to occur more often (between POPs in EU) than in other regions.

This is all in my basement but I tried to build it like I would a rack in a datacenter the stack is:

1: EdgeRouter4 connected to ISP. This operates as a typical router, nothing special other than some static routes to item 2.

2: Pair of VyOS routers running in VMs. I think of them as top of rack L3 switches in a datacenter. Their primary purpose is for ECMP [1]. They also use VRRP so the can be upgraded without downtime.

3: Three node caddy cluster I mentioned. Running plain old Debian. anycast-healthchecker configures the anycast IP on each VM's loopback interface. Bird advertises the address to the two VyOS routers via BGP.

4: Caddy binds to each address.

All of the hosts in this network are configured with a default route to the VRRP address on the VyOS VM's. That's pretty much it.

I've recently started using Calico in k8s on some separate VM's, and they work the exact same way, advertising Services of type LoadBalancer to the VyOS routers which does ECMP across the nodes. I learned quite a bit configuring anycast-healthchecker with Caddy, then comparing it to how Calico works.

[1]: https://codecave.cc/multipath-routing-in-linux-part-2.html

Got it, ECMP was the missing link to that and makes sense. Thanks!

Thanks for a great piece of software that I use every day in production and just works.

At first I was scared of how stupid simple it is. It feels like web servers are supposed to have giant config files with a hundred mysterious knobs to twiddle. Now I always default to Caddy, and have yet to find an instance where it didn't fit my needs. Congrats.

This is my experience as well. I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt. Figured it out with Caddy within minutes. Really happy I found this software.

> I've spent weekends trying to figure things out with Apache + Nginx + Let's Encrypt.

Curiously, new versions of Apache include mod_md, which lets you use Let's Encrypt for SSL, without the need for something like certbot: https://httpd.apache.org/docs/2.4/mod/mod_md.html

I actually wrote about it in a blog post of mine, "How and why to use Apache httpd in 2022": https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-...

That said, while the capability is there, things like the DNS-01 validation need additional work and Caddy comes with more sane defaults out of the box and arguably a way easier config format.

It would also be really cool if Nginx came with its own implementation of ACME certificate provisioning out of the box, following the trend of other servers, but certbot is also perfectly passable.

Either way, it seems like most of the "popular" web servers are viable for most use cases nowadays and I can say that Caddy definitely belongs in that list!

I very recently tried Caddy for the first time and it is one of the best software experiences I had especially because of it's ease of use.

My only complaint with it is this half-way json vs standard config file situation in which it finds itself. The documents push you to the json system, but most of the available documents are in the config system.

There's also some handy features that are only available in the JSON format. e.g. `strip_path_suffix` and `http_redirect` for the `rewrite` handler (https://github.com/caddyserver/caddy/issues/2011#issuecommen...), which would be a more ergonomic way to set a rule to strip trailing slashes than handling via regex.

While it is true that some features are only possible via JSON config, the ones you mentioned are available in the Caddyfile. See the uri directive which lets you strip a suffix, and redir which performs Location redirects. Also, the issue you linked to is from an early beta of Caddy v2, and the discussion and config there no longer applies.

The uri directive does expose stripping a suffix, but that's an internal rewrite, correct?

For any /foo/ /bar/ /baz/ I want to redirect to /foo, /bar, or /baz. Unless I'm missing something, this needs to be done like so right now:

  @trailing_slash path_regexp trailing_slash ^/(.*)/$
  redir @trailing_slash /{re.trailing_slash.1} 308

The uri directive can strip_prefix, strip_suffix, or do substring replacements: https://caddyserver.com/docs/caddyfile/directives/uri

It can also do regex replacements on the path portion of the URI. And yes, these are internal rewrites since uri is a directive that wires up the rewrite handler.

We actually have a special section in the docs all about enforcing trailing slashes, including external redirects: https://caddyserver.com/docs/caddyfile/patterns#trailing-sla...

Note that the file_server will automatically enforce canonical URIs, including redirecting to add or remove the trailing slash according to whether the resource is a file or a directory.

To redirect many unspecified paths, your regex is probably the best way to do it for now. Feel free to open an issue to propose an alternative!

Thanks! I'd read all that documentation and came to the same conclusion. I'm using disable_canonical_uris as I prefer avoiding trailing slashes even for directories (as I see it, page_name/index.html layout is an implementation detail, and I'd like Caddy to expose it as /page_name).

I'll follow up with issues tomorrow (I think an issue for updating the common caddyfile patterns bit on the website also makes sense - I suspect a blanket rule for removing trailing slashes is more likely to be a common pattern people seek a solution for than removing a trailing slash on explicitly enumerated paths).

The problem is, in the browser, relative URLs (like on CSS/JS assets and such) won't be relative to your index.html if you remove the trailing slash, and they'll instead be relative to the parent directory. That can mess things up.

Of course if you're using absolute paths in your HTML then this isn't a problem, but it does become a problem if you want to move to serving this same site from a subpath, because then your absolute paths won't be constrained to this subpath.

JSON is the "machine readable" config language for Caddy, but a great majority of users are using the Caddyfile. What are you confused about specifically?

Weirdly enough, among the reasons to switch from nginx to Caddy I never find the lack of observability nginx and its almost useless Prometheus exporter have.

I see several posts discussing under which circumstances or use cases one might outperform the other but they never seem to care about having decent metrics.

I might overrate the importance of this, who knows...

What would decent metrics be to compare the performance of Nginx and Caddy?

It's not really possible to compare the performance of webservers apples to apples. Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time.

What is this I'm reading here, the voice of reason?!

The last frontier for these types of comparisons is unusual circumstances like embedded platforms with limited resources.

Indeed, this is why those hello world benchmarks are just noise.

> It's not really possible to compare the performance of webservers apples to apples.

I'd say that it's possible, just not always easy, due to how many different configurations there can be for any given workload - the same servers might be used to achieve the approximately same configuration in different ways.

However, if you took a common enough use case or workload, such as reverse proxy with SSL and gzip compression for $FOO resource types, serving files for $BAR application and proxying $BAZ API, then it should definitely be possible. You would just need some meaningful real-world test instead of a purely synthetic benchmark, otherwise you'll probably test something slightly different than what your server will be doing in practice.

The other option is just to settle on some bit of common enough functionality and try to increase your sample size as much as possible.

For example, some attempts have been made by OpenBenchmarking and at least give a vague idea of the performance of some servers:

Nginx: https://openbenchmarking.org/test/pts/nginx

Apache:https://openbenchmarking.org/test/pts/apache

I'm mentioning this because while not everybody has the time to reproduce a given setup in any number of technologies, even similar enough tests in a controlled environment can produce meaningful information, at least to let you infer what orders of magnitude you're working with. For something concerning programming languages themselves and their frameworks, one just needs to look at what TechEmpower is trying to do: https://www.techempower.com/benchmarks/#section=data-r21

> Really, you need to do your own benchmarking to determine which solution is best for you. But keep in mind that the web server is rarely the bottleneck, usually your app and database IO are where it takes the most time.

This is well said, though.

In general, I'm inclined to agree: most of the time, the performance of most web servers can be described as "good enough".

The exception to this might be using your application servers (e.g. Tomcat) as a web server and running into situations where serving static assets (that might be baked into your application) would slow down because of API calls being slower and processing them digging into comparatively more conservative HTTP thread limits for the whole thing. Then again, personally I'd argue that you should have one of the popular web servers in front of your applications (and typically serving static assets) to act as an ingress in most cases, but I've seen some interesting things over the years.

Regarding that last point, that's often done with the X-Accel-Redirect header; backend can decide which file to serve, responds with the path to that file as that header, then the webserver can serve that file if that response header is there. Caddy can do this btw, with response handlers in the reverse_proxy directive. This pattern can be used to serve uploaded files stored in a non-public location, after the backend authenticated the request etc.

> This pattern can be used to serve uploaded files stored in a non-public location, after the backend authenticated the request etc.

That's a perfectly valid use case!

Packaging a front end application (e.g. React/Angular/Vue) inside of a .jar file and making Tomcat or something else serve it, as a part of a larger Java application, though? Perhaps less of an optimal solution than just using Apache/Nginx/Caddy for it, outside of really wanting to be able to deploy everything as a single package.

lucaslorentz/caddy-docker-proxy works like Traefik, in that Container metadata labels are added to the reverse proxy configuration which is reloaded upon container events, which you can listen to when you subscribe to a Docker/Podman_v3 socket (which is unfortunately not read only)

So, with Caddy or Traefik, a container label can enable HTTP/3 (QUIC (UDP port 1704)) for just that container.

"Labels to Caddyfile conversion" https://github.com/lucaslorentz/caddy-docker-proxy#labels-to...

From https://news.ycombinator.com/item?id=26127879 re: containersec :

> > - [docker-socket-proxy] Creates a HAproxy container that proxies limited access to the [docker] socket

There is also nginx_proxy, based on nginx.

https://github.com/nginx-proxy/nginx-proxy

Note on both projects, if you care about security, you should split the generator/controller container out of the main webserver container, so the docker socket is not used in container that is directly exposed.

The point of the link in OP is that now in v2.6, Caddy enables HTTP/3 by default, and doesn't need to be explicitly enabled by the user.

So I'm not exactly sure the point you're trying to make. But yes, CDP is an awesome project!

That is a good point. Is there any way to disable HTTP/3 support with just config?

The (unversioned?) docs have: https://caddyserver.com/docs/modules/http#servers/experiment... :

> servers/experimental_http3: Enable experimental HTTP/3 support. Note that HTTP/3 is not a finished standard and has extremely limited client support. This field is not subject to compatibility promises

TIL caddy has Prometheus metrics support (in addition to automatic LetsEncrypt X.509 Cert renewals)

Yeah those docs are for v2.5. The experimental_http3 option is removed in v2.6 (which is currently only in beta, stable release coming soon). To configure protocols and disable HTTP/3, you'll now use the "protocols" option (inside of the "servers" global option), and it would look like "protocols h1 h2" (the default is "h1 h2 h3").

Yes, the docs have been updated at https://github.com/caddyserver/website but haven't been deployed yet. There is a new protocols option:

    protocols h1 h2

will disable Http/3 but leave 1.1 and 2 on.

This is a little-known awesome project! If you like how Traefik is configured with Docker labels but want to use Caddy, well now you can have your cake and eat it, too.

Will have to give Caddy a try. Been a long time nginx user, but they have been very slow to implement new features (including HTTP/3), unfortunately.

It should come in the next release

> Two critical capabilities for security and scalability of web applications and traffic, HTTP3 and QUIC, are coming in the next version we ship

https://www.nginx.com/blog/future-of-nginx-getting-back-to-o...

I tried caddy for my home setup. Replacing nginx and traefik. I am wondering if I should ditch them in production too. I am hesitant, cause I don't see caddy as "business" software: nginx has a learning curve and configuration to master, where caddy just does the job, sane defaults and not much to "work". It just doesn't feel like work using caddy...

Large companies like Stripe use Caddy. It's definitely not a toy. It is, and has been production ready for many years.

I guess it's just me than; all the stuff I have learned: setting the right labels for traefik, debugging it for hours. Now it's just:

```

caddy: example.com

caddy.reverse_proxy: {{ upstream 8080 }}

```

Yep! The CDP plugin https://github.com/lucaslorentz/caddy-docker-proxy makes configuration per docker service pretty easy!

Tangential, but Google is serving HTTP/3 by default out of Google Cloud now: https://cloud.google.com/blog/products/networking/cloud-cdn-...

Pretty cool stuff.

I have been using HAproxy and it has been very performant but the lack of documentation for its APIs may be the reason I will start playing with Caddy.

I'm surprised you find the documentation lacking. What type of API are you talking about? The internal C API? Lua? Configuration? Something else?

Disclosure: I'm a community contributor to HAProxy and help maintain the issue tracker.

The documentation is extensive but I was unable to find tutorial that could help me dynamically add/remove backend servers using the dataplane API. Also being able to dynamically refresh certificate. These are pretty much the only reasons for why I am considering Caddy.

The dataplane API is an external product that itself uses the "stats socket". I personally don't have any experience with the dataplane API.

Changing certificates (https://docs.haproxy.org/2.6/management.html#9.3-add%20ssl%2...) and adding servers (https://docs.haproxy.org/2.6/management.html#9.3-add%20serve...) using the stats socket is both officially documented.

If you find the documentation insufficient or unclear, then this can be considered a documentation bug that I recommend filing here: https://github.com/haproxy/haproxy/issues

I'm curious about the implementation, and haven't looked at the source of quic-go yet: Does it use a single UDP socket to handle datagrams for all QUIC connections, does it use connected UDP sockets per connection, or does it use multiple UDP sockets, where each handle a certain set of connections - and where an external load balancer is required to redirect to the sockets?

Unfortunately there's no best answer for this: Using a single socket will allow for connection migration, but it will end up being a bottleneck in terms of scalability since it will serialize access on a lot of kernel and driver datastructures (just a single transmit/receive queue). Connected sockets avoid that, but don't allow for address migration. And doing external load balancing gets far more complex than just starting a binary - even the most simple solution requires running XDP code.

> connected UDP sockets per connection

I think I'm stumbling over terminology here, but UDP is connectionless, right? I think the QUIC protocol re-implements connections on top of it, but I've never heard of "connected UDP".

A socket pool does seem to make sense from a performance standpoint!

UDP is connectionless. Connected UDP sockets are a OS construct, and don't influence anything that happens on the wire. If you "connect" a UDP socket, you don't have to use sendto and recvfrom anymore, and all packets received and sent (via send/recv APIs) are targetting the same peer. It's kind of just sets up some routes in the kernel, but doesn't make anything stateful or reliable at the networking layer.

Beware of a performance hit (in term of bps not req/s) if you push big data with Caddy.

Go implementation of HTTP/2 already took a /5 hit over http/1.1 (Go http/2 implementation is 5x slower than Go http/1.1)

With HTTP/3 our early benchs indicate /2 ot /3 from HTTP/2 (so /10 from http/1.1)

Marten is in the process of making a bunch of optimizations [0], as are we within Caddy, and 2.6 includes some of those. It will only get better with time.

Most users do not operate at nearly the volume required to feel the impact, including enterprises.

Would be interested in repeating your experiments and knowing your real world use case and seeing how similar they are.

[0]: https://github.com/lucas-clemente/quic-go/issues/3526

We do speed tests hence we saw the issue. See https://github.com/golang/go/issues/47840

Yeah I agree this is a edge case, 99,99% of Caddy users don't push so much data ;)

But Caddy can't be used as a big files download server for instance, unless sticking with HTTP/1.1 (that and sendfile + kTLS not been supported last time I checked?) (at least with a single http instance).

Another issue for HTTP/3 is https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-B... Current Linux in WSL for instance has this issue. This will probably resolves over time as more and more Linux distros will take HTTP/3 requirements into account.

Caddy 2.6 has some optimizations for serving large files, including sendfile, but other optimizations too. No ktls yet, but maybe someday. It's just... kinda gnarly.

thx i'll look at v2.6.

with BPS you mean throughput on a single request? On an unconstrained connection (like loopback)? In this case you are mostly measuring CPU efficiency, since for a real world deployment the network connection would likely be the bottleneck. In that real world setup then loss of CPU efficiency becomes a scaling challenge, which actually makes it less RPS.

Btw: A good recommendation for anyone doing benchmarking is to both measure throughput for a single connection, but also throughput for the whole server when using lots of connections (e.g. 100 connections per core, and each connection doing multiple concurrent requests). The latter will point out further scaling issues.

In terms of full system efficiency HTTP/2 vs HTTP/1 (with TLS) is actually not that bad for good implementations - might have 10-50% more cost but in the end the performance critical parts (doing syscalls for TCP sockets) are the same for both. QUIC can be much worse. Well optimized setups (which e.g. use optimized UDP APIs with GSO or even XDP) somewhere between 50-100% more expensive than TCP+TLS setups. And simpler implementations can be 5x as expensive, and might not even scale beyond a single CPU core.

For HTTP/3 support with python clients:

- aioquic supports HTTP/3 only now https://github.com/aiortc/aioquic

- httpx is mostly requests-compatible, supports client-side caching, and HTTP/1.1 & HTTP/2, and here's the issue for HTTP/3 support: https://github.com/encode/httpx/issues/275

Since address validation is about blocking senders which forge their IP address, I think the number of connection attempts which where the client doesn't eventually validate its address should factor into the decision to enable this feature. This should rarely happen for legitimate clients (e.g. connection loss/cancellation during the first roundtrip) but always happen for IP address forgers.

Or perhaps simply use the number of half-open/embryonic connections as the metric.

Good point. Our current metric (number of active requests) is a bit naive, but mostly does the job essentially for free. If your proposed metric doesn't cost too much in memory, it might be a good fit.

How does such an attacker influence your active requests metric? Is it incremented when the initial UDP packet arrives (even though the http request itself hasn't arrived yet) and decremented when it times out?

No, currently we increment it when the HTTP request is parsed. (It counts HTTP requests, not packets/connections.) This was a quick and easy way to count; but I am very open to improving this! Feel free to submit a proposal or a PR.

Ref: https://github.com/caddyserver/caddy/blob/50748e19c34fc90882...

In that case, I don't think your metric works at all, since legitimate users will increase it, but attackers won't. During an attack it will either remain the same, or even go down as fewer legitimate users manage to connect.

This attack is the QUIC equivalent of a SYN flood, which results in half-open connections, because the attacker is unable to complete the connection by responding to message the server sends. RequireAddressValidation corresponds enabling syn cookies.

Unfortunately I don't use caddy myself, and don't have time to get involved deeper in improving this.

I opened an issue with quic-go to get ideas for a better metric: https://github.com/lucas-clemente/quic-go/issues/3549

does it work with "tls internal" or do the certs need to be signed by an outside CA for HTTP/3 to be enabled.

Yes it works with self-signed certificates!

I'm really glad to hear this is an option. But will browsers accessing the website accept the self-signed certs when setting up the TLS connection required by QUIC? I don't know how they behave. I'm genuinely asking.

Yes they do, as long as you installed Caddy's root CA cert in the browser's trust store. Caddy will attempt to install it automatically with https://github.com/smallstep/truststore if possible (usually requires root) but if it fails you can try again with "sudo caddy trust". You might be using a client or trust store that isn't supported though, in which case you'll need to install the root cert manually.

What if I just want to generate and use a self signed cert without any CA involvement? I do that with HTTP all the time and it works for the public web even if the browsers like to scaremonger about it.

Does a QUIC connection work without a CA (corporate) root cert being referenced?

To be clear, Caddy itself manages its own internal CA. It's the same thing as self signing except you have a root cert you can trust, to trust every cert Caddy generates; that's instead of having individual self signed certificates. This makes it significantly easier if you have many subdomains for each of your services, etc. It has nothing to do with corporate CAs, no external entities.

QUIC/HTTP3 use the same TLS stack as H1/H2, same things are trusted. So there's no difference there.

(Fun-fact, "self-signed" certs also have "CA involvement" -- it's just that the leaf is also the signing authority, which is generally bad practice. Caddy does this properly with a separate signing root -- that stays offline and has long validity -- as Francis explained.)

Bad idea to use HTTP/3, HTTP/2 should be good enough

This is actually true some of the time.

Reply on news.ycombinator.com