There isn't much point to HTTP/2 past the load balancer
byroot.github.io
byroot.github.io
However, I think the problem is the Poisson process isn't really the right process to assume. Most websites which would run afoul of the 2/6/8/etc connections being opened are probably trying to open up a lot of connections at the same time. That's very different from situations where only 1 new person arrives every 6 minutes on average, and 2 new people arriving within 1 second of each other is a considerably rarer event.
[1]: https://www.johndcook.com/blog/2008/10/21/what-happens-when-...
Because a lot of their customers are too stupid to understand timezones.
Gary Indiana is a population center on Lake Michigan, across the Illinois/Indiana border from the Chicago exurbs. They’ve elected to be in the same time zone as Illinois, while the rest of the state is in the same time zone as Ohio. And 1997-2006 was just a clusterfuck where another county joined with Gary, and a bunch of counties decided not to follow DST.
I am not smart enough to know what time it is without asking a computer.
Because everyone is too stupid to understand them at 2 am when alarms are going off. All backends should always be in UTC, even if your boss tells you we will “never” move our servers to another time zone. Because they’re going to buy or be bought by a company based in Chicago or New York or find a vertical so fat they want a second office there, or both. And then they will not remember promising you that you would never have to do a time zone migration. Why are you being so grumpy, just fix it?
These cutoff means banks have certainty about the transaction, as these reconciliation is batched, rather than real time.
Which is why it is highly annoying when there's only one worker at the coffee stand, and there's always this one jerk at the front of the queue who orders a latte when you just want a coffee. With two workers, the people who just want coffee won't have to wait 15 minutes for the lattee people
And I've also noticed a social effect, when people wait a long time it seems to reinforce how they perceive the eventual serviced, that is, they want more out of the interaction, so take longer. Which makes it the situation even worse
Here in the espresso world, that’s not so bad. But the ‘vanilla oat milk decaf, and also a hot muffin with butter’ is tedious.
There is a roaster in Auckland that’s been there since the ‘80s. On the counter it says ‘espresso, flat white or fuck off’. Clear and concise. I like it. https://millerscoffee.co.nz/
Sounds a bit pretentious to me. I generally order a coffee, no milk ... ta.
Queuing theory is really weird.
Little's Law is fundamental to queueing theory, but there's also the less well-known Kingman's formula, which incorporates variability of arrival rate and task size [2].
[1] https://www.johndcook.com/blog/2009/01/30/server-utilization...
The first one is even worse; by 80% you're already seeing twice the delay of 70%.
If I were to describe the second chart I'd say 80% is when you start to get into trouble, not just noticing a slowdown.
I said minimize latency, not optimize latency.
0-60 and you're still at minimum latency. 60-80 you're at twice the latency but it's probably worth the cost savings of the extra compute density since it's still pretty low. Higher than 80 and things are already slowing down and getting exponentially worse by the request
--
Nowadays with DB stuff I tend to get assigned new infra leads who see a DB cluster at 50% CPU utilization and think they can go down two instances sizes without severely impacting latency.
And the thing is, even if you stuff a low priority service onto a bunch of boxes, and convince the OS to honor that priority fairly, the fact that the service runs reasonably at all gets baked in as an expectation. Maybe it’s the hedonic treadmill, one kid expects dessert with every meal because they’ve always gotten it, and another knows it’s a special occasion. But anything given is jealously guarded when you have to take it away. Even a “best effort” batch process that is supposed to finish on some interval, is missed when it no longer does. And somehow it’s always your fault.
I’m sure the grocery store employees who are assigned as backup tellers constantly get grief for not getting their other tasks done “on time”.
for context: 10 yrs experience as a software engineer and only a couple on high traffic products. Some things (like this) continue to surprise me.
> In addition to the low roundtrip time, the connections between your load balancer and application server likely have a very long lifetime, hence don’t suffer from TCP slow start as much, and that’s assuming your operating system hasn’t been tuned to disable slow start entirely, which is very common on servers.
A single HTTP/1.1 connection can only process one request at a time (unless you attempt HTTP pipelining), so if you have N persistent TCP connections to the backend, you can only handle N concurrent requests. Since all of those connections are long-lived and are sending at the same time, if you make N very large, you will eventually run into TCP congestion control convergence issues.
Also, I don't understand why the author believes HTTP/2 is less debuggable than HTTP/1; curl and Wireshark work equally well with both.
That said I still disagree with the article's conclusion: more connections means more memory so even within the same dc, there should be benefits of http2. And if the app server supports async processing, there's value in hitting it with concurrent requests to make the most of its hardware, and http1.1 head of line blocking really destroys a lot of possible perf gains when the response time is variable.
I suppose I haven't had a true bake off here though - so it's possible the effect of http2 in the data center is a bit more marginal than I'm imagining.
The problem? Well, browsers don't have a JS API for bidirectional HTTP/2 streaming, and many don't see the point, like this article expresses. NGINX doesn't support end-to-end HTTP/2. Feels like a bit of a shame, as the streaming aspect of HTTP/2 is a more natural evolution of the HTTP/1 request/response cycle versus things like WebSockets and WebRTC data channels. Oh well.
WebTransport is a bit different - it offers raw QUIC streams that are running concurrently with the requests/streams that carry the HTTP/3 requests on shared underlying HTTP/3 connections and it also offers a datagram API.
Which is why the Rails crowd want it.
Is that not the case?
H3 is supposed to solve that.
There is a distinct lack of elegance in the HTTP/2 protocol. It's exceptionally complex and it has plenty of holes in it. That it simply does a job does not earn it "elegant."
Versus HTTP/1.1, some details are simplified by moving the request and status line parts into headers. The same HEADERS frame type can be used for both the headers and trailers on a given stream. The framing protocol itself doesn't really have a whole lot of cruft, and versus HTTP/1 it entirely eliminates the need for the dancing around with Content-Length, chunked Transfer-Encoding, and trailers.
In practice, a lot of the issues around HTTP/2 implementations really just seem to be caused by trying to shoehorn it into existing HTTP/1.1 frameworks, where the differences just don't mesh very well (e.g. Go has some ugly problems here) or just simply a lack of battle-testing due to trouble adopting it (which I personally think is mainly caused by the difficulty of configuring it. Most systems will only use HTTP/2 by default over TLS, after all, so in many cases end-to-end HTTP/2 wasn't being tested.)
The problem that people run into load balancing grpc is that they try to use a layer 4 load balancer to balance layer 7 requests; that is, if there are 4 backends, the load balancer tells you the address of one of them, and then you wonder why the other 3 backends don't get 25% of the traffic. That's because grpc uses 1 TCP connection and it sends multiple requests over that connection ("channel"). If your load balancer tells you the addresses of all 4 servers, then you can open up 4 channels and load balance inside your application (this was always the preferred approach at google, with a control channel to gracefully drain certain backends, etc.). If your load balancer is aware of http/2 at the protocol level (layer 7), then you open up one channel to your load balancer, which already has one channel for each backend. When a request arrives, it inspects it and picks a backend and proxies the rest of the exchange.
Ordinary http/2 works like this, it's just that you can get away with a network load balancer because http clients open new connections more regularly (consider the lifetime of a browser page with the lifetime of a backend daemon). Each new connection is a load balancing opportunity for the naive layer 4 balancer. If you never make new connections, then it never has an opportunity to load balance.
grpc has plenty of complexity for "let applications do their own load balancing", including built-in load balancing algorithms and built-in service discovery and health discovery (xDS); http/2 doesn't have any of this. Whether these are actually part of grpc or just random add-ons to popular client libraries is somewhat up for debate, however.
Some applications it is important but for most websites the benefits just dont outweigh the costs.
It starts to really perform badly when you have dropped packets. So any kind of medium quality wifi or 4/5g kneecaps performance.
It was always going to do this, and as webpages get bigger, the performance degradation increases.
HTTP2 fundamentally underperforms in the real world, and noticeably so on mobile. (My company enthusiastically rolled out http2 support when akamai enabled it.)
Personally I feel that websockets are a hack, and frankly HTTP 3 should have been split into three: a file access protocol, a arbitrary TCP like pipe and a metadata channel. But web people love hammering workarounds onto workarounds. so we are left with HTTP3
However, HTTP/3 solves that problem and performs very well on both poor quality and good quality networks.
Typically, I use HTTP/2 to refer to both HTTP/2 and HTTP/3 since they are basically the same protocol with different transports. Most people don't really need to care about the distinction, although I guess since it doesn't use TCP there are cases where someone may not be able to establish an HTTP/3 connection to a server. Still, I think the forward looking way to go is to try to push towards HTTP/3, then fall back to HTTP/2, and still support HTTP/1.1 indefinitely for simple and legacy clients. Some clients may get less than ideal performance, but you get the other benefits of HTTP/2 on as many devices as possible.
HTTP3 is basically just HTTP2 on top of QUIC… so you already have the tcp-like pipe, it’s called QUIC. And there’s no reason to have a metadata channel when there are already arbitrary separate channels in QUIC itself.
There is not even a fixed order between 2) and 3). The server can start sending headers or body data before the client sent any body byte.
What is correct is that a lot of servers and clients (including javascript in browsers) don't support this and make stricter assumptions regarding how HTTP requests are used - e.g. that the request bytes are fully sent before the response happens. I think ReadableStream/WritableStream APIs on browsers were supposed to change that, but I haven't followed the progress in the last few years.
NGINX falls into the same category. It's HTTP/2 support (and gRPC support) had been built with a very limited use-case in mind. That's also why various CDNs and service meshes use different kinds of HTTP proxies - so that various streaming workloads don't break in case way the protocol is used is not strictly request->response.
Anyways, yes, this is technically true, but the streaming semantics are not really that well-defined for HTTP/1.1, probably because it was simply never envisioned. The HTTP/1.1 request and response were viewed as unary entities and the fact that their contents were streamed was mostly an implementation detail. Most HTTP/1.1 software, not just browsers, ultimately treat the requests and responses of HTTP as different and distinct phases. For most uses of HTTP, this makes sense. e.g. for a form post, the entire request entity is going to need to be read before the status can possibly be known.
Even if we do allow bidirectional full-duplex streaming over HTTP/1.1, it will block an entire TCP connection for a given hostname, since HTTP/1.1 is not multiplexed. This is true even if the connection isn't particularly busy. Obviously, this is still an issue even with long-polling, but that's all the more reason why HTTP/2 is simply nicer.
NGINX may always be stuck in an old school HTTP/1 mindset, but modern software like Envoy shows a lot of promise for how architecting around HTTP/2 can work and bring advantages while remaining fully backwards compatible with HTTP/1 software.
To say nothing about the many http proxies in between.
There has been a lot of pushback against supporting full-duplex streams[0].
One thing the article gets wrong is that non-encrypted HTTP/2 exists. Not between browsers, but great between a load balancer and your application.
Which has the benefit of encrypting everything and avoids the overhead of starting a TLS socket for every http connection.
Indeed, I misread the spec, and added a small clarification to the article.
The difference is that you have a h2 connection to the proxy, but everything past that point is up to the proxies routing. End-to-end h2 would be more like a websocket (which runs over HTTP CONNECT) where the proxy is just proxying a socket (often with TLS unwrapping).
Each connection need to keep state of all processed requests (the HPACK dynamic headers table), so all request for a given connection need to be proxied through the same connection. Not sure I got what you meant, though.
Apart from that, I think the second sentence of my comment makes clear there is no smuggling as long as the connection before/past proxy is http2, and it's not downgraded to http1. That's all that I meant.
HTTP/2 compresses headers, and that alone can make it worthwhile to use throughout a service fabric.
That seems like a massive benefit to me.
Anyone remember those sprite files?
But I'd like to introduce you/them to tight mode:
https://docs.google.com/document/d/1bCDuq9H1ih9iNjgzyAL0gpwN...
https://www.smashingmagazine.com/2025/01/tight-mode-why-brow...
HTTP/2 solves this, as you can multiplex requests to backend servers over a single TCP socket. So there is actually a point of using HTTP/2 for load_balancer <-> backend_system connections.
But I agree, it is a solved problem unless you really have a lot of incoming connections. When you use multiple outgoing ip addresses that fixes that even for very busy load balancers, and since IPv6 is common today you will likely have a /64 to draw addresses from.
I don't see how that is a fringe or rare case. With a loadbalancer (using no pipelining or multiplexing), the number of simultaenous outgoing http connections to backend systems is at least the number of simulatenous open incoming http connections. Having more than 28k simultanous incoming http requests is not a lot for a busy load balancer.
Now with pipelining (or limiting to 28k outgoing connections), the loadbalancer has to queue requests and multiplex them to the backends when connections become available. Pipelining suffers from head-of-line blocking, increasing possible latency caused by the loadbalancer further. In any case, you will increase latency to the end-user by queing. If you use HTTP/2 multiplexing, you can go past those 28k incoming connections without queing on the loadbalancer side.
"Doctor it hurts when I hit myself"
> the number of simultaenous outgoing http connections to backend systems is at least the number of simulatenous open incoming http connections
No it isn't. You establish a pool of long lived connections per backend. The load balancer should be coalescing in flight requests. At that traffic volume you should also be doing basic in-memory caches to sink things like favicon requests.
I am not going to respond further as this chain is getting quite off topic. There are plenty of good resources available from relevant Google searches, but if you really still have questions about how load balancers work my email is in my profile.
Yes, and you would do the same with HTTP/2. You haven't addressed the head-of-line blocking problem caused by HTTP/1.1 pipelining, which HTTP/2 completely solves. Head-of-line blocking becomes an increasing issue when your HTTP connections are long lived, such as when using websockets or large-media transfers or streaming.
The case where ports are quickly exhausted is with long connections, typically WebSocket. And with properly tuned servers, reaching the 64k ports limit per server comes very quickly. I've seen several times the case where admins had to add multiple IP addresses to their servers just to hack around the limit, declaring each of them in the LB as if they were distinct servers. Also, even if Linux is now smart enough to try to pick a random port that's valid for your tuple, once your ports are exhausted, the connect() system call can cost quite a lot because it performs multiple tries until finding one that works. That's precisely what IP_BIND_ADDRESS_NO_PORT improves, by letting the port being chosen at the last moment.
H2 allows to work around all this more elegantly by simply multiplexing multiple client streams into a single connection. And that's very welcome with WebSocket since usually each stream has little traffic. The network also sees much less packets since you can merge many small messages into a single packet. So there are cases where it's better.
Another often overlooked point is that cancelling a download over H1 means breaking the connection. Over H2 you keep the connection opened since you simply send an RST_STREAM frame for that stream in the connection. The difference is important on the frontend when clients abort downloads multiple times per browsing session (you save the TLS setup again), but it can also make a difference on the backend, because quite often an aborted transfer on the front will also abort an H1 connection on the back, and then that's much less fun for your backend servers.
I've built multiple systems at 1M+ r/s and Tb+ scale.
> The case where ports are quickly exhausted is with long connections, typically WebSocket
Yes, HTTP2 is great for websockets. I was never advocating against it. The comment I was replying to was under the false assumption that you needed an outbound backend connection for every incoming connection. All of his concerns are solved problems in any modern open source load balancer. See https://www.haproxy.com/blog/http-keep-alive-pipelining-mult... ;)
Also there's a cascade effect on large sites, where as long as your servers respond fast, everything's OK. Suddenly a database experiences a hiccup, everything saturates, and once you enter the situation where the LB has all of its ports in use, it can take a while to recover because of connect() getting much slower (I already observed delays up to 50ms!). At this point there's no hope to recover in a sane time, because excess connections are not even served by the servers, they're in the accept queue in the system, so they keep a port busy, slowing down connect() which means more even connections are needed for other incoming requests. If the LB is not properly sized and tuned, you'd rather just kill it to get rid of all the connections at once, wait a second or two for the RST storm to calm down and start again.
H2 can avoid that, at the expense of other issues I mentioned in another response above (i.e. don't multiplex too much to the servers, 5-10 streams max, to avoid the risk of inter-client HoL). But H2 also comes with higher xfer costs than H1 for large objects due to framing.
That’s likely not because of the wisdom in the article per se, but because of rising complexity in managing streams and connections downstream.
I think the author wrote it with encryption-is-a-must in the mind and after he corrected those parts, the article just ended up with these weird statements. What complexity is introduced apart from changing the serving library in your main file?
If you assume no multiplexing, you can write a much simpler server.
> it's not http/2 fault, but rather ruby
My post is to be read primarily in the context of Ruby, as the intro clearly explains it. I'm not the one who posted it here, it really isn't intended for the HN audience. I would never submit my posts here.
Many of my points are more general than just Ruby-centric, but yes, if your stack of choice have very good support for HTTP/2 I'm not saying not to use it in your DC.
My point is that as a Ruby user, there isn't much reason to lament over the lack of HTTP/2 support in Puma or some other servers.
So that's Python, Ruby, Node. Elixir won't care and C# and Java... well hopefully the HTTP/2 library takes care of the multiplexing of the replies, then you're good.
I didn't say forking per request, good god. I meant running a process per core, or some ratio to the cores to achieve full server utilization. Limiting all of HTTP/2 requests per user to one core is unlikely to result in good feelings for anybody. If you let nginx fan them out to a couple cores it's going to work better.
These are not problems Java and C# have.
Past the reverse proxy, is there a point to HTTP at all? We could also use SCGI or FastCGI past the reverse proxy. It does a better job of passing through information that's gathered at the first point of entry, such as the client IP address.
Lots of places (including a former employer) have done tons of work to upgrade internal infrastructure to support HTTP/2 just so they could use gRPC. The performance difference from JSON-over-HTTP APIs was meaningful for us.
I realize there are other solutions but this is a common one.
> But in the data center, not so much.
That's a very bold claim.
I'd like to see some data that shows little difference with and without HTTP/2 in the datacenter before I believe that claim.
> If you have a saturated http/1.1 network (or high CPU use) within a DC you can usually just add capacity.
Then I'd be curious how much a lack of HTTP/2 support in the application layer costs. It might be small change for an early stage startup, but when companies get large, these corner cases end up paying peoples' salaries.
The problem will be that QUIC uses more userland code and UDP is not as optimized as TCP inside kernels. So far, the extra CPU penalty has discouraged me from adopting QUIC everywhere, I've kept it mostly on the edge-out where the network is far less reliable.
I don't understand this super well, but could not get keepalives to cross the LB boundary w/ GCP
it's a shame server side hunting/push never caught on. that was always one of the more interesting features.
Essentially, the server had to do things like compute RTT and understand the status of the browser's cache to do optimal push.
although i suppose it's a solved problem these days.
Here is a silly thing that is possible with h2 over a single connection, but not with websockets:
Multiple page components (Islands) each have their own stream of events over a single h2 connection. With websockets, you will need to roll your own multiplexing[1].
[1]: I think you can multiplex multiple websockets over a single h2 connection tho, but don't quote me on this.
You will often be in the state where the client uses HTTP2, and the apps use HTTP2 to talk to the third party, but inside the data center things are HTTP1.1, fastcgi, or similar.
But I did start by saying it's not important. It's a small difference, unless you hit a connection limit.
> HTTP/2 is fully encrypted, so you need all your application servers to have a key and certificate
Nope. h2c is a thing and is official. But the article is right, the value HTTP/2 provides isn't for LAN, so HTTP 1.1 or HTTP/2 it doesn't matter much.
HTTP/3 however, is fully encrypted. h3c doesn't exists. So yeah, HTTP3 slower you connection, it isn't suited for LAN and should not be used.
BUT if you actually want to encrypt even in you LAN, use HTTP/3, not HTTP/2 encrypted. You will have a small but not negligible gain from 0-RTT.
The only field that is computed in UDP is checksum and the same exists in TCP and it must be recomputed each time someone actually re-route the packet (eg: bridge to VM) since TTL is decreased.
So I doubt your assertion.
_____
Writing my comment I understood what your are talking about. There is a bunch of encryption done at user mode in HTTP/3 that doesn't need to be done in user mode. In HTTP/2 it was sometime done in kernel mode (kTTL), so was quicker. The slowness comes for the CPU needed it to be copied out of kernel mode. I didn't follow the whole story so I trust you on this.
That is not to say that h3 does not have its place, but the networking stacks are not optimized for it yet.
The technical reason is 30+ years of history of TCP being ≥90% of Internet traffic and services. There's several orders of magnitude in resources more spent to make TCP fast starting at individual symbols on Ethernet links all the way up into applications.
(Also, what is your definition of "quick"? I have no association for that particular wording, are you referring to achievable thruput, CPU load, latency, congestion control, …?)
Here "quicker" was about CPU load and well defined.
Please re-read too.
Whomever downvoted you is probably unaware words like SHOULD have specific meaning in RFCs
(Also, why do you even want a load balancer/reverse proxy, unless your application language sucks? The article says it "will also take care of serving static assets, normalize inbound requests, and also probably fend off at least some malicious actors", but frankly your HTTP library should already be doing all of those. Adding that extra piece means more points of failure, more potential security vulnerabilities, and for what benefit?)
Well, that depends...
At a certain scale (and arguably, not too many people will ever need to think about this), using UNIX sockets (instead of HTTP TCP) between the application and load balancer can be faster in some cases, as you don't go through the TCP stack...
> Also, why do you even want a load balancer/reverse proxy, unless your application language sucks?
Erm... failover... ability to do upgrades without any downtime... it's extra complexity yes, but it does have some benefits...
Sure (although as far as I can see there's no reason you can't keep using HTTP for that). You can go even further and use shared memory (I work for a company that used Apache with Jk back in the day). But that's an argument for using a faster protocol because you're seeing a benefit from it, not an argument for using a slower protocol because you can't be bothered to implement the latest standard.
I thought we were discussing HTTP/2 but now you seem to be invoking HTTP/3? It's even faster indeed but brings a whole lot of baggage with it. Nice comparison point though: Do you want to add the complexity of HTTP/2 or HTTP/3 in your backend? (I don't.)
The article talks about HTTP/2 but I suspect they're applying the same logic to HTTP/3.
> Do you want to add the complexity of HTTP/2 or HTTP/3 in your backend? (I don't.)
I'd like to use the same protocol all the way through. I wouldn't want to implement any HTTP standard by hand (I could, but I wouldn't for a normal application), but I'd expect an established language to have a solid library implementation available.
> But also the complexity of deployment. HTTP/2 is fully encrypted, so you need all your application servers to have a key and certificate, that’s not insurmountable, but is an extra hassle compared to just using HTTP/1.1, unless of course for some reasons you are required to use only encrypted connections even over LAN.
> So unless you are deploying to a single machine, hence don’t have a load balancer, bringing HTTP/2 all the way to the Ruby app server is significantly complexifying your infrastructure for little benefit.
EDIT: Based on the HTTP/2 FAQ, pure h2c is not allowed in the standard as it requires you to implement some HTTP/1.1 upgrade functionality: https://http2.github.io/faq/#can-i-implement-http2-without-i...
It does. Just use `--http2` or `--http2-prior-knowledge`, curl deduce the clear or not clear by `http` or `https` URL protocol prefix (clear the default).
Thanks for bringing up --http2-prior-knowledge as a solution!
- To terminate SSL
- To have a security layer
- To load balance
- To have rewrite rules
- To have graceful updates
- ...
To make sure that your connections can be snooped on over the LAN? Why is that a positive?
> To have a security layer
They usually do more harm than good in my experience.
> To load balance
Sure, if you're at the scale where you want/need that then you're getting some benefit from that. But that's something you can add in when it makes sense.
> To have rewrite rules > To have graceful updates
Again I would expect a HTTP library/framework to handle that.
That makes no sense. The latency from your load balancer to your application server should be a tiny fraction of the latency from the user to the load balancer (unless we're talking about some kind of edge deployment, but at that point it's not a load balancer but some kind of smart proxy), and the load balancer decrypting and re-encrypting almost certainly adds more latency compared to just making a straight connection from the user to the application server.
Lots of application frameworks also just don't bother to have a super high performance path for static/cached assets because there's off-the-shelf software that does that already: caching reverse proxies.
No, to keep your app from having to deal with SSL. Internal network security is an issue, but sites that need multi-server architectures can't really be passing SSL traffic through to the application servers anyway, because SSL hides stuff that's needed for the load balancers to do their jobs. Many websites need load balancers for performance, but are not important enough to bother with the threat model of an internal network compromise (whether it's on the site owner's own LAN, or a bare metal or VPS hosting vlan).
> Sure, if you're at the scale where you want/need that then you're getting some benefit from that. But that's something you can add in when it makes sense.
So why not preface your initial claims by saying you trust the web app to be secure enough to handle SSL keys, and a single instance of the app can handle all your traffic, and you don't need high availability in failure/restart cases?
That would be a much better claim. It's still unlikely, because you don't control the internet. Putting your website behind Cloudflare buys you some decreased vigilance. A website that isn't too popular or attention-getting also reduces the risk. However, Russia and China exist (those are examples only, not an exclusive list of places malicious clients connect from).
Yeah, I phrased things badly, I was trying to push back on the idea that you should always put your app behind a load balancer even when it's a single instance on a single machine. Obviously there are use cases where a load balancer does add value.
(I do think ordinary webapps should be able to gracefully reload/restart without losing connections, it really isn't so hard, someone just has to make the effort to code the feature in the library/framework and that's a one-off cost)
> To make sure that your connections can be snooped on over the LAN? Why is that a positive?
Usually your "LAN" uses whole link encryption, so that whatever is accessed in your private infrastructure network is encrypted (being postgres, NFS, HTTP, etc). If that is not the case, then you have to configure encryption at each service level, which is both error prone, time consuming, and not always possible. If that is not case then you can have internal SSL certificates for the traffic between RP and workers, workers and postgres, etc.
Also you don't want your SSL server key to be accessible from business logic as much as possible, having an early termination and isolated workers achieves that.
Also, you generally have workers access private resources, which you don't want exposed on your actual termination point. It's just much better to have a public termination point RP with a private iface sending requests to workers living in a private subnet accessing private resources.
> > To have a security layer
> They usually do more harm than good in my experience.
Right, maybe you should detail your experience, as your comments don't really tell much.
> To have rewrite rules
> To have graceful updates
> > Again I would expect a HTTP library/framework to handle that.
HTTP frameworks handle routing _for themselves_, this is not the same as rewrite rules which are often used to glue multiple heterogeneous parts together.
HTTP frameworks are not handling all the possible rewriting and gluing for the very reason that it's not a good idea to do it at the logic framework level.
As for graceful updates, there's a chicken and egg problem to solve. You want graceful update between multiple versions of your own code / framework. How could that work without a third party balancing old / new requests to the new workers one at a time.
The point being that all the code on the stack is not necessarily yours
Security and availability requirements might vary, so much to argue about. Usually you have some kind of 3rd party service you want to hide, control CORS, Cache-Control, etc headers uniformly, etc. If you are fine with 5-30 minutes of outage (or until someone notices and manually restores service), then of course you don’t need to load balance. But you can imagine this not being the case at most companies.
On one IP, sure, for one domain you could use an API gateway.
Load Balancer maing purpose is to...balance the load across multiple backends.
Just because both can be implemented with a reverse proxy such as NGINX doesn't mean it's the same thing.
DNS is usable, but some clients and recursive resolvers like to cache results for way beyond the TTL provided.
Most load balancer/reverse proxy applications also handle TLS. Security-conscious web application developers don't want TLS keys in their application processes. Even the varnish authors (varnish is a load balancer/caching reverse proxy) refused to integrate TLS support because of security concerns; despite being reverse-proxy authors, they didn't trust themselves to get it right.
An application can't load-balance itself very well. Either you roll your own load balancer as a separate layer of the application, which is reinventing the wheel, or you use an existing load balancer/reverse proxy.
Easier failover with fewer (ideally zero) dropped requests.
If the app language isn't compiled, having it serve static resources is almost certainly much slower than having a reverse proxy do it.
If your application is in a non-memory-safe language, sure (but why would you do that?). Otherwise I would think the risk is outweighed by the value of having your connections encrypted end-to-end. If your application process gets fully compromised then an attacker already controls it, by definition, so (given that modern TLS has perfect forward secrecy) I don't think you really gain anything by keeping the keys confidential at that point.
Nobody, nobody, writes application servers with the intent of having them exposed to the public internet. Even if they're completely memory safe, we don't do DOS protections like checking for reasonable header lengths, rewriting invalid header fields, dropping malicious requests, etc. Most application servers will still die to slowloris attacks. [1]
We don't do this because it's a performance hog and we assume you're already reverse proxying behind any responsible front-end server, which all implement these protections. We don't want to double up on that work. We implement the HTTP spec with as low overhead as possible, because we expect to have pipelined HTTP/1.1 connections from a load balancer or other reverse proxy.
Your application server, Gunicorn, Twisted, Uvicorn, whatever, does not want to be exposed to the public internet. Do not expose it to the public internet.
I likewise assume that all servers are insecure, always, and we do not want them exposed without a sane load balancer layer.
Your server was probably not made to be exposed to the public internet. Do not expose it to the public internet.
What application servers have you written? I have never seen an application server readme say DON'T EXPOSE DIRECTLY TO THE INTERNET, WE ASSUME YOU USE REVERSE PROXY.
* "We strongly recommend using Guincorn behind a proxy server" [1]
* "As a general rule, you probably want to: ... run behind Nginx for self-hosted deployments." [2]
* "A reverse proxy such as nginx or Apache httpd should be used in front of Waitress." [3]
For some, like uWSGI, they don't even want to talk HTTP (uWSGI supports its own protocol) and it's just assumed you're using a dedicated webserver to talk to public traffic. [4]
[1]: https://docs.gunicorn.org/en/latest/deploy.html
[2]: https://www.uvicorn.org/deployment/
[3]: https://flask.palletsprojects.com/en/stable/deploying/waitre...
[4]: https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django...
For rust, go, lua (via nginx openresty) and a few others this is a viable path. I probably wouldn't do it with node (or bun or deno), python, or similar but there are languages where in certain circumstances it is reasonable and might be better.
I'm unfamiliar with the common rust frameworks for http, but find it unlikely the situation is very different.
Also don't expose plain text traffic to the internal corpnet where most attack originate from.
That's already a fringe case. Do you really think everyone's writing web applications in a language like rust without any unsafe (or equivalent)?