Hacker News Reader

Comment by interroboink | Hacker News Reader

A small note on "least connections" load balancing. The article says:

    Because the load balancer sits between the server and the user, it can accurately keep track of how many outstanding requests each server has.

and this is the common case. But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the load balancer. So, the load balancer doesn't know how busy the server is, unless that information is communicated separately, through some other channel. One variation is "least sessions", where the number of active sessions rather than connections is used, and as mentioned, that info is stored separately from the "main" connection, and available on the load balancer for its decision-making.

Example: https://www.youtube.com/watch?v=tAAmZ3bz8AA&t=681s

View on HN

I would in fact argue that in large systems it is infeasible to have a load-balancer sit in between. Especially if you adopt a microservice-based architecture: it doesn't make sense for every microservice to have its own load balancer to proxy all traffic. Instead the load balancer simply assigns traffic to specific services without being in the middle. It is the client's responsibility to obey the load balancer's decision.

> But there are some cases, such as when serving audio/video streams (looking at you, porn :) where the server will send its response straight to the user, not back through the load balancer. So, the load balancer doesn't know how busy the server is, unless that information is communicated separately, through some other channel.

Loadbalancer can still see it's an incoming connection as long as user is sending something (even if returning traffic goes directly) so at the very least it can keep number of connections same.

Of course one might stream 240p while other user streams 4k but for example in HAProxy there is an option to use external agent for health checks, that also have option to modify traffic weights and cut traffic to ones that are being swamped

I didn't really want to go in to direct server return. I think most people will be more familiar with using something like nginx as a reverse proxy and load balancer at the same time. It is very cool tech, though. :)

Yeah this was the most confusing part of the whole article to me, and seemed like the most important. I was like "how is it keeping track of how many connections are still being processed?"

In most situations (not all) the load balancer is actually a proxy. So when you type in example.com, the dns points to a load balancer. You connect to a load balancer, the load balancer opens a connection with the desired server (probably sitting in a private network somewhere), and proxies the traffic to the server.

You never actually connect to the server in this case. You stay connected to the load balancer the whole time, the load balancer makes a request to the server, which processes your request, and sends it back to the load balancer, which then proxies the data back to you in the response.

In this case the LB sits between you (the user) and the actual server running the application. In this case the LB knows how many connections each server has in the system because it is actually the one connected to the servers. If a server drops a connection, the LB knows because its the one that actually is losing connection. So it is fully aware of what all servers are doing in this scenario.

The only thing the load balancer might still need to do is ping servers for health checks to make sure they are online when the servers are not connected, this is also how it knows latency and other metrics.

Reply on news.ycombinator.com