Hacker News Reader

Comment by SteveNuts | Hacker News Reader

The last two companies I've worked for have paid for Nginx+ since software LB is all we really need.

Handling a few thousand RPS is nothing to nginx, and doesn't require fancy hardware.

That said, it replaced Kemp load balancers, which it seems is the next biggest competitor in the hardware load balancer appliance space.

View on HN

The world has moved on in the sense that "good enough" and cloud eats into their balance sheets I'm sure, but there's loads and loads of banks and legacy enterprises that maintain their ivory tower data centers and there's nothing to replace these with AFAIK. Google has Maglev, AWS perhaps something similar, MS no idea, everyone else just buys F5 or doesn't need it.

Lots of people are using haproxy

My org moved off nginx for haproxy after we learned that (at the time, maybe it changed) reloading an nginx config, even if done gracefully through kernel signals, would drop existing connections, where haproxy could handle it gracefully. That was a fun week of diving in to some C code looking for why it was behaving that way.

How did you come to that conclusion? I always believed a reload spawned new workers and let the old one drain off.

Yes I reload nginx all the time and it doesn’t drop connections. I just use the debian nginx package. Not sure what the gp is talking about.

Nginx abruptly drops http/1.1 persistent connections on reloads. This has been an issue forever and Maxim refused to ever fix it, saying it was to spec (yes it was, but there are better ways to deal with it).

It’s a reason why many large, modern infra deployments have moved away from nginx.

It doesn't drop it, it's just not persistent on reload, isn't that what you mean? Actually dropping a connection mid-request is something I haven't seen nginx (or indeed Apache) do for many years despite doing some weird things with it.

I can see where you're coming from, but it's not unreasonable behaviour, is it? Connections needs to migrated over to the new worker and that's how all major servers do it. If that's a problem then maybe something designed as proxy only instead of a real server is the way to go?

Isn't the typical behaviour of an application to re-establish the persistent connection on demand? I wonder what the requirement is to have these persistent with no timeout.

Yep. Persistent connections are bound to fail sooner or later anyway, so a robust application should have its own recovery.

It doesn't drop mid request. But it closes the TCP socket abruptly after any in flight requests are completed. Clients have no idea the connection is closed, and try to reuse it and get back a RST. In heavily dynamic environments where nginx reloads happen frequently, it leads to large amounts of RSTs/broken connections and high error rates (you can't necessarily auto-retry a POST, a RST could mean anything).

The sane approach is connection draining - you send a `connection: close` response header on the old worker, then finally remove any remaining idle connections at the end of the drain.

In http/2 it's not an issue as it has a way for the server to explicitly say the connection is closed.

I'm late to return to the thread, but this was the exact scenario we hit. We had it behind a CDN as well as behind an L4 load balancer for some very high volume internal consumers, and when it would just blast back RST packets, the consumers would freak out and break connection, returning errors on their end that weren't matched in our logs, unless we were lucky and got a 499 (now Maxim can talk about standards). As a general purpose reverse proxy for many clients on the Internet, I'm sure that's fine, but in our use case this made nginx unpredictable and no longer desirable.

What you describe is basically how persistent http works, is it not? Even a persistent connection terminates at some point. Which web server does not work like that?

I guess you could send the connection header on draining, but anything less than what the big servers do is bound to cause some compatibility problem with some niche client somewhere. I can certainly see why a web server with millions of installs would be reluctant to change bevaviour, even if it is within spec.

I can only guess at the use case here, but maybe something designed from the start as a stateless proxy and not a general purpose web server would be a better fit.

nginx supports graceful reloading and I’m pretty sure it has for a very long time - there are references to it in the changelog from 2005

https://nginx.org/en/docs/control.html

we went in the opposite direction, not because haproxy was bad, just because nginx had a simpler config, and i think we were paying for haproxy but don't pay for nginx.

all that said, neither drops existing connections on reload

Another issue with nginx IIRC is that it allows HTTP request smuggling, which is a critical security vulnerability.

That's been fixed for years. The CVE I can find was resolved in 1.17.7 (Dec 2019), and further hardening was applied in 1.21.1 (Jul 2021).

Can nginx send requests upstream over HTTP/2?

I see this question has remained unanswered for a couple of years.

https://security.stackexchange.com/questions/257823/what-are...

It cannot. There's more detailed reasoning at https://trac.nginx.org/nginx/ticket/923, but the tl;dr is:

> nginx is already good at mitigating HTTP desync / request smuggling attacks, even without using HTTP/2 to backends. In particular because it normalizes Content-Length and Transfer-Encoding while routing requests (and also does not reuse connections to backend servers unless explicitly configured to do so)

Amazon used to run entirely behind Citrix NetScaler hardware; no F5 at all. This was back in the early 2010s so I assume things have changed by now.

Yup - there was a massive internal push to move off of SSL terminating LBs back in ~2018

How come?

Cost.

Now, SSL termination is done at the host level, using a distributed SSL termination proxy developed by S3 called "JBLRelay"

AVI if you're using VMware already

I'm pretty sure that AVI just wraps Nginx, even though they claim otherwise.

I think this because Nginx has a bunch of parsing quirks that are shared with AVI and nothing else.

Reply on news.ycombinator.com