Hacker News Reader

The problem is opening >30k connections to a single host during a TIME_WAIT period, 60 seconds. 500 connections per second. For backend servers, eg, an application server talking to a database, 500 connections per second is easy (although it's preferable if they can keep-alive).

I liked Vincent Bernat's post about TIME_WAIT: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li...

Could we say it a bit stronger than that? Opening and closing connections from the same source to the same service is wasteful since you need multiple extra round-trips and it resets all the TCP dynamic tuning like CWND.

I thought your talk was great; one minor niggle: you said that the result of too many sockets in TIME_WAIT would be dropped packets; it should refuse to open the new connection if no slots are available.

Yes, it's really wasteful to have unnecessary connect()/accept() calls, plus handshaking and buffer allocation, and TCP dynamic tuning, etc.

And you're right, thanks, TIME_WAIT full should just error on the Linux client. I was thinking of a different kernel which has bugs in this area, and ends up dropping SYNs...

> Again, why? My understanding is that this controls the range of ports that the kernel selects from for new sockets; e.g., if you make a TCP connection to google.com on port 443, on _your side_ the connection is <your ip> : <a port from that range>; the default range is [32768, 61000], and this is per destination IP. (You can have two connections to two separate IPs with the same local port.) The default range is nearly 30k ports wide. Are you opening >30k connections to a single host?

This port range is sometimes known as the "ephemeral port range" and works as you described. How can you have 32 tabs open to news.ycombinator.com port 80? The source port on your machine are all different ports and from that range.

The 30K range is not to a single host, it's just all open connections waiting for data to return. That is to say, the connections are established, being established, or being torn down. If the connection was completely torn down you'd be able to reuse the port and there would be no issue.

So if your network working set is >28K ports, you may need to change this setting. Most people probably don't need to change this. If you do need to change this because you find your application is throwing errors about binding to ports in use, the above suggestion is fairly decent for setting and forgetting.

The one problem with the above suggestion is if you have an application binding to a port somewhere in the range of 10240-32768 or 61001-65535 (http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_number... for examples, it's obviously not complete). You can't just say that 10240-65535 is fair game for ephemeral connections, because inevitably an ephemeral connection will block a known port bind attempt, and your service will fail to run.

The ephemeral port range is per host because TCP connections are identified by the four-tuple: local IP, local port, remote IP, remote port. You can have active connections from localhost:12345 to both some_server:80 and some_other_server:80 or even some_server:443.

Didn't realize it was a 4 tuple, I was thinking of just the binding to the local port. Still, increasing the ephemeral range will cause problems with applications relying on ports outside the default ephemeral range.

It can, though it's not likely. The kernel only uses free ports even in the ephemeral range, so if an application binds to something in the ephemeral range first, then the kernel just won't pick it for new connections. You've effectively removed one port out of tens of thousands.

You may be in trouble if the kernel happened to choose an ephemeral port for an outbound connection and then an application tried to bind to it for receiving new connections.

TIME_WAIT: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li...

"to prevent delayed segments from one connection being accepted by a later connection relying on the same quadruplet (source address, source port, destination address, destination port). The sequence number also needs to be in a certain range to be accepted. This narrows a bit the problem but it still exists, especially on fast connections with large receive windows."

The last point means that while Netflix trusts these recommendations to be good for them today, if you are looking at this in the future, you should verify that the performance gains are still there for those changes. Perhaps you are running in a system where one or more of these no longer gives a performance gain, or is even detrimental to your overall system health.

Trust, but verify.

I'm not sure exactly what state the connections were in, but I've had very obnoxious situations where I shut down a daemon and can't bring it back up quickly because the port it listens on is 'in use'.

Harmless-seeming timeouts can cause stupid problems.