Hacker News Reader

Comment by terom | Hacker News Reader

Re the EADDRNOTAVAIL from socket.connect(),

If you're connecting to 127.0.0.1:8080, then each connection from 127.0.0.1 is going to be assigned an ephemeral TCP source port. There are only a finite number of such ports available, on the order of ~30-50k, which limits the number of connections from a single address to a specific endpoint.

If you're doing 100k TCP connections with 1k concurrent conections, it's feasible that you'll run into those limits, with TCP connections hanging around in some TIME_WAIT state after close().

Not that this would be a documented errno for connect(), but it's the interpretation that makes sense..

http://www.toptip.ca/2010/02/linux-eaddrnotavail-address-not... http://lxr.free-electrons.com/source/net/ipv4/inet_hashtable...

View on HN

Generally its the upper 32k ports that are ephemeral, and if your churn more than that per minute in connections, you'll run into that TIME_WAIT issue.

Hacky way to get around that is to enable tcp_tw_reuse which will let you reuse ports, but it can be risky if you get a SYN from the previous connection that happens to lineup with segment number of the current connection (which will close your connection). Shouldn't happen often, and if you can tolerate a small amount of failure is an easy way to get around this limit.

[0] http://blog.davidvassallo.me/2010/07/13/time_wait-and-port-r...

For benchmarking loopback connections, addressing really shouldn't be an issue, as you have an entire /8-subnet to split between your client(s) and server(s) (127.0.0.0/8). You would need some logic to set up eg 10.000 listening servers, and 1000.000 clients to get it working, and at some point you'd probably run into memory or other limits.

I'm a little surprised some simple googling didn't turn up any examples of this - I'm sure someone have tried it out in order to do some benchmarking of high-performance network servers/services?

Apparently ipv6 changes this to a single (loopback) address, but then again, with ipv6 you can use entire subnets per network card.

> Hacky way to get around that is to enable tcp_tw_reuse which will let you reuse ports, but it can be risky if you get a SYN from the previous connection that happens to lineup with segment number of the current connection (which will close your connection)

Actually Linux will fall back to using TCP timestamps to distinguish between different connections. Ironically people will disable timestamps too to "fix" other issues[1] which also break PAWS[2] and may cause the issue you describing.

[1] It can break with some NAT and some load balancers. Actually the way I learned about tcp_tw_reuse was when we plugged in a new load balancer. We tested everything worked fine, but as soon as we sent production traffic many connections took few seconds to complete. Took 2 weeks to find the cause and looking at packet dumps. Turns out that the issue was that the load balancer was set up in active-active configuration, so different connections had different timestamps. This caused Linux to get confused and ignore some packets. Turned out one of managers wanted to make everything performant and copied some sysctls (that included tcp_tw_reuse and tcp_tw_recycle) from Internet without much though. After restoring the setting everything worked flawlessly.

[2] https://en.wikipedia.org/wiki/Transmission_Control_Protocol#...

Localhost goes from 127.0.0.1 through 127.255.255.254. By binding each connection to a random IP in that range [1], one could get a better mileage.

[1]: https://idea.popcount.org/2014-04-03-bind-before-connect/

Reply on news.ycombinator.com