Maintaining 65k open connections in a single Ruby process | Hacker News Reader

Maintaining 65k open connections in a single Ruby process

wjwh.eu

58 pointsWJW28 comments

> The maximum amount of connections achieved was 65523, just 13 short of 2^16. This is the theoretical maximum number of connections for a single IP address on a single port,

That's not right. Connections are identified by the 4-tuple of (source ip, source port, destination ip, destination port). The server ip and port are fixed in this example, but there's still 2^48 combinations of client ip/port.

> It is amazing that a problem which took huge engineering efforts to solve back in the early 2000s can now be solved easily by anyone in just a handful lines of code.

But you didn't solve the problem! The whole point of C10K was the need for scalable ways to figure out which connections could be operated on (i.e. had something to receive, or could be written to after having previously been blocked for writes). Keeping an array of sockets and just repeatedly iterating over it with the assumption that all those sockets are active is the opposite of scalable.

Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports. This seemed supported by answers like https://www.quora.com/What-is-the-maximum-number-of-concurre.... If you can enlighten me, I'd be much obliged.

You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO. Maybe for next month I'll implement basic epoll functionality to make an echo server or something :)

If I'm a server, and my ip is 192.0.2.1, and i listen on port 80, my nearby clients 192.0.2.2 and 192.0.2.3 can each establish just about 64k connections to me (port 0 doesn't generally work). I like to think of it as a 5-tuple

{Protocol, PeerA, PeerB, PortA, PortB} ex:

{tcp, 192.0.2.1, 192.0.2.2, 80, 12345} is distinct from {tcp, 192.0.2.1, 192.0.2.3, 80, 12345}

With roughly 2^32 ipv4 addresses and 2^16 client ports for each ip, you should be able to establish 2^48 connections to your server IP. Although, it usually requires a bit more than standard effort to actually use quite that many client ports per IP, by default you have to fight against high vs low ports, and anyway not a lot of OSes have a very sane allocation policy for ports.

Note also: if you're testing this on localhost, IANA has assigned you 2^24 ipv4 addresses for localhost, 127.0.0.0/8, so if i assume you're going to listen on all addresses, and connect to all addresses from all addresses, that gives you 2^24 x ( 2^24 x 2^16 ) = 2^64 connections. It might take you some time to iterate through an array that large :)

> Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports.

32b IP * 16 bit port?

> You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO.

It's not a solution to the C10K at all, because the C10K problem is outdated and was trivialised a decade back. Whatsapp was doing 2 million connections on a single box (with resources to spare) back in 2012: https://blog.whatsapp.com/196/1-million-is-so-2011 and folks were working on C10M: http://highscalability.com/blog/2013/5/13/the-secret-to-10-m...

> Maybe for next month I'll implement basic epoll functionality to make an echo server or something :)

Epoll is bad, why would you want to do that?

> epoll is bad

Source please! I'm genuine - what is bad about it?

https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... and https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-... is a pretty good baseline.

It comes up all the time but I'm not qualified to speak to it off the top of my head.

Google comparisons with kqueue from BSD and iocp on Windows.

I'm sure that your ruby can hold +100k connctions with enough RAM.

I ran 4 `ab` processes against it, each with 20k concurrent connections, so the maximum amount of connections "should" have been 80k. The fact that the actual number was so close to 65536 makes me very suspicious that there is some kernel level thing at play here. I agree that the ruby program can probably scale up much more with enough RAM.

Would this have been affected by the number of available ports on your single loopback address, assuming 127.0.0.1. You'd have to connect via multiple network devices.

Edit: comment below clarifies servers.

Was iptables running? Maybe they hit a conntrack limit

What does ulimit -a say?

It does make you wonder what was throttling it to 65k. I didn't see any mention of tweaking kernel/ulimit/systemd max open file settings.

Since client connecting to a single port? So you'd get 1 client IP, 65k client ports, connecting to 1 destination IP on 1 port.

"Eventually we got it working by using four servers each running a single instance of ab in addition to the one running our Ruby process"

So 4 source ips. Guessing they were just bumping into a (configurable) max open file setting.

Reminds me of https://medium.freecodecamp.org/million-websockets-and-go-cc...

As they say: is that a lot?

As they say: it depends :) . Practically speaking, think of it this way...

Total computer power - Cost of maintaining sockets = Available computer power to use sockets.

If total computer power - cost of maintaining sockets is close to 0, there's nothing one can really do with those sockets. If the system has 65K sockets open is it's barely breaking a sweat, you can still do plenty of work!

Well, if it's TCP/UDP connections, there's only 65k ports available to assign, so it would be all the ports. That's a lot of ports, but I'm not sure why it's supposed to be particularly hard. Off to read the article now...

Edit: As masklin referenced in reply to me[1] that jsnell pointed out, that's the ports for a single IP. So yeah, it's not all the ports, but it is all the ports for a single source IP, unless I'm missing something. You can of course assign multiple source IPs to extend this, but it is a limit to be dealt with.

1: And then deleted it, at least as of now, even though I think it was a good point.

I think you're confused about network programming in general. In a post like this, the assumption is that the concurrency is happening over a single network interface. It doesn't matter really. The most conspicuous limiting factor on a POSIX-compliant system is the number of available file descriptors. When you listen() on an address and accept() a connection, the networking stack allocates a file descriptor for that connection. Then you can handle more connections.

http://pubs.opengroup.org/onlinepubs/9699919799/functions/ac...

http://pubs.opengroup.org/onlinepubs/9699919799/functions/li...

Beej's guide is a popular intro, but I'd bet any text on network programming covers sockets.

https://beej.us/guide/bgnet/

Yes, I was actually mistaken in assuming a server would use a port only once per IP for all connections, and not just once per client IP (and even farther, client IP and client port combo).

It makes much more sense, and in fact I'd been pondering that since I commented as how that could work in practice with servers that actually keep connections open but can handle a lot of requests wasn't obvious in my prior mental model.

> Beej's guide is a popular intro, but I'd bet any text on network programming covers sockets.

I highly suspect I knew this at some point, but forgot it over the past 15-20 years since it didn't have direct relevance to most of my projects since that time. :/

Neat idea! If you're interested in production quality high concurrency Ruby check out https://github.com/postrank-labs/goliath/ which can support hundreds of thousands of concurrent connections.

That looks really interesting! I built this during a hackathon and ran out of time before managing to get beyond the 2^16 barrier. I'll definitely take a look at Goliath to see if I can borrow some tricks to push it even further. :)

The issue was solved with epoll back in 2004. All modern socket processing should be using async IO. This is just an example of how to hold the sockets open and use it more like a TCP server. Ruby might be the good choice for this if you only know Ruby.

https://illumos.org/man/5/epoll

While a best effort has been made to mimic the Linux semantics, there are some semantics that are too peculiar or ill-conceived to merit accommodation. In particular, the Linux epoll facility will -- by design -- continue to generate events for closed file descriptors where/when the underlying file description remains open. For example, if one were to fork(2) and subsequently close an actively epoll'd file descriptor in the parent, any events generated in the child on the implicitly duplicated file descriptor will continue to be delivered to the parent -- despite the fact that the parent itself no longer has any notion of the file description! This epoll facility refuses to honor these semantics; closing the EPOLL_CTL_ADD'd file descriptor will always result in no further events being generated for that event description.

Brian Cantrill on epoll https://youtu.be/l6XQUciI-Sc?t=3424

Hey, author here! This thing came out one of the internal hackathons in our company and I decided to make a blog post about because it gives some interesting insights into Ruby internals. It's not meant for production and if you need something like this "for real", I would always advise Go, Elang/Elixir or Haskell as a solid base to build on.

With MRI Ruby you'll still effectively be limited to one core due to the GIL, so if you need to do any compute intensive work for each socket instead of just sending a string you'll rapidly run into that barrier. As I state in the post, memory usage is not all that great either.

> The issue was solved with epoll back in 2004.

You mean kqueue, and that was in 2000. Epoll came later, and is bad.

i think that the `conn_waiting_area` variable is undefined in the source code

Reply on news.ycombinator.com