Comment by cphoover | Hacker News Reader

Comment by cphoover | Hacker News Reader

This is perhaps only tangentially related... but I'm trying to do something to support sharding of websocket connections based on a user_id....

I want the connections to be distributed to separate instances of a WS server based on modulo-based or consistent hashing algorithm... above that in the stack we have a message broker or queue partitioned using the same sharding algorithm for the number of websocket servers running. This queue/message broker sends publishes messages to each web socket service which in-turn pushes the message to the client. This is how I want to handle horizontally scalable web socket connections...

The question I have is the best way to do this and the best way to migrate connections. From one service to another when/if the replication factor of web socket servers changes (we add/remove servers)...

One way I would think would be to send a disconnect message to the websocket service when this event happens and it would disconnect it's clients and then have a random delay for when re-connection happens... so as to avoid trying to reinitialize all of those connections at once. This seems sub-optimal as with consistent-hashing many of the connections will stay on the same server, and terminating those connections would be unnecessary.

What is the best way to migrate an open TCP from one server to another? Or if the connection must be closed... how can we minimize the number of connections that are to be closed.

BTW: I know I can broadcasts to all instances of the websocket server, and then unless that server holds the connection it is no-op... if the server does have a connection then send the message back... This also seems it would have a limit to how far this scaling-strategy would grow before running into limitations.

We're solving exactly this problem with Hathora. Checkout a blog post that we recently wrote: https://blog.hathora.dev/scalable-websocket-architecture/

You're welcome to checkout the docs and get started: https://docs.hathora.dev/#/buildkit/README

If you'd like to get in touch, feel free to shoot me a message: sid [at] hathora.dev

Linux has TCP connection repair, which can be used to transparently migrate an active TCP session between two Linux boxes. The TCP traffic will be paused for the duration of the migration, but the other endpoint will not notice as long as the migration is complete before TCP timeouts kick in.

https://lwn.net/Articles/495304/

Discussion: https://news.ycombinator.com/item?id=33348319

I am one of the co-founders of https://ably.com. We have built a Serverless websockets platform that is near infinitely scalable and designed to handle the complexity you’re describing such as cluster resizing, connection failures, automatic sharding and reallocations using hash rings across our global edge service etc. I’d be really interested to hear why you wouldn’t consider offloading this to a service like ours which powers the likes of HubSpot, Expedia, Spotify etc. Feel free to contact me directly @mattheworiordan if you’d prefer not to comment here. I’m really keen to hear different perspectives on what we’re doing right and where we can improve!

The simplest might be an endpoint which asks 'which server should I connect to?'

Then you can use whatever logic you want for it. If the client disconnects, it just asks again and reconnects.

Use a token from the directory endpoint to authorize the connection so they have to do it.

Can you just store the shard id for each user in a database? Each server could check if it is authorative for each incoming connection, and redirect if not.

That way, you can migrate users one by one (sending a 'redirectUserConnections(userId)' to the original server, and even pick a shard that is geographically close to the user. (If you care about race conditions during migrations, you need to be really careful though.)

Yea something like this makes sense...

race conditions during migrations are the biggest concern.... You do not want a client with a connection to server they will never receive a message on.

By curiosity, can I ask you which sharded message broker you are using? (The one "above in the stack")

Could be anything (redis pub/sub with a channel for each shard, sqs with separate queues for each shard, separate partitions in kafka, etc)...

haven't committed to using one particular technology. queues that support retries could be useful in the event that a disconnect happens messages wouldn't be necessarily lost. Something custom built on top of Redis could accomplish the same perhaps.

migrating an open socket to me sounds like it'd be more hassle than it's worth. It's possible to do, but unusual.

Can't you not involve the client and send it an explicit "reconnect to server XYZ" message, only move clients that really need to move?

Yes... but this seems to be then a timing issue to me... what happens if the replication factor changes again while they are reconnecting to the wrong server... then they will miss the second 'reconnect' message... and will stay connected to this server where it will never receive messages... perhaps an edge-case scenario.

I recommend storing the server topology somewhere that's super highly available, like Cloudflare. Then having the client choose from that. If their connection drops (use heartbeats), retry a different one you have and refresh the server topology in parallel. If the second connection fails, retry again with the new topology. This is convergent and avoids putting load on your database to connect, so you don't suffer from thundering herd problems.

Reply on news.ycombinator.com