Hacker News Reader

Comment by jhgg | Hacker News Reader

At Discord, we maintain long lived web-sockets, and need to be able to deploy our real-time elixir based services (which runs on the BEAM VM just like Erlang) without any disruption to user traffic.

To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed.

We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.

View on HN

>For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.

Is this based on Alex Snoeren's TCP Connection Migration draft ? - https://datatracker.ietf.org/doc/html/draft-snoeren-tcp-migr...

No, it's entirely in user-space. Our websocket protocol can just negotiate a resumption. https://discord.com/developers/docs/topics/gateway#resuming

This is so cool! I often wondered how possible would it be to keep a TCP connection alive during a server restart. How is the connection severed and the client not being aware of it works?

Probably a layer of indirection. One layer acting as the mailbox for sessions which then communicates with whatever services handles the connection to the user client. Effectively builds up a buffer until the service tree is restarted. Pub/sub abstractions basically. I could be wrong.

Depends what you mean by server.

Some old game server processes would support a "copyover" function. They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket. Unsophisticated, but it works. It's completely transparent to the client because the connection is never actually closed. The worst the client might experience is some latency as their input gets buffered by the OS while the server process restarts.

Of course the connection would not persist over a physical server restart (or something like a pod being killed or whatever). I imagine it's possible to move a connection between nodes, though, with a fairly similar process.

>They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket

Nope. When a Process exits for whatever reason, the OS closes and releases all resources connected with all File/Socket descriptors for that Process.

You will have to architect your System to explicitly accommodate this scenario. Some techniques here : https://stackoverflow.com/questions/55006657/can-i-allow-my-...

Ah, yeah, "restart the process" was not the right description of what was happening in the cases I was thinking of - they'd exec themselves to hot reload code which I see is the first suggestion on the SA answer you linked.

Not quite. Terminating a process releases its filehandles, and if any sockets are now unreferenced, they are closed. But if there is a live filehandle in another process, the socket stays alive.

You can send socket file handles over UNIX domain sockets, so you could use this to export your list of sockets to persist over restart.

Yep (reference counting in play), It is already mentioned in the SO answer i linked to.

>I often wondered how possible would it be to keep a TCP connection alive during a server restart

If you define a "Server" as a logical piece of functionality not directly mapped to a OS Process, then it can be done. See https://news.ycombinator.com/item?id=32067570

Great context, thanks.

Do you have a ballpark feel for the cost of just terminating connections? Is it mainly a tail latency thing where you might try to push a message down a socket that is currently reconnecting, and have to wait way longer than normal? Or a CPU/network overhead thing due to how many open connections you have, with a full restart causing a thundering herd of activity?

Damn, that's cool. Does Discord have any blog posts talking about this?

Just seconding this - would love to hear more about Discord's use of Elixir

And thirding, I'd like to subscribe to this newsletter.

I don’t know if discord uses that (guess not) but it’s partially built into beam: beam can have multiple versions of the same code loaded, and old code will only call new code when using “external” calls (module:function()), as long as you keep calling bare function() you live in the old code.

Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.

The bit about migrating to other nodes doesn't flow from the builtin code loading functionality.

Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun.

[1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.

This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable.

We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.

Reply on news.ycombinator.com