Migrating Millions of Concurrent WebSockets to Envoy
slack.engineering
slack.engineering
This way you could restart service without killing ws connection, move all overhead of handling millions of connections to the lb.
Plus it requires the backend to support that custom multiplexing layer.
For anyone wanting to do this sort of thing, check out Pushpin [1]. It's a proxy server that can manage raw WebSocket connections on behalf of any HTTP backend. There are middleware libraries (such as [2]) to make it easy to write handler code.
Disclosure: lead dev. Came up with the idea after developing a custom connection manager thing at a previous company.
[1] https://pushpin.org [2] https://github.com/fanout/js-serve-grip/blob/master/examples...
“This part of our back-end, responsible for most of the real-time interactions happening in various clients, is implemented in Java, and accessed via a WebSocket API described here: Real Time Messaging API. A great deal of the user experience that makes Slack feel like Slack is the result of careful work on the message server. Many of the distributed systems challenges we face on the back-end are in coordinating this service with the rest of the (LAMP-ier) back-end.”
It would be really interesting to compare the current scale of the two offerings, in terms of peak concurrent users, hardware needed and engineering team size.
The user base is very different (business for Slack and gaming for Discord), but has started overlapping more and more in recent years.
Also I prefer Dhall's evaluation model (all values are at collapsed at compile time into a single output value with no side effects) to Starlark's full-fledged 'runnable' model, e.g. with the 'print' function I can print out anything I like, including possibly secrets and passwords. This is impossible with Dhall.
That said, I do wish that Dhall had something closer to Starlark's syntax and better type inference :-)
I had that feeling with nginx vs Apache too of course, but this is another level of that same thing.
This would also have the effect of allowing slack to peer more or less normally even if Slack was down (of course bots, search, etc wouldn’t work).
With WebRTC you give up the nice ultra-low-common-denominator "outbound port 443/TCP needs to work" requirement and replace it with "UDP networking generally healthy, possible to establish port mappings, possible to maintain stable port mappings over time, possible to not have mappings go away due to lack of traffic" etc etc
Ironically getting machines connected across the internet with WebRTC has so far been relatively smooth sailing.
Could you expand a bit on why MediaChannels need TURN specifically? Is there some complication that prevents just dumb relaying of the packets ala what you're suggesting?
It's definitely more complex than websockets, but removing STUN servers from the equation does simplify connection initiation a bit.
My own interest comes from many years of building multiplayer games professionally and wanting to make smaller scale multiplayer games for fun without needing to invest too much in infrastructure.
First of all, while WebRTC has its share of complexity when using it for videoconferencing, here we are talking about using the DataChannel, which is really straightforward to use and doesn't need additional infrastructure.
> not forgetting WebRTC still relies on centralized components to coordinate
It needs a centralized component to setup the connection (signaling), if it fails later, your communication channel is still up. And the good thing if you have a websocket-based chat service, is that you can directly use it for the signaling purpose with zero modifications on the back-end side.
> routers in many scenarios hate it and even where they allow it, the combinatorial explosion in possible configurations to support and diagnose between peers is a problem nobody should willingly invite unless they can't achieve a solution any other way
When using the Datachannel, your failure mode is can't establish a connection, not some hard to understand Heisenbug. All you need is to provide a centralized fallback for clients who cannot establish a connection. This fallback will depend on the centralized service being up, but in case of failure you'll keep most of your users without disturbance (at least in the first world, the network is not as WebRTC friendly in other places of the world). And because the DataChannel's API is close to the WebSocket's one, implementing the fallback is straightforward.
Though, in Slack's situation there is a good reason not to use WebRTC: they can have several thousands of people in the same channel (IIRC IBM uses Slack and have most of their employees in a shared channel for official announcements). You won't be able to do that with WebRTC[1] if a user needs to establish a connection with every other users in the channel (there's just not enough ports available). And even worse, back in 2016, Chrome's implementation of the DataChannel was so poor, you could not establish more than a handful of PeerConnection before feeling the browser's becoming sluggish (this wasn't the case in Firefox so maybe Google fixed that since then).
Also, Slack's users are likely to be in some enterprise network, which makes WebRTC more likely to fail than when you customers are home, which reduces the opportunity.
Main takeaway: WebRTC-based chat is probably not a great fit for Slack, but don't be afraid of using it: it's not hard, it combines well with your already existing centralized infrastructure, and can massively reduces the load on it.
[1] unless you want to build some fancy sparse mesh network, but this is likely overengineering.
If you do both STUN and TURN it works on most networks. I've worked at really restricted work sites, and while STUN fails at those if you have a TURN server then it almost always works.
These sort of comments are why people think webRTC is unstable while the same people use slack calls which literally use webRTC.
I might be wrong, but please don't talk about network reliability in webRTC without specifying if you have a working STUN and/or TURN setup.
> If you do both STUN and TURN it works on most networks
TURN requires clean outbound UDP/TCP connectivity which is far from ubiquitous, there are numerous corporate firewalls where "CONNECT <x>:443" is the best that could be hoped for, and even in some of those, where if the resulting connection did not include an obvious SSL handshake the connection would be immediately reset.
> Consul-template renders a new version of the host list, and a separate program developed at Slack, haproxy-server-state-management, reads that host list and uses the HAProxy Runtime API to update the HAProxy state. ... However, over the course of the day, a problem developed. The program which synced the host list generated by consul template with the HAProxy server state had a bug.
This "drain" pattern is great for maintenance, upgrades, etc too.
The only caveat is that the clients need to be given time to migrate. How long that is depends on how well the clients behave. A hot restart may be much faster.
Dependents on how many config changes you need (per day).
Besides Envoy supports it and I would call it a bonus if you can reload your configuration without client interruption. As for complicating things the implementation for hot reload isn't terribly complicated in Envoy.
Reading between the lines, I think what they would need is a way to tell clients to move to a new websocket connection at the proxy layer. I don't think there is an easy built-in way to do this in the websocket protocol, so they would have to implement something custom in their application layer. This would also require triggering custom code in the client to make a new websocket connection, start using it, and then close the old connection.
I feel like it would have been simpler to just have the client do a graceful reconnect every 5 minutes. But they probably decided to use envoy so they could have the other advantages too.
If you do restart with reconnect, many clients will need to reconnect multiple times, as it they get kicked off early, they'll tend to connect to another old server, and have to get kicked off again later. This isn't great because connection startup is usually expensive; also a (hopefully) small minority of clients may have difficculty connecting because of network issues and it may take them a substantial time to get reconnected.
If you have a working hotload path, you get clients to new server server code faster using less resources.
Load balancing software is also often behind some other layer of load balancing/availability (DNS or ECMP or BGP or CARP), and a hot load means you don't need to churn that layer, which can avoid any issues with changes in that layer.
Plus, it is just chat. We re really ok with it being down