How Discord handles over a million requests per minute with Elixir’s GenStage
discord.engineering
discord.engineering
Basically it is just some type of feedback so that you don't overload subsystems. One of the most common failure modes I see in load balanced systems is when one box goes down the others try to compensate for the additional load. But there is nothing that tells the system overall "hey there is less capacity now because we lost a box". So you overwhelm all the other boxes and then you get this crazy cascade of failures.
Corollary: If you have 2 boxes, each of them has to be able to handle all the traffic, so you can't save money by using smaller boxes :D
Corollary #2: If you have 2 datacenters, each of them has to be able to handle all the traffic, so you burn a lot of money :D
Hopefully your clients retry server errors with exponential backoff, if you lose a datacenter you can send half of the requests 503 until you're back at a manageable load.
Hopefully detecting the load/generating the 503 is really cheap.
Nope. Gotta have the capacity at hand. You can't rely on getting it later when everything is fucked and you're already down.
Down = Loosing clients = Loosing money = NoNonononono
Source: Lost 4 million dollars last week because of that.
Exponential backoff is to prevent cascading failure (and retry causing DDoS), that is a failsafe in case of failure, but an excuse for failing.
More broadly, it's super easy to find yourself in a situation where throughout is constrained by an external resource. Especially common in any business that has partners (i.e. most of us) or has some kind of fulfilment to do.
It may display that there are "no more tickets for this concert [at the moment]" but it shouldn't display a 503 error to half of the users and it shouldn't overload the external-ticket-reservation-system either.
That makes an interesting problem where some systems will have to handle queueing/buffering/waiting/erroring nicely for each other =)
Running at a 2N (or more) configuration 24/7 is a great way to light money on fire.
This can be mitigated though if you have say 3 datacenters and you want to be able to run with 1 down, because that divides the load.
Till DynDns goes down for 8 hours and you're the only player left in town.
Then you profit the whole day and afterwards you check the revenues you've not lost + the new users and revenues that came in with the circumstances + the support calls and the engineering emergency that were avoided... turns out all of that cover the yearly datacenter expenses by tenfold 8)
YMMV.
You have 3 boxes, they're all running at 100%, or worse if the load is not perfectly balanced (it never is). You've got bad latency and you're up for cascading failures. Your system is in bad shape.
Now, supposing you have to deploy your application. Gotta take a server offline temporarily, the load will be 100% on the last 2 servers left... that can only take 66%. Booya! You're down :p
That's what happen when people are cheap on servers, in high performance systems.
In your example, two boxes would have meant downtime no matter the capacity of each box too. All your example demonstrates is a reason why you might want to have capacity that allows taking more than one box offline....
Which could be handled with bigger servers, or more smaller servers.
I write this to illustrate that you must have a significant batch of servers (about 5) in order to achieve costs savings (by using smaller servers) WHILE not affecting reliability.
Well, that is, if you care about rare cascading failures and performances issues.
This is a good article about overload and back pressure. It also lists some tools in Erlang to solve these sorts of issues. It also mentions genstage (very) briefly.
It's also part of http in a lot of ways. Browsers can make N simultaneous connections to a server. If those requests get queued up on the server side, then the client won't proceed with a new request until one finishes.
The pattern I see all to frequently is asynchronous worker queues. Those can very easily undermine backpressure. The rise of Ruby/Python and their limited concurrency models has placed a taboo on synchronous operations. However, synchronicity natural lends itself to backpressure.
Ignoring Elixir and Erlang - when you discover you have a backpressure problem, that is - any kind of throttling - connections or req/sec, you need to immediately tell yourself "I need a queue", and more importantly "I need a queue that has a prefetch capabilities". Don't try to build this. Use something that's already solid.
I've solved this problems 3 years ago, having 5M msg/minute pushed _reliably_ without loss of messages, and each of these messages were checked against a couple rules for assertion per user (to not bombard users with messages, when is the best time to push to a a user, etc.), so this adds complexity. Later approved messages were bundled into groups of a 1000, and passed on to GCM HTTP (today, Firebase/FCM).
I've used Java and Storm and RabbitMQ to build a scalable, dynamic, streaming cluster of workers.
You can also do this with Kafka but it'll be less transactional.
After tackling this problem a couple times, I'm completely convinced Discord's solution is suboptimal. Sorry guys, I love what you do, and this article is a good nudge for Elixir.
On the second time I've solved this, I've used XMPP. I knew there were risks, because essentially I'm moving from a stateless protocol to a stateful protocol. Eventually, it wasn't worth the effort and I kept using the old system.
This service buffers potential pushes for all users being messages, that then watches the presence system to determine if they are on their desktop or mobile (this is millions of presence watchers and 10s of millions of buffered messages), and users are constantly clearing these buffers by reading on the clients and finally when a user is offline or goes offline we emit their pushes to them (which is what this article talks about). This service was evolved from our push system from the game we worked on and when it just did pushes only and no other logic it could push at 1m/sec in batches, but its responsibility has changed.
Context matters :)
For example, workers could discard messages older than some threshold, quickly emptying the queue if there are expired messages. Clients might not even queue messages if the queue is currently too long, perhaps even providing a convenient signal for them to back off from their most chatty behaviour.
Some messages will not be delivered on time if there is significant backpressure. There is not much you can do about it, apart from avoiding choking yourself.
Perhaps the queue could work with a LIFO policy, to help at least some messages go through in time instead of having most messaged delayed near to the expiration threshold.
Then I would just say, XMPP is new and fancy, but consider the old fashioned stateless HTTP interface. When I was implementing my own service, I was worried Google is not going to handle the load. Since we were partners with Google for a good while I was able to climb the ladder of people to get an answer, and plow through their closed-door policy for questions such as "Will you guys handle this load? (5M msg/min)". I wrote a huge email explaining every edge case and what I'm doing. The answer was "We will handle it.". No detail, no context, no buts. I wasn't confident at all. But in the end, they did handle it :)
Could you explain why using RabbitMQ is more transactional?
[1] "We've raised over $30,000,000 from top VCs in the valley like Greylock, Benchmark, and Tencent. In other words, we’ll be around for a while."
> mine that data [...] more addictive [...] games
Oh, fuck no. No, no, no. I can not think of a more abusive thing for a company to do to its customers than that suggestion right there. How little respect would a company have for their fellow humans that anyone could even consider such a move?
Maybe that's just a failure of imagination on my part, but I'm ok with that.
I don't mean to imply that you yourself are being dishonest or that you will go back on your word at some point. I'm sure you have noble intentions and want the best for your users. But it might be beyond your control.
If the company gets sold or goes public, you could have drastic changes in management and their philosophy. You acknowledge this outright in your privacy page, under the "OUR DISCLOSURE OF YOUR INFORMATION" heading:
https://discordapp.com/privacy
Discord has a policy in place that explicitly says they can do whatever they want with the information without limit, including selling it or transferring it without consent. This directly conflicts with employees' public claims about intentions.
The difference is that one of them is a legal acknowledgement end users must make before they use the software and the other is a 'feel good' thing to hear on someone's blog or forum post.
If they just wanted protection against an accidental slippage of data then that privacy page could be changed substantially. Instead, they pave the road for the explicit sale of data at a later time.
Or, if they wanted to leave the option open in the future they could say "This privacy policy is subject to change" and give users an opportunity to opt-out when it changes without historical data being subject to undisclosed future use. But this weakens their value in an eventual acquisition by someone who wants to monetize the data.
As it stands right now, in an eventual acquisition or even just some internal shifts of philosophy in the organization, all historical data is up for grabs for any potential use.
Discord really shouldn't have employees state that they will never sell users data when they explicitly allow and plan for that option. It may not be intentional dishonesty, but it comes close.
No, if it's from a known employee, it counts as a legal advertisement.
How about if streamers wanted to put ads in their channel, and the company would take a little handling fee? eh?
"Support this other streamer/studio/client, subscribe to remove ads"
Destroy IRC
Getting flagged for a Danger 5 quote? Someone just lost all their nerd cred.
They have found a pretty good niche with a lot of users. Just like how reddit did with subreddits. There are a lot of "smallish" communities out there, based around games, likes, twitch/youtube channels, etc, and they all need some place to hang out when they're not in saig game/channel.
I think right now what matters most is who comes out winner with the biggest community, monitezation can come later.
As soon as screensharing lands it'll be an across the board upgrade to Hipchat at my work (only missing feature I can think of is video calls, but quality over hipchat has always been a bit sketchy so we usually fall back to Hangouts).
[0]: www.discordapp.com
If not, I don't understand why they bothered mentioning "how will we make money?" on their website, and then why they didn't answer with, "By charging game vendors a ton of money for the SDK!"
In other words, if you and one friend join a game you can either talk/listen to that one friend alone or the entire 6 man team, and switching between the two requires multiple menu clicks.
So we use Teamspeak alongside the game to allow us to talk privately with each other seprate from the whole team in game. With any modern gaming PC the "wasted RAM and CPU cycles" are negligible.
I would happily still use Discord if they provided the exact same thing with a monthly fee. Hopefully at some point they throw in some extra features for a pro version and start charging.
I just use Discord for gaming and haven't used Slack a lot, but I think Discord will be great for work as soon as they release search.
GenStage has a lot of uses at scale. Even more so is going to be GenStage Flow (https://hexdocs.pm/gen_stage/Experimental.Flow.html). It will be a game changer for a lot of developers.
How many is a few? It looks like the buffer reaches about 50k, does a few mean literally in the single digits or 100s?
This system seems great for at most once delivery. I wish I had more problems to solve with that constraint.
FWIW, the buffer only fills up to the peak about once a month. Load shedding is the last ditch effort in catastrophic situations so that everything doesn't fall apart.
- To avoid OOMing the Erlang VM. - If the notification queue is backed up then older notifications are not worth delivering if we can speed up delivering of more recent ones.
If the buffers are filling faster than the servers can clear them, then you're headed for "catastrophic" failure anyway and notifications are going to get dropped regardless. You can handle it more or less gracefully while you spin up more capacity.
The other choice is to always keep around more spare capacity, but that can get expensive if as they describe these drastic peaks happen once a month.
The other category which I meant is that in such situations the system should not run into inconsistent/weird behavior which is catastrophic from the end user point of view (not the system). E.g. in a chat application if user A sends a message to B and on his client he gets an acknowledgement that the message has been delivered. However if the server under high load simply drops the message before forwarding it to B that user might never get it. A sees that something was delivered while in reality it was not. If A depends on that information it surely might be catastrophic to him.
All in all you should have a complete system design that even under high pressure works deterministically for the users. E.g. user A only gets an ACK that the message was sent after it was somehow persisted on the server. And if the server can't deliver the message to the other client because it was dropped it is still marked somewhere for retry later on. Or it will get fetched at a later time by the client through some poll operation.
As you say, you might not get the notification pushed to the device, but you should still see the message if you open the messaging app as normal.
What if the Push Collector is down or has a random bug where it throws away XX% of requests for no good reason? How would you know if you don't instrument the other end? Something like StatsD works fantastic for this, but also just logging those failures and using a log search/aggregation tool like Kibana or Splunk would be a step in the right direction.
Makes me think of the Abraham Simpson quote: "My car gets 40 rods to the hogshead and that's the way I likes it!"
The same is hardly true for the conversion of minutes to seconds.
The requests per minute number is an average.
The requests per second number should be given for peak load. That is a very important metric, a system has to be scaled to sustain the peaks, not the average.
We'd need to know the traffic pattern to know the multiplier, that is certainly not 60 :p
QPM is a useless metric. When talking about distributed systems from engineering point of view, you always want to use QPS. QPM is simply not fined-grained enough to show whether the traffic is bursty or not. For example in this particular case, when you say 1M QPM that can mean anything - they might be idle for 50s and then get 100k QPS for the next ten seconds, or they might be getting 15k QPS all the time (like it's visible on the graph). Distributed systems are designed for the peak workload, not for the average one. Using misleading numbers like QPM leads to bad design and sizing decisions.
The only case where you would use QPM, QPD and similar metrics is when you want to artificially show your numbers bigger than they are (10M transactions a day sounds better than 115 transactions a second). But those should be used by sales, not by engineers.
Edit: Also worth mentioning, the 50k buffer is for a single server, we run multiple push servers in the cluster.
We currently have 3 machines doing this for millions of concurrent users. At the writing of this article it was 2 machines.
These are getting easily over 30,000 requests a second each about updating queues for new messages. And also are subscribed to presence events from our presence system to millions of people. It is a very busy service ensuring we only deliver messages to people not at their computer.
Unfortunate that the final bottleneck was an upstream provider, though it's good that they documented rate limits. I feel like my last attempt to find documented rate limits for GCM/APNS was fruitless, perhaps Firebase messaging has improved that?
It seems to have totally taken over a space that wasn't even clearly defined before they got there.
It's just too bad there are a dozen IMs/voice/video and a dozen slacky/feed companies.
I feel like everything down with Discord could be done with IRC in a open source way. IRC for the 21st Century?
IRC for the 21st century IS these apps like slack, discord, etc.
Bundling up everything into a centralized, proprietary package will go bad for the consumers eventually. But for now they'll trade away their freedom for convenience and enjoy it.
IRC is a fine platform. That doesn't mean it's what everybody needs/wants.
"wait, why are you walking away from me, i'm talki-"
Edit: Matrix already works great as an IRC bouncer for networks which have been bridged.
Then again, using a monospace font like I did for years (ssh+screen+irssi) wastes a lot of screen area just because every character is so wide.
But at some point we heard of Discord, which posed itself as a chat/vent replacement, started using it, and it just works. Which is huge, since the other stuff generally didn't (Axon was actually good).
TS supports plugins. No lag issues, can run on your own server. Closed source.
Mumble is open source, no lag issues. Interface is slightly less good than TS. Supports SSL.
Discord, like you say, Just Works (tm). It is very easy to use, the interface is amazing, its in active development, and setting up a server is free. It also works in the web browser.
If you're into Blizzard games, Battle.net recently added native VoIP in their client. The advantage that has as Blizzard gamer is you don't have to install any 3rd party software.
This is what makes Discord so good. Before Discord, when I wanted to play online with a friend for the time, I'd have to convince them to download Mumble or Ventrilo, teach them how to connect to a server, and help them set up their mic. With Discord I just send them a link and we're talking. They can get the client later, and having a persistent chat area is a fun way to build up a sense of community.
The average per minute only gets to be used because many systems have so little load that the number per second is negligible.
I'd definitely be able to put to use things like flow limiters and queuing and such, but none of my company's projects use Elixir :(
Highly recommend the Reactive series of libs. They're typically very well done.
The guy below is right that Akka is perfectly suited.
Some implementations are listed here: http://www.reactive-streams.org/announce-1.0.0
- 100 /s = typical limit of a standard web application (python/ruby), per core
- 1.000 /s = typical limit of an application running on a full system
- 10.000 /s = typical limit for fast systems (load balancers, DB, haproxy, redis, tomcat...).
- Over 10.000/s You gotta scale horizontally because a single box [shouldn't] can't take it.
The difficulty depends on the architecture and what the application has to do (dunno, didn't go through the article). You make something that can scale by just adding more boxes, then it's trivial, just add more boxes. Well, it's gonna costs money and that's about it.
So no. Not a big deal at all... if you've done that before and you've got the experience :D
The 6 figures benchmarks that send/receive 1 byte data with all unsafe flags enabled are not representative of real usage.
At 100k requests/s on a load balancer. The HTTP headers alone (450 bytes) with zero content are more than 1300 MBps of traffic. And well, the CPU will bottleneck long before that amount of requests to parse.
That makes it possible for one processor heavy operation to take over and slow down everything else. BEAM ensures that if you have millions of request coming through and suddenly 1000 4 day long operations kick off on the machine, that the millions of normal, smaller operations continue responding and performing as expected.
Fairly critical for the stability of real time systems.
The other piece here is that these processes are cheaper on the BEAM than any other platform in terms of RAM cost.
.5Kb / process on the Erlang VM. A goroutine in Golang is the next closest at 2kb.
The two combined are one of the big reasons why benchmarks don't tell the whole story with Erlang/Elixir. It's harder to measure consistency in the face of bad actors.
So... get 100 firebase accounts and blast them in parallel.