Discord Postmortem from Friday
status.discordapp.com
status.discordapp.com
We believe this triggered a bug in the redis-py python driver we use (specifically this one: https://github.com/andymccurdy/redis-py/pull/886) that made us have to rolling restart our API cluster in the first place, to get the connection pools back into a working state. redis-sentinel had appropriately detected the instance going away, and initiated a fail-over almost immediately following the instance going offline, but due to the odd network situation that was caused by the migration (absolute packet loss instead of connections being reset) - the client driver was unable to properly fail-over to the new master. We already have work planned for our own connection pooling logic for redis-py - as right now the state of the drive in HA redis is actually pretty awful, and the maintainer doesn't appear to have the time to close or look at PRs that address these issues (we opened one that fixes a pretty serious bug during fail-over in march https://github.com/andymccurdy/redis-py/pull/847 that has yet to be addressed).
At this scale, it seems like it may be warranted to start using reliability testing in production in like with Netflix.
At the end I see mention of a library with flaws. I am curious as to which library that is, given I develop some projects in Elixir.
Reliability testing is definitely something we're interested in as we spin up more SRE/reliability focused individuals, but also has probably the least amount of cost-benefit for us (compared to engineering effort on improving the things we know need work). Some of the failure in the system we experienced is related to issues we know about, but haven't prioritized (read; had time for) yet.
For the library, we believe the bug is related to hackney and the fact it uses the high priority setting for its pool process. For some reason (this is the part we're not entirely sure on, and still spending some time investigating) this high priority process got stuck and consumed all of the scheduler time (presumably related to the earlier API degradation), breaking the distribution port and the application in a weird way. Oddly enough the systems we run on are SMP, so in theory one rogue process should not be able to have this effect.
The solution we came up so far is to disable automatic migrations. Not sure if that option actually does anything.
It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used.
My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the frame rate up and looked presentable at least to 720p.
And their lack of scalable monetization leads me worried about its longterm success as a platform - they are adding more cost-intensive features and continuing to try to support it with what is essentially a $5 monthly donation model.
Complaints aside, I love the service. Echoing what avree said, long term monetization worries me. I would like to see discord survive, but its story looks similar to Trello right now in terms of monetization.
FWIW, it generally works pretty well, and overall, Discord is a fantastic product that I'm happy to be using.
This probably involves not being Electron (from my own adventures in the area), so I don't hold out much hope, but it keeps Discord from replacing Extremely Expensive And Bad solutions like SkypeTX for me.
I'm trying to figure out what the actual context in question is, particularly in terms of technical connectivity. Is this being used for remote DJing? Or conferencing? Or an audio recording situation?
If you're prepared to throw money at the situation, it's possible this may be fixable with a simple bespoke solution. I say "possible" because, unfortunately, I just did some digging and found https://bugs.chromium.org/p/chromium/issues/detail?id=453876:
> Unfortunately we don't support multi-channel > 2 nor multiple devices at the moment.
> ...
> Are there any future plans to support these two features? Is this a w3c issue or a Chrome issue?
> ...
> I am quite skeptical about this; I was told this requires a huge change in our WebRTC-side infrastructure, but I am not sure what the current status is.
> The spec indicates getUserMedia can be configured with 'channel count', so I assume this is Chrome issue.
That immediately nukes WebRTC :(
Could make for a fun project. I'm very fascinated with audio handling myself and this sounds interesting, but I'm unsure I'd personally have the skills (or mental stamina/attention span :< ) to be sure I could follow through. I'm also only on a Linux box, which brings up the platform-native problem.
Honestly, the best answer is probably to continue using multiple Skype instances. Which is gross. But, y'know.
I see. I get the impression this is collaborative podcasting with multiple people that have multiple microphones. (I can't figure out why else you'd need multichannel A+V transport.) FWIW, it does sound like Skype is probably your best bet for the time being (unfortunately). It's simple, it works for everyone, etc.
It's also annoying having to adjust each users volume slider individually, I wish Discord would just do that automatically and normalize the levels of everybody.
I have used Discord quite a bit for gaming, but it hasn't proved a better option than Slack to me (at least in a work context).
We're stuck with Skype for official stuff and telephony though.