Hacker News Reader

Comment by lol768 | Hacker News Reader

These events seem to be happening almost on a monthly basis now. IRC was never this unreliable and at least with netsplits it was obvious what had happened because you'd see the clients disconnect.

IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice.

It's really poor.

View on HN

Its especially strange when you think about how unoriginal Slack's product domain is, and how comparable, and in some cases small, their userbase is.

* iMessage, which likely handles something in the range of 750M-1B monthly actives.

* WhatsApp, 2B users [1], though no clarity on "active" users.

* Telegram, 400M monthly actives [2]

* Discord, 100M monthly actives [3]

* Slack, 12M daily actives [4]

* Teams, which is certainly more popular than Slack, but I shudder to list it because its stability may actually be worse.

The old piece of wisdom that "real-time chat is hard" is something I've always taken at face-value as being true, because it is hard, but some of the most stable, highest scale services I've ever interfaced with are chat services. iMessage NEVER goes down. I have to conclude that Slack's unacceptable instability, even relative to more static services like Jira, is less the product of the difficulty of their product domain, and moreso something far deeper and more unfixable.

I would not assume that this will improve after they are fully integrated with Salesforce. If your company is on Slack, its time to investigate an alternative, and I'm fearful of the fact that there are very few strong ones in the enterprise world.

[1] https://blog.whatsapp.com/two-billion-users-connecting-the-w...

[2] https://techcrunch.com/2020/04/24/telegram-hits-400-million-...

[3] https://wersm.com/discord-reaches-100m-monthly-active-users-...

[4] https://www.cnbc.com/2019/10/10/slack-says-it-crossed-12-mil... (this was also announced on Slack's blog, but that's down).

I didn't realize that Discord has way more active users than Slack. I'm glad, Discord is a fantastic service in my experience. It's a shame they got shoe horned into a mostly gaming oriented service. I've never had a class or worked somewhere where Discord was a considered solution instead of Slack, but I can't think of anything that Slack does better (in my experience). In general, I think Discord has the best audio and video service that I've used, especially kicking Zoom to the curb.

Discord is definitely in the same realm of scale as Slack, and probably bigger (they publish different metrics, so its hard to say for sure).

The really impressive thing about Discord's scale is the size of their subscriber pools in the pub-sub model. Discord is slightly different than Slack in the sense that every User on a Server receives every message from every Channel; you don't opt-in to Channels as in Slack, and you can't opt-out (though some channels can be restricted to only certain roles within the Server, this is the minority of Channels).

Some of the largest Discord servers have over 1 million ONLINE users actively receiving messages; this is mostly the official servers for major games, like Fortnite, Minecraft, and League of Legends.

In other words, while the MAU/DAU counts may be within the same order of magnitude, Discord's DAUs are more centralized into larger servers, and also tend to be members of more servers than an average Slack DAU. Its a far harder problem.

The chat rooms are oftentimes unusable, but most of these users only lurk. Nonetheless, think about that scale for a second; when a user sends a message, it is delivered (very quickly!) to a million people. That's insane. Then combine that with insanely good, low latency audio, and best-in-class stability; Discord is a very impressive product, possibly one of the most impressive, and does not get nearly enough credit for what they've accomplished.

For comparison; a "Team" in Microsoft Teams (roughly equivalent to a Discord Server or Slack Workspace) is still limited to 5,000 people.

>I didn't realize that Discord has way more active users than Slack

Keep in mind you're comparing daily active users vs monthly active users. I'd guess most slack users are online weekday for pretty much the entire day (because it's for work and your boss expects you to be online), whereas a good chunk of discord users are only logging in a few hours a week when they're gaming.

At 12:00pm EDT on a workday:

Minecraft official server: 190k online users. | Fortnite official server: 180k online users. | Valorant official server: 170k online users. | Jet's Dream World (community): 130k online users. | CallMeCarson server (YouTuber): 100k online users. | Call of Duty official server: 90k online users. | Rust (the game) official discord: 80k online users. | League of Legends official server: 60k online users. | Among Us official server: 50k online users.

Their scale is insane. Even with their usage spiking during after-hours gaming in major countries, their baseline usage at every hour of the day, globally, makes it one of the most used web services ever created.

Slack's DAU and MAU numbers are probably pretty close to one-another. Discord's MAU/DAU ratio is probably bigger than Slack's. That just means that Discord is, again, solving a harder problem; they have much bigger (and more unpredictable) spikes in usage than Slack. Yet, its a far more stable and pleasant product.

Our secret sauce is Elixir/BEAM and Rust :)

Well for the real time side, I can't tell you how big a boon it's been to build our platform on top of Elixir/BEAM. Hands down the best runtime / VM for the job - and a big big secret to our success. Where we couldn't get BEAM fast enough - we lean on rust and embed it into the VM via NIFs.

2021 is the year of rust - with the async ecosystem continuing to mature (tokio 1.0 release) we will be investing heavily in moving a lot of our workloads from Python to Rust - and using Rust in more places, for example, as backend data services that sit in front of our DBs. We have already piloted this last year for our messages data store and have implemented such things as concurrency throttles and query coalescing to keep the upstream data layer stable. It has helped tremendously but we still have a lot of work to do!

To help scale those super large servers, in 2020 we invested heavily in making sure our distributed system can handle the load.

Did you know that all those mega servers you listed run within our distribution on the same hardware and clusters as every other discord server - with no special tenancy within our distribution. The largest servers are scheduled amongst the smallest servers and don't get any special treatment. As a server grows - it of course is able to consume a larger share of resources within our distribution - and automatically transitions to a mode built for large servers (we call this "relays" internally.) At any hour, over a hundred million BEAM processes are concurrently scheduled within our distributed system. Each with specific jobs within their respective clusters. A process may run your presence, websocket connection, session on discord, voice chat server, go live stream, your 1:1/group DM call, etc. We schedule/reschedule/terminate processes at a rate of a few hundred thousand per minute. We are able to scale by adding more nodes to each cluster - and processes are live migrated to the new nodes. This is an operation we perform regularly - and actually is how we deploy updates to our real time system.

I was responsible for building and architecting much of these systems. It's been super cool to work on - and - it's cool to see people acknowledge the scale we now run at! Thank you!! It's been a wild ride haha.

As for scale, our last public number perhaps comparable to Slack is ~650 billion messages sent in 2020, and a few trillion minutes of voice/video chat activity. However given the crazy growth that has happened last year due to COVID - the daily message send volumes are well over the 2 billion/day average.

Assuming this is real - very interesting read, thanks for sharing.

I really agree Discord is amazing and wish I could use it for work instead of Slack.

I think the big things that prevent it from being adopted more for professional use is the lack of a threading model (even though I hate it when people use threads in Slack) and the whole everyone in every channel except for role-based privacy settings. The second one especially is a big deal because you can't do things like team-only channels without a prohibitive amount of overhead.

That said (with zero knowledge of their architecture) I have to feel like both of those missing features aren't too terribly hard to build. Its very likely Discord is growing as a business fast enough on the gaming and community spaces they don't feel the added overhead of expanding into enterprise (read: support, SLAs, SOC, etc) makes sense and are waiting until they need a boost to play that card.

> I think the big things that prevent it from being adopted more for professional use is the lack of a threading model

They do have a threading model now (if you are talking about replying to a message in a channel and having your reply clearly show what you are responding to). If you are talking about 1-on-1 chats with other people in your same server then yes, that is still lacking IMHO in discord. The whole "you have to be friends" to start a chat (or maybe that's just for a on-the-fly group) is annoying.

Discord gives every user an identity that is persistent beyond the server; you have a Discord account, not a server account. Slack does the opposite. Enterprises would hate Discord's model, as they prefer to control the entire identity of every user in their systems, such that when they leave the company they can destroy any notion of that identity ever existing.

Absolutely agree. I like the 1 main discord account but I wish I could have 1 "identity" per-server as well. I don't love that I am in some discords that I don't want tied to my real name and others where I've known these people for over a decade and would see in person multiple times a week (before the pandemic). I know you can set your name per-server but you can't hide your discord username (or make it per-server) which sucks.

Agreed completely. Discord has always been much smoother for me than Slack, and the voice/video chat quality is literally the best I've ever seen anywhere. If they made their branding a bit more professional and changed the permission model from the (accurate) garbage you described to something closer to Slack then I think Slack would be doomed.

We use Discord exclusively at my day job.

We have a few bots we've integrated with things (deployment, stats, etc).

We use it for all our voice/video calls.

Edit: We've got roles setup well for things like contractors, devs, marketing, etc, so it's easy to lock down different conversations in channels.

It's been fantastic.

The only thing I'm not a huge fan of is the (IMO) poor implementation of threaded discussions.

Edit: it definitely has issues with connectivity from time-to-time too, but not bad overall.

TBH, I'm not sure why companies use Slack (I use it for other organizations, so have experience with it too, but not extensive).

It isn't just # of users, though - SlackOps is probably unique to Slack in that list (minus Teams, I guess) - so # of messages per month is a better metric. Not that I'm letting Slack off the hook, it still may be that their codebase and/or dev process is just nasty.

Slack and the others have different contractual guarantees and different regulatory environments. Comparing them is not really fair because the reality is that these other services probably just lose tons of messages and slack/teams can't do that! They have to have better guarantees.

IME, Slack is far more likely to lose my message than iMessage. I believe that's part of the point being made above.

I've never had slack lose a message when it's up

That's kind of the definition of a service being up. :) I've experienced numerous "soft" outages which result in messages not sending and getting lost - and even more double sends, sometimes very distant from where the message was originally sent.

ITT: Anecdotes

Just anecdotal, but as someone who has used Teams continuously for 1.5 years, I can say that it has never been down for me.

That being said, individual instances of the app are notoriously unstable causing random annoyances. But, I am on a very early build of Teams, which is buggy by definition.

Telegram is closer to 500 million now.

https://t.me/durov/142

EFnet was always splitting every few hours. I don't really miss IRC compared to modern chat systems.

I'm the opposite. Back when in my early teens, friends and I would attempt to hijack opposing groups' channels via takeovers during net-splits (and ofcourse having the same done to us). What a time to be alive.

In the early battle.net days competing clans would split and steal channels. It was tons of fun. Taught me lots about bots, proxies, simple scripting, in the process too.

Oh yeah, those were the days. Causing server splits to get your nick back that was stolen in a previous server split...

I do miss them, terribly. Lightweight, fast, brutally simple. Even with splits, it was better, and ever since IRC bouncers exist, like ZNC, they are rock solid.

I spent a few hours setting up a chat client for a reason. Slack takes all this away from me.

My feeling is this is an AWS issue. Our services hosted in AWS are not working either.

Do you have more info like services, regions etc? I see all green checks on the AWS Status page.

Down detector shows quite a lot of issues across a broad spectrum of services, including AWS and Google.

Our prod systems seem to be working, but our lower environments seems to be not working. I don't know enough about where these things come from. I wonder if the real problem is regional. Some connections work and some don't.

> I see all green checks on the AWS Status page.

I'm sure you know this already, but that status page isn't worth the cycles on your CPU, you would be better served asking the toaster if AWS is functioning properly than checking that status page.

Of course, yeah, but at least you can sometimes see a yellow and infer it really means red :/.

yellow requires an issue that every customer is aware of and red requires a thermonuclear strike.

If one's smart toaster depends on AWS one might very well do that.

I have also been having intermittent issues with Twitter also this morning (can't load tweets etc) and was wondering if it was connected.

Naaah, Twitter always fails to load for me. It's more surprising if it loads from the first attempt.

Which AZ?

Availability zones are unique for each account. So my zone A could be your zone C, for example.

Woah, thanks for clarifying--I had no idea!

I never knew this, but I think it makes sense. Is there any documentation that explains why this is the case? I suspect it is to distribute bias to the first option, but I'd love to read about it.

[edit] Nevermind, I just needed the right combination of terms to find it: https://docs.aws.amazon.com/ram/latest/userguide/working-wit...

This is so everyone doesn't launch in one zone, "us-east-1a".

Downdetector indicates a possible correlation between Slack issues and AWS reports, eventhough Slack peaks at 13960 problems and AWS at 111:

- https://downdetector.com/status/slack/

- https://downdetector.com/status/aws-amazon-web-services/

Most people that are affected by AWS outages wouldn't report it as such...

I thought Slack was not on AWS, but Oracle.

You may be thinking of Zoom, who signed a massive Oracle contract.

What issues are you seeing on AWS?

The dashboards are all green. (which doesn't mean that much ... I'm aware...)

To be fair, IRC doesn't do a lot of things Slack does. Where is the logging and audit trails, access control, search, etc.

I'm still dreaming of a world where everyone uses IRC through an interface identical to Slack or Discord or whatever, and features like these are implemented.

You might appreciate matrix.org / element.io if you haven’t seen them yet

I setup my own Matrix homeserver recently with several bridges to all my current chat services:

https://battlepenguin.com/tech/matrix-one-chat-protocol-to-r...

It works fairly well.

You might like irccloud; it's a web client (similar to slack) and bouncer, with support for image uploads, has a decent app, preserving history and I think it supports search too.

Not really a fan of the Slack or Discord user interface myself, but there are modern looking web clients for IRC such as thelounge[0] or kiwiirc[1] that might be what you are after.

[0] https://thelounge.chat/ [1] https://kiwiirc.com/

I agree in principle, but IRC is a poor way to do this. I love IRC for it's simplicity, but that makes it hard to do more advanced features. It's a text-only protocol (other than DCC), so if you want to do something like allow users to click phone numbers to dial them then you have to regex it and hope for the best. Any kind of link is the same way. If you want to show images inline, you'll have to search for links, then either do another regex to see if the link is an image or prefetch the page to see if it's an image. Most servers still implement user authentication as a secondary service (i.e. it isn't part of the IRC server itself) afaik. I think the newer IRC specs include those, but support for it is missing in many servers.

Really a huge part of IRC's difficulty and beauty is in not having a markup language, but most of that beauty is for the eyes of the developer, not the user.

I like the concept of Matrix. That's kind of what they're trying to do by creating an open protocol, but when I looked at implementing a client it was non-trivial. For IRC, you can usually send someone a telnet log of you joining an IRC server and they could implement a client. I don't get the impression that that's true for Matrix.

https://news.ycombinator.com/item?id=20948530 is my attempt to demonstrate that implementing a Matrix client is almost as trivial as telnetting to port 6667 on an IRC server, fwiw :)

Several IRC servers do have support for authentication and access control (and audit trails as well I suppose).

Only centralized history/logging and search would need to be bolted on if needed. In the non-centralized case your IRC client takes care of all of that.

You might be interested in IRCv3. https://ircv3.net/irc/ https://ircv3.net/software/servers https://ircv3.net/software/clients

Lack of logs and history is a feature not a bug.

For business users, there are regulatory requirements. You need to keep information around for some period of time, but not forever. History and searching is useful for spreading tribal knowledge throughout an organization.

Does that actually extend to Slack/slack-like things though?

Since I would see Slack more of a replacement for phone calls or hallway discussions. Neither of which typically has any logs or recordings (and I wouldn't want to work somewhere that did keep such logs).

It does yes. This is why for example message history data export is a paid feature. Its a requirement for certain types of compliance.

In what areas would you find such requirements? And shouldn't the default position be that it is illegal to keep those logs? Especially those involving direct messages between employees.

Sure I understand but I don’t think that aligns with the spirit of the comment I replied to. I read it as FOMO.

At least when irc goes down you can still access your logs

Our company uses Cliq. I wouldn't say that it's as good as Slack, but it's probably 80-90%, and even has a few unique features (integration into Zoho's suite, remote work checkin, integrated bot development environment, etc)

Reply on news.ycombinator.com