Real-time messaging

My guess: it’s persisted mutations that need their own retry- and deduplication logic, as well as user facing error handling. Since it hits the db in the main region anyway latency is similar.

Yeah it's a pretty cool design. Discord's real time system operates on a similar (although much more complicated system) on top of hash rings as well.

The biggest difference is that if you shoved a million active users in a single chatroom back then it would have crashed all of AOL. The same problem can have vastly different solutions at different scales.

Nothing. Not the geographically distributed customers, (unlike America Online), not the fact that this works both on the web and on mobile devices, nor the fact that mobile devices can't open persistent background connections due to power requirements, not the much larger scale, nor the fact that latency characteristics over dial-up modems were much, much higher than they are now, not even the fact that organizations get their business done using this architecture while AIM was mostly (not always) used to just chat with friends and strangers casually. No nothing at all.

But really is the internet actually doing anything new? My grandfather used to send telegrams overseas all the time. Does anyone remember teletype? How is that any different than us chatting with each other here on HN? Phaw, tech bros just keep reinventing the wheel.

Consul, Envoy, websockets. These are architectural components called out in the article. Are you saying that AOL used this same architecture and you don’t see anything “fundamentally unique”?

If you’re using the word architecture even though you mean product functionality, you can simply go to https://slack.com/help/articles/115003205446-Slack-plans-and...

Look at the functionality listed there and see how many things AOL supported which Slack does not and vice versa.

What makes you recommend Cassandra here over the other?

MySQL can scale extremely well if you tune and use it correctly - Twitter was still backed primarily on MySQL at least through Obama’s second election, and I’m pretty sure for some time after that.

Discord uses Elixir. They're fans.

https://elixir-lang.org/blog/2020/10/08/real-time-communicat...

Notably, with regards to the hiring pool:

> None of the chat infrastructure engineers had experience with Elixir before joining the company. They all learned it on the job.

It is definitely an area in which Elixir excels. My personal feeling, however, at this advanced age of late 30s, is that I just can't do dynamically typed languages for large projects anymore. A statically typed Elixir would be a dream I think. There is https://gleam.run/, but I don't know how mature it is.

Slack started out on LAMP stack, so a lot of their choice of technologies stem from there. There's a mention of using Hack in this article, which is a PHP dialect. People frequently reach for things more complicated than MySQL, but the truth is that there are likely very few use cases that won't work with MySQL. Vitess makes MySQL manageable at scale (YouTube uses it too) as long as you invest in getting the tooling & automation ready. Vitess itself is fairly straight forward to understand and operate. It's ultimately just MySQL + a side car service and a query layer that handles reading/writing to shards for you, and some other admin tools for managing shards, migrations, etc. If you need to debug something you can just SSH into a host and connect directly to the mysql instance. It uses standard mysql replication and you can tune it the same way you would tune mysql normally. There's no shortage of resources available when it comes to dealing with mysql.

Seems to me like their Channel Servers pretty much does the work of what Cassandra would do, with Consistent Hashing and their Consistent Hash Ring Managers, although it's not detailed how the Channel Servers persist their data and if like Cassandra, they have replication to N Channel Servers. Looks like a specialized variant of how Cassandra works for their use-case, possibly without all the drawbacks of the underlying data structure Cassandra uses for persistence (SSTables), like needing exact ordering of clustering keys when performing queries or dealing with tombstones.

I could see these Channel Servers using something like SQLite for persistence, which would allow the full suite of SQL features without any of the drawbacks of CQL, of which there are many.

The blog post felt a bit high-level and lacking some technical/architectural details that would've given more insight, but nonetheless it was an interesting read!

> Wonder how scalable the code base really is.

Are you for real? This isn't an hypothetical codebase. This is slack, used by millions of people.

I get it, they're using boring tech, but come on.

> I think it would make an intriguing alternative to voice calls

'talk' was an everyday tool back in the 1980s and early 90s that sent characters as they were typed. I just checked and it's installed on my Mac today. It's person-to-person, not group chat, but there's something magical about seeing every keystroke in near-real-time. You can feel the person at the other end.

That was a standard chat feature back in the day, certainly with AIM and probably ICQ.

Its fine if you don't care enough to edit what you say, but if you do it requires composing the sentence first in your head.

They're preparing you to the real-time messaging experience when you're working remote from Mars.

Noticed this today, too. I turned on a computer I haven’t logged in to in 2 months, and clicking a channel took at least 5 seconds to open/load. And then it worked normally after that.

Agreed, something changed that made Slack usage across multiple devices much worse. Loading a channel tends to sputter as various state and messages load, messages I already read on different devices change state, it's pretty jarring to watch. The "unread messages" badge is often out of sync.

E: if you reach out to support, they can reset stuck state for you. They were very helpful but didn't sound sure it would work.

I’d say anything that users perceive as instantaneous.

> Talk to a software developer and it means something like "as fast as we can, without any purposeful delays or buffers"

No, "real time" in a software context means having a persistent stateful TCP/UDP connection open between a client and a server (or two clients in case of WebRTC and the like) to exchange data, skipping over paradigms like having to establish new HTTP connections or being blocked on database fetches. This is a well-established definition, and has nothing to do with any specific millisecond threshold.

Real time means that there is a requirement on maximum time (end-to-end latency, photon to motion, whatever), and that the worst case behavior of the system meets the requirement.

There's "hard real time", where a system must respond within a specific time (eg. vehicle or rocket control, or PLC systems you mention, etc) and "soft real time" meaning the response should be quick but nothing catastrophic will happen if it doesn't.

Voice, video, and even text chat, falls into this category. You want minimal layency but nobody will die if the screen freezes for a bit.

It's customary to talk about these systems as "real-time communication" and nobody working in the field would confuse it with hard real-time.

More info and examples: https://en.m.wikipedia.org/wiki/Real-time_communication

I came out of college with a (not computer) engineering degree, and went into software. I totally flunked a software interview one time early in my career because they asked me how to implement "a real-time clock UI," and I spent the whole time asking questions trying to understand what they meant by "real-time," rather than just implementing a basic clock UI with 4 numbers, a colon, and a 1-second timer updating the UI, and calling it a day.

500ms is definitely not real time, even dialup can do better than that. my ping is often 5ms

The word real-time has more than one meaning. This is about real-time chat, where the message is expected to be read immediately, as opposed to something like email where the message is expected to sit in a buffer for a while before being read.

You wouldn't ask for the deadlines of a video game speedrunner talking about "real time", either. That means real wall-clock time rather than time displayed by the video game (differences can arise due to e.g. pausing)

Who suggested they're not doing that already?

I'm sure they have whole teams dedicated to that, but even perfect software would still need these sorts of health checks at the scale slack operates. Sometimes you just have random failures in hardware or the OS, or you need to migrate software around the datacenter without bringing down the service. This is a generic way to route around all of that.

Silly engineers, implementing things like failovers and redundancy. Why can't they just ensure that their servers never crash?

> It seems rather counter to what the platform is meant for.

Well that's just silly, if you've ever used used slack. It's corporate chat rooms, with real-time chat being a fundamental feature of it.

> counter to what the platform is meant for.

The platform is meant for communication. Async communication is one (probably smallish, at least in my experience) slice of that.

Sounds fun. Is the repo still up?

Did you actually test it with unlimited users?