Enter the Matrix – A technical overview and guide to all things Matrix
brendan.abolivier.bzh
brendan.abolivier.bzh
Jabber implemented a protocol with similar mechanisms several years ago, with a custom xml protocol which could also pass not only text but also rendezvous data for voip/video calls and whatever else custom data you could codify.
Jabber has native protocol implementations for both clients and servers and can handle a reasonable amount of connections with ease. Clients are available throughout all platforms.
I'm asking this question because I believe the reason Jabber "failed" (read: didn't catch as much as other closed alternatives like WhatsApp) wasn't because of technology hurdles, because it was poised to solve the very same problem Matrix works against - which is 'IM fragmentation', but actually a lack of a proper plan to generate interest with its intended population.
I'd say work on your release plan and make your clients 'tasty' and snappy, because technologically speaking, I believe Jabber has already solved all the technological problems Matrix strives for.... and it sadly wasn't enough.
That said, I think that main problem with Jabber is lack of a single entity pushing it. So it'll naturally lose fight against corporation with billion-dollar marketing budgets. With Matrix it might be better.
Sometimes technologies losing or winning for very ackward reasons. I don't believe that Jabber will win, but with Matrix it's possible. We should keep trying.
You can't build a sane protocol when something like "IDs" are a complete fend-for-yourself wilderness.
I've written about this previously: https://news.ycombinator.com/item?id=12908619
Long story short, if you ever try to implement something in XMPP/XEPs, you'll learn the hard way why it never seems to gain any traction. Very, very quickly. There are real technical reasons.
Extensibility is good. I believe the Matrix developers when they say it's extensible. (I've been a user for several years now. Features regularly get added in backwards-compatible and smooth ways.) Extensibility in the way XMPP tried to go about it -- for patching over core inadequacies of the basic protocol -- is not the same beast.
They are only part of the solution however because:
- Open-source projects are developed by volunteers, and implement what their users (actually typically their developers) want implemented. Having a document that says "you should implement these things" is good for guidance, but it doesn't magically get them implemented.
- Commercial projects will implement the things that they can get revenue from. Having a document that says "you should implement these things" doesn't magically make those things earn the company money.
E.g. in the open-source category, Pidgin is the classic example of a once-popular client that has fallen behind (with feature requests for some "new" protocol features open on their bug tracker for many many years). The document doesn't fix this part.
But yes, I do agree it's an essential part of solving the problem.
One can dream anyway.
Like, in the abstract, I agree with you completely.
But IDs.
And it never will, because having some way to identify a message is a prerequisite for even phrasing the question "does device_foo have message_x synced yet?"
Bouncers don't count. They reintroduce centralization as a way to solve the problem. It's a solution, of a sort, but it's a fairly poor one if your original aim was decentralization.
And "working fine" is also frankly a bit of a stretch, isn't it? I can't count how many times I've seen a "netsplit" in freenode or the other IRC networks. That means messages aren't delivered. It's so common we have a word for it: "netsplit".
I shouldn't dunk on IRC too much, because at least it's honest about itself for being simple, and there's a virtue in that. But if we were to construct a statement like "IRC manages reliable message delivery without any concept of message IDs", then that would be false (or technically, vacuous), because it doesn't manage reliable message delivery.
If you build an IM client for XMPP (which is just one of the use cases of XMPP), you absolutely have to implement at least a few of those extensions in order to have basic functionality. Heck, there's even a separate RFC for XMPP-IM on top of the XMPP-Core RFC. That's expected, by design and very fine.
Of course, what's considered "basic" changes over time and it's easy to find outdated clients out there. That's why Compliance Profiles exist, so you can look around for clients that implement, for instance, 2018 desktop IM profile. It also imposes some burden on new implementations to consider backwards compatibility with those clients. No solution is perfect, but while I think Matrix' way allows them to move forward faster now, in the long term it's probably XMPP that will age more gracefully.
I don't think you read the other comments I linked to. My fault for not copy-pasting the entire thing every time the "y u not be exactly XMPP" train comes on HN again...
---
> - unique message IDs? Absent. XEPs kind of provide; but I can't tell you which of the three or for relevant ones are the most relevant (AMP IDs from XEP 0079? Stream Management from XEP 0198? Acking from XEP 0184? Something from Carbons or MAM in 0313 or 0280? You know, if you wanted some light reading...).
> - multi device? Oh. My. God. It's bananas. The spec behavior is that whenever a client sends a message, the server is supposed to consider that one the most alive, and then route all future messages exclusively to that one. So you send a message on your phone? Yeah, your desktop is just going to silently stop receiving messages.
> - there's a concept of "message carbons" to deal with this. This involves re-sending all your messages back to the server after you receive them, with special instructions to send them back again to your other clients. The amount of redundantly redundant XML involved is eyewatering.
> Combine that multidevice behavior (messages can get randomly routed anywhere at any time) with the wild-west nature of message delivery acks, and you can see how ridiculously difficult this makes the basic idea of "all clients should see the same picture".
> Overall, the XEP process, conceptually, is a great example of open extensibility. The trouble is, so much of this stuff is core to sane message delivery semantics that it really, practically speaking, causes huge problems when it's all considered "extensions". Stuff like message IDs fundamentally shouldn't be an extension because it's just too critical that all minimum-viable clients agree. You just can't build higher level stuff without that. XEPs are great. A community process for extensions should exist. It just needs to exist for extensions, not subsume the total set of realistic minimum viable features.
(All of this is a shameless repost from https://news.ycombinator.com/item?id=9772968 ; please forgive my humble attempt to save the dear reader from the need to traverse any links.)
---
I get the idea of extensions in XMPP. I have tried to implement them. It is based on this first-hand experience that I am now arguing that the XEPs are a swamp. It is time to move on.
Yes, even with Compliance Profiles. It's too late. The horses have left the barn, developed tools, formed their own complex civilizations, invented light speed travel, and in fact left the galaxy. That's how far out of the barn they are.
Message ID and multi-device messaging is in fact fundamentally broken in such a unique, fascinating way in XMPP that all other clients will cease to operate correctly in the presence of a single bad client.
That's right. Any bad client can cause other clients which are following the spec to the letter to begin to drop messages, forward them incorrectly, and generally lose history and form desynchronized views of the world. I'm not kidding. If you haven't read the XEPs, and don't believe me, go read the XEPs. The only way to implement a reasonably correct multi-device behavior requires ignoring significant parts of XMPP because it is wrong.
Start. Over. XMPP is not your savior.
Someone else really wanted to build on XMPP though. XMPP has fundamental issues. No thanks.
> but I can't tell you which of the three or for relevant ones are the most relevant
I can. Check out XEP-0359. MAM and some other XEPs depend on it.
Also, unique message IDs are not a feature. They are means to provide some feature. You will deal with different kinds of IDs when trying to ensure network reliability (0198) or when trying to implement read markers and delivery receipts (0333, 0184). Without telling us what feature would you like to implement (threading? attachments? something like 0367?) just spewing all different kinds of IDs from various XEPs starts to look like a FUD to confuse uninformed reader.
> multi device? Oh. My. God. It's bananas.
I can agree with opinion that the protocol is really "bananas", but it works. Basically 0280 solves this issue, especially when combined with 0198 and 0313 - the only situation where it breaks is if you connect with a legacy client that doesn't support carbons and set it a presence with higher priority than any other client. You'll still get the content of the messages on modern clients thanks to MAM though. So, basically, don't use legacy clients on your account, cause they will degrade your overall experience. Doesn't seem very surprising, especially given that you can't connect at all to most IM networks out there using outdated clients. My N900 with stock OS won't connect to Skype or Facebook at all, even though it tells me it can. It will to XMPP - with limited functionality, but it's better than nothing.
XMPP accumulated its great share of backwards compatibility weirdness, but it doesn't seem unmanageable and I believe that the way XEPs are handled made it better, not worse. I suspect that Matrix might be in worse shape when it reaches current age of XMPP. I would like to be wrong though, I wish them best.
> So, basically, don't use legacy clients on your account, cause they will degrade your overall experience.
Yes. As I said.
But let's not put a bouquet of roses on it: it's even worse than that, because you don't just have a degraded experience when using a legacy client. Using a "legacy client" -- of which there are many because of the XEP swamp -- will make all other clients behave wrong. And it is impossible[1] to tell which client (or server) to blame. Thus resulting in even the most modern and chose-the-right-XEPs clients regularly getting bug reports about lost messages... even if some other client is at fault.
If that's not a trainwreck I don't know what is.
---
[1] inb4: nothing is "impossible". But "practically speaking impossible for an end user who has other things to get done today and a limited time budget for debugging a problem which is a $n$-dimensional matrix of their clients and servers in use", yes.
on xmpp clients: "Baseline feature set is so minimal that fragmentation of features between clients and servers is common, especially as interoperability profiles for features have fallen behind (as of July 2015)"
they are spot on in this one.
It leads many people to believe that XMPP is incapable of many of the features that are considered normal in modern messaging applications.
A hard break in the protocol would solve it (i.e. so old clients would no longer be able to communicate with new clients), but it would also frustrate and fragment the significant existing userbase. Given that open decentralized standards-based networks are something to be treasured if we're to stem the tide of proprietary walled-garden messengers, that option would potentially be shooting the open movement in the foot.
EDIT: typo
Actually, most of this FAQ is severely outdated, XMPP already catched-up in most of these points. Some clients are lagging, sure, but that's just a matter of time for Matrix to suffer from the same issue.
As in these mobile clients for ios and Android, these for desktop and these for cli. With these servers - Together give you in-band registration, ssl-only client/server and server/server, maybe audio/video, file-sharing, server-side history - e2e encryption (I'd settle for otr) - and sane defaults for each client and server to make it difficult to accidentally disable encryption or certificate validation?
Honest question - last time I looked I couldn't find a simple&secure, recommend setup for xmpp.
But you can't chat using a piece of digital paper. What matters not is what the protocol can theoretically do, but the experience you can have if you open a client right now.
Not sure what this means. XMPP gateways have been around forever, tend to actually work and are completely standardized with a specification and everything. The reason you don't hear more about them is because there is nothing much to gateway to any more. All the other IM systems have managed to close off their gardens to such an extent that gateways are pointless...
I guess that raises an obvious question. Is there an XMPP gateway to Matrix? I know there are some projects that are about gatewaying from Matrix to XMPP. One potential complication is that Matrix isn't an IM network as such, it is more of a IRC like distributed conference. There is a XMPP to IRC gateway, so it must be possible at some level...
* https://github.com/matrix-org/purple-matrix
... which in theory would work with:
... running in XMPP gateway/transport mode...
However, architecturally they really couldn't be more different:
* Matrix's main data primitive is synchronising conversation history within a room - not message passing. In fact, there is no way to just 'send a user a message' in Matrix: instead, you can only synchronise your copy of a room's state and history with someone else's copy of it.
* Matrix rooms are replicated equally over all participating servers - there is no focal point as there is in a XMPP MUC.
* Matrix provides a single monolithic (beta) spec, which compliant clients have to implement (for a given class of clients). There are no optional features or competing extension proposals for a given feature for a given class of client (e.g. 'desktop messenger'), to try to avoid fragmentation.
* Matrix's baseline transport and encoding is super-simple-stupid HTTP+JSON, but with room for folks to propose superior transports & encodings as we see fit. So far there's been WS+JSON (which nobody seems to use, as it's not that massive an improvement over HTTP+JSON), and this year there's a GSoC project to look at MQTT-and-similar as an alternative.
Plus, XMPP can be used for great amount of stuff non-IM related, while Matrix seems very focused around its IRC-like use case, where XMPP doesn't really improve much over IRC.
I don't really agree here. While Riot is focused on a IM use of Matrix, Matrix itself is open enough to be usable in any use case requiring a payload to travel from A to B. You could think of Matrix use in IoT, social networks, forums, etc. Some people even made a working PoC of a blog using Matrix, and I heard some others are also working on building a system to bypass information censorship using Matrix as its only back-end.
I don't know XMPP enough to state which spec is wider than the other, but what I know is that Matrix isn't limited by what Riot or other Matrix-based IM clients can do.
GP already stressed it out:
> Matrix's main data primitive is synchronising conversation history within a room - not message passing.
So Matrix might be well suited for things like blogs, but XMPP will be certainly better for push notifications, for instance. However, you can easily build Matrix-like primitives on top of XMPP, while the other way around will be less flexible.
I've read in a different thread that sending a message in Matrix can be easily achieved with a simple curl. Is it true? Synchronizing my copy of room state doesn't sound so simple. Maybe it's optional or I misunderstood the "message via curl" post?
If you want to send someone a message in Matrix it's indeed a trivial HTTP PUT; something like:
curl -XPOST -d '{"msgtype":"m.text", "body":"hello"}' 'https://example.com/_matrix/client/api/r0/rooms/!CvcvRuDYDzT...
However, what's happening under the hood is that you're not saying "Hi Bob, here's a message from Alice" - instead, you're saying "Hi everyone, I've added a message to the history of this room. Please can everyone sync their copy of the room with mine?". In other words, it's talking about the way the federation (server<->server) protocol is architected.
There literally isn't a way in the federation protocol to say "please send a message from Alice to Bob"; instead the way you do it is to say "create a room and invite Bob to it (if you don't have one already); add a message to the room's history, and once the history has synchronised with Bob's server he'll have a copy of the message too".
On a more serious note, this seems to be a trend; to model chat as a converging set of edits. From an architectural pov it sounds similar to Google wave and lotus notes.
In a world dominated by Slack, which removes privacy/history control from users and place walls, Matrix is the promise to have a better open scenario than IRC was.
I am crossing my fingers. I hope with all my heart to this technology to flourish.
- work on stability: the status quo is unreliable. Until this is solved it's hard to recommend Matrix to anybody who is looking to use it for serious work
- disable signup on main server: this is a decentralized netword and the main server is overloaded already. It is overloaded so badly, that devs decided to turn off presence for as long as I can remember first seeing matrix. So even IRC is more usable in that regard because I know when I see a user they are online or they even have presence and an away message
There's more but those are two issues I think should be dealt with urgently.
I've recently been wondering why the Matrix team wouldn't want to lose some of the control they have over the Matrix universe..
They could solve their scalability problem by bringing up other homeservers (why not a paid riot.im server even?) or promoting other open servers hosted by the community.
Seems like Mastodon folks are favoring horizontal scalability by making people use various servers, but the Matrix team wants to keep mostly everyone on matrix.org.
(I hope this doesn't sound too negative.. I know it's not like Matrix folks want to centralize things.. It's just, I wonder why they don't promote other homeservers more)
Once we have account migration sorted out (hopefully coming sooner than expected thanks to work being done for GDPR), then the situation should be much better as folks can flee off the overloaded server onto one of their choice... but first-time newbies don't need to make the complicated and confusing call on picking an alt server.
(Users who have already signed up on the matrix.org HS are stuck there until account migration is a thing, of course.)
That's a number of 'ifs' there, though.
Matrix.org is still running, so no complaints there. You'll figure things out soon enough.
For context, a typical synapse actually only uses around 300MB of RAM. It only spikes up to 1-2GB when trying to resolve state on big rooms like Matrix HQ, and then python doesn’t relinquish the RAM.
We do cache responses in JSON to avoid serialisation overheads.
Without going and reading the doc(yet); does this relate to:?
https://jneem.github.io/merging/
It would seem that a simpler, deterministic merge algo would be possible to parallelize - but I'm not sure if it's easy to match matrix idea of merges with what's discussed in that post/paper?
It's not directly related to the Categorical Theory of Patches paper - the merge resolution here is much simpler than reasoning about VCS patches, although the approach of taking a formal mathematical approach is similar :)
I also know at least a halfdozen people who immediately said "yes" to the devops burden of maintaining their own homeserver (and reportedly, it's actually really easy).
Different strokes for different folks. If self-hosting it was a pre-req for using it, I'd still be saying to myself "yeah, I'll do that riiiight after I get my closet k8s cluster just the way I want it", and, well, I know myself better than that, so I use the public one, and I'm happy.
* Fractal (https://matrix.org/docs/projects/client/fractal.html) from the GNOME community (who just had a big hackathon in Strasbourg last week: https://wiki.gnome.org/action/login/Hackfests/Fractal2018). They're looking to add E2E and VoIP in the relatively near future.
* Nheko (https://matrix.org/docs/projects/client/nheko.html) - a very Telegram-looking Qt client, which has been in very steady progress for ages. They're actively working on E2E right now.
* Quaternion (https://matrix.org/docs/projects/client/quaternion.html) - a more IRC-like looking Qt/QML client, also in very active dev.
* Gomuks (https://matrix.org/docs/projects/client/gomuks.html) - this one's new; a really nice dedicated text-user-interface client written in Go - again, in very active dev.
Meanwhile we're also doing everything we can to improve Riot, but for desktop usage a native client will always beat an electron one :)
PS not requiring anything, just my 2 cents. I, personally, like Riot.
That said, Windows is bad at UX consistency. Native used to mean WinAPI with built-in window classes handling input, buttons, etc (and wrappers like MFC or Windows Forms). Then it was WPF with its own DirectX-based stack. Now it's UWP with even different code. I guess, Qt is not the worst solution. macOS is much more coherent ecosystem when it comes to native.
Then again if you try to run a gtk3 app like Fractal mentioned above on Mac OS, you will see that it looks exactly like a Gnome application.
To develop a new communication platform today without E2E as a core functionality is odd.
> As of May 2017 Riot’s end-to-end encryption is technically in beta, but this is due to some residual stability bugs and missing usability features. Once these are resolved we plan to get the full implementation security assessed and out of beta. End-to-end encryption will then be turned on by default for private conversations.
* Cross-signing devices when you sign up to remove verification pain
* Ability to access history from before the point you join the room (if desired)
* Better UX for verification (comparing mnemonic phrases rather than public key fingerprints)
* Ability to optionally back up E2E keys, encrypted, on the server so you don't lose your history if you lose all your devices
* Ability to search E2E rooms (using a clientside search engine)
* ...and fixing any last key management bugs, although right now we believe we've fixed almost all of them.
I wish I had a single app, be it on desktop and mobile, to use, with the combined feature set of all of those (filtered down to whatever my communication partner supports). I know: how dare I dream of such a magnificent beast?
What are the chances Matrix is turning into this?
Of course it's not there yet, though it's quite likely to get there in the future.
" Tox.chat looks to be a very cool clone of Skype - a fully decentralised peer-to-peer network. Matrix is deliberately not a ‘pure’ peer-to-peer system; instead each user has a well-defined homeserver which stores his data and that he can depend upon. "
What's the rationale behind "impure" decentralizing? Why not go fully decentralized like Tox does?
That said we will hopefully move to a more hybrid model in time.
Assuming for a small company with a dozen employees, all of whom sometimes work remotely.
IRC is reliable, can be forced into encrypted client-server communications, and has the fewest client-side features. The strength is the number of available clients and the primacy of channels (rooms). File-sharing is available but not a focus. There are lots of bot libraries.
XMPP is reliable, the server can mandate client-server encryption, and has some client-side features including simultaneous appearances (being one person logged in through several devices) and better one-to-one chat than IRC. File-sharing is an optional add-on. There is no good iOS client that I know of; mobile clients tend to be bad at requesting history.
Mattermost, Zulip and Rocketchat I have the least experience with; they all seem to be very popular with small groups of users but are complex to install and have very few clients.
Matrix is promising. It does one-to-one and chat rooms smoothly, server config is pretty easy, and you can have the server mandate client-server encryption. There are lots of clients that all seem to be in mid-alpha development. If it survives a year or two, I think it will be widely adopted.
At least for Mattermost I have to disagree with the "complex to install". All you need is a sql database (MySQL/MariaDB or Postgres), for production you then also should put a reverse proxy in front.
With "very few clients" you are referring to client software, correct?
ChatSecure works on iOS and has history fetching features (MAM), for Android there is Conversations.im, IMHO the best XMPP client to date.
With an IRC- or Slack-like communication model, waking up to 100+ new messages every morning is a big drain on time and energy. Also, once a channel starts getting 100+ messages a day, it stops being useful for real communication. If someone starts a conversation at 10am, and you come by after lunch, it's hard to respond in context.
You can see how Zulip solves this problem on the Zulip community server, https://chat.zulip.org (send test messages to "#test here", follow the code of conduct, etc., it's a live server). Feel free to PM me there if you set things up and have any questions/feedback.
To address a previous post: Zulip, Mattermost, and Rocket.chat all take about 5 minutes to install, so I wouldn't worry too much about that. Zulip and Rocket.chat also come with SaaS versions if you don't want to run your own infrastructure.
It may be that zulip has "scaled down" a bit now - when I looked, it seemed you got the complexity of serving 10k users even if you only had 100s.
(Not meant as a dig at zulip, I think it's awesome that the project was made open source!)
haha, that's a good way to put it. We've put in a lot of work over the last few months to make Zulip easier to deploy for a team of 20.
> Last I looked, only rocket.chat had a sane, minimal docker-compose setup
Fair enough! It's on our roadmap, so hopefully we'll have it by the next time you're looking for a chat :).
> A device is bound to an access token and E2E encryption keys (which I’m not covering in this post).
This is far from 'all things Matrix' then.
This is HN, after all.