Hacker News Reader

Comment by fb03 | Hacker News Reader

What are the differences between Matrix and Jabber?

Jabber implemented a protocol with similar mechanisms several years ago, with a custom xml protocol which could also pass not only text but also rendezvous data for voip/video calls and whatever else custom data you could codify.

Jabber has native protocol implementations for both clients and servers and can handle a reasonable amount of connections with ease. Clients are available throughout all platforms.

I'm asking this question because I believe the reason Jabber "failed" (read: didn't catch as much as other closed alternatives like WhatsApp) wasn't because of technology hurdles, because it was poised to solve the very same problem Matrix works against - which is 'IM fragmentation', but actually a lack of a proper plan to generate interest with its intended population.

I'd say work on your release plan and make your clients 'tasty' and snappy, because technologically speaking, I believe Jabber has already solved all the technological problems Matrix strives for.... and it sadly wasn't enough.

View on HN

One use of Matrix is defragmenting IM, and it's true that both XMPP and Matrix can be used to power chat systems.

However, architecturally they really couldn't be more different:

* Matrix's main data primitive is synchronising conversation history within a room - not message passing. In fact, there is no way to just 'send a user a message' in Matrix: instead, you can only synchronise your copy of a room's state and history with someone else's copy of it.

* Matrix rooms are replicated equally over all participating servers - there is no focal point as there is in a XMPP MUC.

* Matrix provides a single monolithic (beta) spec, which compliant clients have to implement (for a given class of clients). There are no optional features or competing extension proposals for a given feature for a given class of client (e.g. 'desktop messenger'), to try to avoid fragmentation.

* Matrix's baseline transport and encoding is super-simple-stupid HTTP+JSON, but with room for folks to propose superior transports & encodings as we see fit. So far there's been WS+JSON (which nobody seems to use, as it's not that massive an improvement over HTTP+JSON), and this year there's a GSoC project to look at MQTT-and-similar as an alternative.

IOW, Matrix is best suited for replacing IRC or Slack, for communication around organized teams/communities or topical conversations with strangers; while Jabber/XMPP is a better Facebook Messenger, WhatsApp, Hangouts or Gadu-Gadu, best for staying in touch with friends and talking in ad-hoc group chats between them.

Plus, XMPP can be used for great amount of stuff non-IM related, while Matrix seems very focused around its IRC-like use case, where XMPP doesn't really improve much over IRC.

> Plus, XMPP can be used for great amount of stuff non-IM related, while Matrix seems very focused around its IRC-like use case

I don't really agree here. While Riot is focused on a IM use of Matrix, Matrix itself is open enough to be usable in any use case requiring a payload to travel from A to B. You could think of Matrix use in IoT, social networks, forums, etc. Some people even made a working PoC of a blog using Matrix, and I heard some others are also working on building a system to bypass information censorship using Matrix as its only back-end.

I don't know XMPP enough to state which spec is wider than the other, but what I know is that Matrix isn't limited by what Riot or other Matrix-based IM clients can do.

Of course you can (ab)use a lot of protocols in various ways, however, XMPP was designed with this extensibility in mind and that's exactly why its basic RFC is so minimal. Heck, even basic instant messaging over XMPP is defined in a separate RFC (RFC 6120 for Core and RFC 6121 for XMPP-IM).

GP already stressed it out:

> Matrix's main data primitive is synchronising conversation history within a room - not message passing.

So Matrix might be well suited for things like blogs, but XMPP will be certainly better for push notifications, for instance. However, you can easily build Matrix-like primitives on top of XMPP, while the other way around will be less flexible.

But flexibility is the problem with XMPP in the first place. None of the clients and servers have compatible feature sets.

> In fact, there is no way to just 'send a user a message' in Matrix: instead, you can only synchronise your copy of a room's state and history with someone else's copy of it.

I've read in a different thread that sending a message in Matrix can be easily achieved with a simple curl. Is it true? Synchronizing my copy of room state doesn't sound so simple. Maybe it's optional or I misunderstood the "message via curl" post?

Sorry, my point was perhaps a bit mangled.

If you want to send someone a message in Matrix it's indeed a trivial HTTP PUT; something like:

curl -XPOST -d '{"msgtype":"m.text", "body":"hello"}' 'https://example.com/_matrix/client/api/r0/rooms/!CvcvRuDYDzT...

However, what's happening under the hood is that you're not saying "Hi Bob, here's a message from Alice" - instead, you're saying "Hi everyone, I've added a message to the history of this room. Please can everyone sync their copy of the room with mine?". In other words, it's talking about the way the federation (server<->server) protocol is architected.

There literally isn't a way in the federation protocol to say "please send a message from Alice to Bob"; instead the way you do it is to say "create a room and invite Bob to it (if you don't have one already); add a message to the room's history, and once the history has synchronised with Bob's server he'll have a copy of the message too".

So if I understand it correctly the client-to-server API is simple but servers internally build a Git-like structure? Interesting design, I think I'll read the specs for how history is stored.

Precisely. Matrix is a shameless rip-off of Git, but applied to realtime messaging.

Seems we now just have to wait for the next generation that's a rip-off of piul[1] rather than git, and we'll be all set!

On a more serious note, this seems to be a trend; to model chat as a converging set of edits. From an architectural pov it sounds similar to Google wave and lotus notes.

[1] https://jneem.github.io/merging/

I think that main difference is the fact that Jabber has important features (in 2018 year) as an optional extensions. So you have to carefully search for a proper client and server which support those features (and your companion's client AND server should support those features as well). Matrix should have those features in core and any server or client is supposed to support them.

That said, I think that main problem with Jabber is lack of a single entity pushing it. So it'll naturally lose fight against corporation with billion-dollar marketing budgets. With Matrix it might be better.

Sometimes technologies losing or winning for very ackward reasons. I don't believe that Jabber will win, but with Matrix it's possible. We should keep trying.

The thing that concerns me about this is that while Matrix might be a more modern protocol today, it won't be long before the world moves and it requires an upgrade, and extendability has been the killer feature that has helped XMPP survive a long time albeit not to prosper. Unless the problem of convergence is solved this will haunt evolving distributed networks no matter how strongly they start.

Matrix is very extensible too; it's just that we aim to merge extensions as rapidly as possible into the core monolithic spec, rather than having them get fragmented into optional extensions.

I wonder what happens when there's 10 implementations of the matrix spec controlled by different groups. Being able to indicate where the implementation gap is seems like a strength rather than a weakness.

Hm, even when there are 10 different server implementations (or the ~50 client implementations there are today), we’d hope they would align to the same spec - just as with HTTP, SMTP, TCP/IP etc. There’s room for experimentation without introducing fragmentation, and any useful experiments would hopefully rapidly get merged into the spec. And if they don’t then yes, you risk a fork on that particular feature (the equivalent of only Exchange supporting email recalls, or browser vendor extensions), but it’s not the end of the world.

I'm not sure if there's really a place around for an IM network that's moving forward about as fast as HTTP, SMTP and TCP/IP are :)

it really depends, if the features are difficult to implement it may not be very quick at all, we see this with XMPP often where features take years to implement or never appear. If a homeserver targets a particular use-case then they may actively choose to ignore certain features that are not relevent. Really when people talk about the flaws of XMPP they often have in mind the problems of private e2e encrypted chat on mobile which is a hard problem to crack, but there's many more private servers inside organisations that don't even federate and work fine for how they are used. This flexibility perhaps makes it better to view XMPP as a toolkit for building networks out of, similar to how IRC networks which have a converged set of servers but outside that network all bets are off. The idea of viewing XMPP as a single network inevitably causes it to look fragmented but there are pockets within the network that interoperate just fine they just don't identify themselves as a group.

The thing is... with XMPP, it's things that are really, utterly critical like the concept of messages having IDs that are left in the "extensions".

You can't build a sane protocol when something like "IDs" are a complete fend-for-yourself wilderness.

I've written about this previously: https://news.ycombinator.com/item?id=12908619

Long story short, if you ever try to implement something in XMPP/XEPs, you'll learn the hard way why it never seems to gain any traction. Very, very quickly. There are real technical reasons.

Extensibility is good. I believe the Matrix developers when they say it's extensible. (I've been a user for several years now. Features regularly get added in backwards-compatible and smooth ways.) Extensibility in the way XMPP tried to go about it -- for patching over core inadequacies of the basic protocol -- is not the same beast.

What would really help with XMPP is some idea of profiles - like a Jabber2018 profile that requires all of the XEPs used by Conversations[0]. That way, client and server implementers would know exactly what they need to be compliant with other implementations using the same profile.

[0]: https://conversations.im/

That exists. Here are the 2018 profiles (they are split into basic/advanced): https://xmpp.org/extensions/xep-0387.html

They are only part of the solution however because:

- Open-source projects are developed by volunteers, and implement what their users (actually typically their developers) want implemented. Having a document that says "you should implement these things" is good for guidance, but it doesn't magically get them implemented.

- Commercial projects will implement the things that they can get revenue from. Having a document that says "you should implement these things" doesn't magically make those things earn the company money.

E.g. in the open-source category, Pidgin is the classic example of a once-popular client that has fallen behind (with feature requests for some "new" protocol features open on their bug tracker for many many years). The document doesn't fix this part.

But yes, I do agree it's an essential part of solving the problem.

It's still so common to see clients tout a list of supported XEPs like they expect users to know what any of it means. It's great there is a list of importantant XEPs but this must be communicated to users better so that it's easier to find suitable clients. It might even be good to refer to compliant clients as something other than Jabber/XMPP then you can easily search for an "XTalk" (for example) client and know this means the full list of XEPs are supported. Part of this might include some new XEPs to indicate to users when they are communicating with non-XTalk users so the experience might differ. A certification programme even might help motivate devs to meet compliance quickly.

One can dream anyway.

I doubt any platform uses vanilla XMPP, that wouldn't be sane, but like using XML, XMPP is a relic of its era. Whose to say what super-critical "how can you not have this as standard!?!" feature won't come up in the next 5 years that Matrix devs hadn't even known of in 2018. That doesn't mean we should flip over the table and start again every now and again and hope the users will follow, this is why extensibility is an important part of the recipe and make ways to ensure the standards of the day are always implemented before being considered ready for prime time.

IDs, though.

Like, in the abstract, I agree with you completely.

But IDs.

I don't know, IRC in its current form doesn't have any message-ids and yet seems to be working fine.

It also does not and never did support consistent history or multi-device.

And it never will, because having some way to identify a message is a prerequisite for even phrasing the question "does device_foo have message_x synced yet?"

Bouncers don't count. They reintroduce centralization as a way to solve the problem. It's a solution, of a sort, but it's a fairly poor one if your original aim was decentralization.

And "working fine" is also frankly a bit of a stretch, isn't it? I can't count how many times I've seen a "netsplit" in freenode or the other IRC networks. That means messages aren't delivered. It's so common we have a word for it: "netsplit".

I shouldn't dunk on IRC too much, because at least it's honest about itself for being simple, and there's a virtue in that. But if we were to construct a statement like "IRC manages reliable message delivery without any concept of message IDs", then that would be false (or technically, vacuous), because it doesn't manage reliable message delivery.

I don't think you get the idea of extensions in XMPP. If something is published as a XEP, it doesn't mean "it's a completely optional thing you can ignore". It does mean "it's a thing that might not be useful in some cases, so it doesn't belong to the core that everyone and their pets must implement".

If you build an IM client for XMPP (which is just one of the use cases of XMPP), you absolutely have to implement at least a few of those extensions in order to have basic functionality. Heck, there's even a separate RFC for XMPP-IM on top of the XMPP-Core RFC. That's expected, by design and very fine.

Of course, what's considered "basic" changes over time and it's easy to find outdated clients out there. That's why Compliance Profiles exist, so you can look around for clients that implement, for instance, 2018 desktop IM profile. It also imposes some burden on new implementations to consider backwards compatibility with those clients. No solution is perfect, but while I think Matrix' way allows them to move forward faster now, in the long term it's probably XMPP that will age more gracefully.

> I don't think you get the idea of extensions in XMPP

I don't think you read the other comments I linked to. My fault for not copy-pasting the entire thing every time the "y u not be exactly XMPP" train comes on HN again...

---

> - unique message IDs? Absent. XEPs kind of provide; but I can't tell you which of the three or for relevant ones are the most relevant (AMP IDs from XEP 0079? Stream Management from XEP 0198? Acking from XEP 0184? Something from Carbons or MAM in 0313 or 0280? You know, if you wanted some light reading...).

> - multi device? Oh. My. God. It's bananas. The spec behavior is that whenever a client sends a message, the server is supposed to consider that one the most alive, and then route all future messages exclusively to that one. So you send a message on your phone? Yeah, your desktop is just going to silently stop receiving messages.

> - there's a concept of "message carbons" to deal with this. This involves re-sending all your messages back to the server after you receive them, with special instructions to send them back again to your other clients. The amount of redundantly redundant XML involved is eyewatering.

> Combine that multidevice behavior (messages can get randomly routed anywhere at any time) with the wild-west nature of message delivery acks, and you can see how ridiculously difficult this makes the basic idea of "all clients should see the same picture".

> Overall, the XEP process, conceptually, is a great example of open extensibility. The trouble is, so much of this stuff is core to sane message delivery semantics that it really, practically speaking, causes huge problems when it's all considered "extensions". Stuff like message IDs fundamentally shouldn't be an extension because it's just too critical that all minimum-viable clients agree. You just can't build higher level stuff without that. XEPs are great. A community process for extensions should exist. It just needs to exist for extensions, not subsume the total set of realistic minimum viable features.

(All of this is a shameless repost from https://news.ycombinator.com/item?id=9772968 ; please forgive my humble attempt to save the dear reader from the need to traverse any links.)

---

I get the idea of extensions in XMPP. I have tried to implement them. It is based on this first-hand experience that I am now arguing that the XEPs are a swamp. It is time to move on.

Yes, even with Compliance Profiles. It's too late. The horses have left the barn, developed tools, formed their own complex civilizations, invented light speed travel, and in fact left the galaxy. That's how far out of the barn they are.

Message ID and multi-device messaging is in fact fundamentally broken in such a unique, fascinating way in XMPP that all other clients will cease to operate correctly in the presence of a single bad client.

That's right. Any bad client can cause other clients which are following the spec to the letter to begin to drop messages, forward them incorrectly, and generally lose history and form desynchronized views of the world. I'm not kidding. If you haven't read the XEPs, and don't believe me, go read the XEPs. The only way to implement a reasonably correct multi-device behavior requires ignoring significant parts of XMPP because it is wrong.

Start. Over. XMPP is not your savior.

I briefly worked on an XMPP client for an internal chat app. I ran into tons of issues (multi device problems being the main one I recall. This was over 5 years ago now.) It would've been simpler to build my own client and server with a custom protocol.

Someone else really wanted to build on XMPP though. XMPP has fundamental issues. No thanks.

Guess what: I was implementing (at least some of) them too, successfully.

> but I can't tell you which of the three or for relevant ones are the most relevant

I can. Check out XEP-0359. MAM and some other XEPs depend on it.

Also, unique message IDs are not a feature. They are means to provide some feature. You will deal with different kinds of IDs when trying to ensure network reliability (0198) or when trying to implement read markers and delivery receipts (0333, 0184). Without telling us what feature would you like to implement (threading? attachments? something like 0367?) just spewing all different kinds of IDs from various XEPs starts to look like a FUD to confuse uninformed reader.

> multi device? Oh. My. God. It's bananas.

I can agree with opinion that the protocol is really "bananas", but it works. Basically 0280 solves this issue, especially when combined with 0198 and 0313 - the only situation where it breaks is if you connect with a legacy client that doesn't support carbons and set it a presence with higher priority than any other client. You'll still get the content of the messages on modern clients thanks to MAM though. So, basically, don't use legacy clients on your account, cause they will degrade your overall experience. Doesn't seem very surprising, especially given that you can't connect at all to most IM networks out there using outdated clients. My N900 with stock OS won't connect to Skype or Facebook at all, even though it tells me it can. It will to XMPP - with limited functionality, but it's better than nothing.

XMPP accumulated its great share of backwards compatibility weirdness, but it doesn't seem unmanageable and I believe that the way XEPs are handled made it better, not worse. I suspect that Matrix might be in worse shape when it reaches current age of XMPP. I would like to be wrong though, I wish them best.

So basically you reiterate what I said. And specifically:

> So, basically, don't use legacy clients on your account, cause they will degrade your overall experience.

Yes. As I said.

But let's not put a bouquet of roses on it: it's even worse than that, because you don't just have a degraded experience when using a legacy client. Using a "legacy client" -- of which there are many because of the XEP swamp -- will make all other clients behave wrong. And it is impossible[1] to tell which client (or server) to blame. Thus resulting in even the most modern and chose-the-right-XEPs clients regularly getting bug reports about lost messages... even if some other client is at fault.

If that's not a trainwreck I don't know what is.

---

[1] inb4: nothing is "impossible". But "practically speaking impossible for an end user who has other things to get done today and a limited time budget for debugging a problem which is a $n$-dimensional matrix of their clients and servers in use", yes.

https://matrix.org/docs/guides/faq.html#what-is-the-differen...

thank you

on xmpp clients: "Baseline feature set is so minimal that fragmentation of features between clients and servers is common, especially as interoperability profiles for features have fallen behind (as of July 2015)"

they are spot on in this one.

Indeed, there are a large number of people still using Jabber/XMPP today with "popular" clients like Pidgin or Adium that haven't stayed up to date with modern advancements in messaging.

It leads many people to believe that XMPP is incapable of many of the features that are considered normal in modern messaging applications.

A hard break in the protocol would solve it (i.e. so old clients would no longer be able to communicate with new clients), but it would also frustrate and fragment the significant existing userbase. Given that open decentralized standards-based networks are something to be treasured if we're to stem the tide of proprietary walled-garden messengers, that option would potentially be shooting the open movement in the foot.

EDIT: typo

The really great thing about XMPP is that basic messaging always works and is super simple to implement. Anything that changed that would be very bad.

We've seen hard breaks before: ssh 1 vs 2 - and we're seing it now with http2. The latter is really crazy, but seem to be actually working.

...and outdated, as it's 2018 now and XEP-0387 exists.

Actually, most of this FAQ is severely outdated, XMPP already catched-up in most of these points. Some clients are lagging, sure, but that's just a matter of time for Matrix to suffer from the same issue.

Is there an up to date wiki/guide of a subset of features and implementations that go well together for modern xmpp?

As in these mobile clients for ios and Android, these for desktop and these for cli. With these servers - Together give you in-band registration, ssl-only client/server and server/server, maybe audio/video, file-sharing, server-side history - e2e encryption (I'd settle for otr) - and sane defaults for each client and server to make it difficult to accidentally disable encryption or certificate validation?

Honest question - last time I looked I couldn't find a simple&secure, recommend setup for xmpp.

A thing I often see is people pointing to specifications and saying "look it's all fixed".

But you can't chat using a piece of digital paper. What matters not is what the protocol can theoretically do, but the experience you can have if you open a client right now.

That point that needed "fixing" was specifically about lack of specification... plus there are multiple clients and servers already implementing this spec.

>Bridging to other protocols and defragmenting existing communication apps and networks is very much a second class citizen feature

Not sure what this means. XMPP gateways have been around forever, tend to actually work and are completely standardized with a specification and everything. The reason you don't hear more about them is because there is nothing much to gateway to any more. All the other IM systems have managed to close off their gardens to such an extent that gateways are pointless...

I guess that raises an obvious question. Is there an XMPP gateway to Matrix? I know there are some projects that are about gatewaying from Matrix to XMPP. One potential complication is that Matrix isn't an IM network as such, it is more of a IRC like distributed conference. There is a XMPP to IRC gateway, so it must be possible at some level...

It does exist, via libpurple. I think it was one of the first bridges. It is not currently maintained, AFAICT due to lack of general interest. Telegram, IRC, discord, slack, gitter etc. bridges are actively maintained, for comparison.

I was not aware that libpurple supported Matrix but there is a plugin:

* https://github.com/matrix-org/purple-matrix

... which in theory would work with:

* https://spectrum.im/

... running in XMPP gateway/transport mode...

Reply on news.ycombinator.com