Hacker News Reader

Comment by dnel | Hacker News Reader

The thing that concerns me about this is that while Matrix might be a more modern protocol today, it won't be long before the world moves and it requires an upgrade, and extendability has been the killer feature that has helped XMPP survive a long time albeit not to prosper. Unless the problem of convergence is solved this will haunt evolving distributed networks no matter how strongly they start.

View on HN

The thing is... with XMPP, it's things that are really, utterly critical like the concept of messages having IDs that are left in the "extensions".

You can't build a sane protocol when something like "IDs" are a complete fend-for-yourself wilderness.

I've written about this previously: https://news.ycombinator.com/item?id=12908619

Long story short, if you ever try to implement something in XMPP/XEPs, you'll learn the hard way why it never seems to gain any traction. Very, very quickly. There are real technical reasons.

Extensibility is good. I believe the Matrix developers when they say it's extensible. (I've been a user for several years now. Features regularly get added in backwards-compatible and smooth ways.) Extensibility in the way XMPP tried to go about it -- for patching over core inadequacies of the basic protocol -- is not the same beast.

What would really help with XMPP is some idea of profiles - like a Jabber2018 profile that requires all of the XEPs used by Conversations[0]. That way, client and server implementers would know exactly what they need to be compliant with other implementations using the same profile.

[0]: https://conversations.im/

That exists. Here are the 2018 profiles (they are split into basic/advanced): https://xmpp.org/extensions/xep-0387.html

They are only part of the solution however because:

- Open-source projects are developed by volunteers, and implement what their users (actually typically their developers) want implemented. Having a document that says "you should implement these things" is good for guidance, but it doesn't magically get them implemented.

- Commercial projects will implement the things that they can get revenue from. Having a document that says "you should implement these things" doesn't magically make those things earn the company money.

E.g. in the open-source category, Pidgin is the classic example of a once-popular client that has fallen behind (with feature requests for some "new" protocol features open on their bug tracker for many many years). The document doesn't fix this part.

But yes, I do agree it's an essential part of solving the problem.

It's still so common to see clients tout a list of supported XEPs like they expect users to know what any of it means. It's great there is a list of importantant XEPs but this must be communicated to users better so that it's easier to find suitable clients. It might even be good to refer to compliant clients as something other than Jabber/XMPP then you can easily search for an "XTalk" (for example) client and know this means the full list of XEPs are supported. Part of this might include some new XEPs to indicate to users when they are communicating with non-XTalk users so the experience might differ. A certification programme even might help motivate devs to meet compliance quickly.

One can dream anyway.

I doubt any platform uses vanilla XMPP, that wouldn't be sane, but like using XML, XMPP is a relic of its era. Whose to say what super-critical "how can you not have this as standard!?!" feature won't come up in the next 5 years that Matrix devs hadn't even known of in 2018. That doesn't mean we should flip over the table and start again every now and again and hope the users will follow, this is why extensibility is an important part of the recipe and make ways to ensure the standards of the day are always implemented before being considered ready for prime time.

IDs, though.

Like, in the abstract, I agree with you completely.

But IDs.

I don't know, IRC in its current form doesn't have any message-ids and yet seems to be working fine.

It also does not and never did support consistent history or multi-device.

And it never will, because having some way to identify a message is a prerequisite for even phrasing the question "does device_foo have message_x synced yet?"

Bouncers don't count. They reintroduce centralization as a way to solve the problem. It's a solution, of a sort, but it's a fairly poor one if your original aim was decentralization.

And "working fine" is also frankly a bit of a stretch, isn't it? I can't count how many times I've seen a "netsplit" in freenode or the other IRC networks. That means messages aren't delivered. It's so common we have a word for it: "netsplit".

I shouldn't dunk on IRC too much, because at least it's honest about itself for being simple, and there's a virtue in that. But if we were to construct a statement like "IRC manages reliable message delivery without any concept of message IDs", then that would be false (or technically, vacuous), because it doesn't manage reliable message delivery.

I don't think you get the idea of extensions in XMPP. If something is published as a XEP, it doesn't mean "it's a completely optional thing you can ignore". It does mean "it's a thing that might not be useful in some cases, so it doesn't belong to the core that everyone and their pets must implement".

If you build an IM client for XMPP (which is just one of the use cases of XMPP), you absolutely have to implement at least a few of those extensions in order to have basic functionality. Heck, there's even a separate RFC for XMPP-IM on top of the XMPP-Core RFC. That's expected, by design and very fine.

Of course, what's considered "basic" changes over time and it's easy to find outdated clients out there. That's why Compliance Profiles exist, so you can look around for clients that implement, for instance, 2018 desktop IM profile. It also imposes some burden on new implementations to consider backwards compatibility with those clients. No solution is perfect, but while I think Matrix' way allows them to move forward faster now, in the long term it's probably XMPP that will age more gracefully.

> I don't think you get the idea of extensions in XMPP

I don't think you read the other comments I linked to. My fault for not copy-pasting the entire thing every time the "y u not be exactly XMPP" train comes on HN again...

---

> - unique message IDs? Absent. XEPs kind of provide; but I can't tell you which of the three or for relevant ones are the most relevant (AMP IDs from XEP 0079? Stream Management from XEP 0198? Acking from XEP 0184? Something from Carbons or MAM in 0313 or 0280? You know, if you wanted some light reading...).

> - multi device? Oh. My. God. It's bananas. The spec behavior is that whenever a client sends a message, the server is supposed to consider that one the most alive, and then route all future messages exclusively to that one. So you send a message on your phone? Yeah, your desktop is just going to silently stop receiving messages.

> - there's a concept of "message carbons" to deal with this. This involves re-sending all your messages back to the server after you receive them, with special instructions to send them back again to your other clients. The amount of redundantly redundant XML involved is eyewatering.

> Combine that multidevice behavior (messages can get randomly routed anywhere at any time) with the wild-west nature of message delivery acks, and you can see how ridiculously difficult this makes the basic idea of "all clients should see the same picture".

> Overall, the XEP process, conceptually, is a great example of open extensibility. The trouble is, so much of this stuff is core to sane message delivery semantics that it really, practically speaking, causes huge problems when it's all considered "extensions". Stuff like message IDs fundamentally shouldn't be an extension because it's just too critical that all minimum-viable clients agree. You just can't build higher level stuff without that. XEPs are great. A community process for extensions should exist. It just needs to exist for extensions, not subsume the total set of realistic minimum viable features.

(All of this is a shameless repost from https://news.ycombinator.com/item?id=9772968 ; please forgive my humble attempt to save the dear reader from the need to traverse any links.)

---

I get the idea of extensions in XMPP. I have tried to implement them. It is based on this first-hand experience that I am now arguing that the XEPs are a swamp. It is time to move on.

Yes, even with Compliance Profiles. It's too late. The horses have left the barn, developed tools, formed their own complex civilizations, invented light speed travel, and in fact left the galaxy. That's how far out of the barn they are.

Message ID and multi-device messaging is in fact fundamentally broken in such a unique, fascinating way in XMPP that all other clients will cease to operate correctly in the presence of a single bad client.

That's right. Any bad client can cause other clients which are following the spec to the letter to begin to drop messages, forward them incorrectly, and generally lose history and form desynchronized views of the world. I'm not kidding. If you haven't read the XEPs, and don't believe me, go read the XEPs. The only way to implement a reasonably correct multi-device behavior requires ignoring significant parts of XMPP because it is wrong.

Start. Over. XMPP is not your savior.

Guess what: I was implementing (at least some of) them too, successfully.

> but I can't tell you which of the three or for relevant ones are the most relevant

I can. Check out XEP-0359. MAM and some other XEPs depend on it.

Also, unique message IDs are not a feature. They are means to provide some feature. You will deal with different kinds of IDs when trying to ensure network reliability (0198) or when trying to implement read markers and delivery receipts (0333, 0184). Without telling us what feature would you like to implement (threading? attachments? something like 0367?) just spewing all different kinds of IDs from various XEPs starts to look like a FUD to confuse uninformed reader.

> multi device? Oh. My. God. It's bananas.

I can agree with opinion that the protocol is really "bananas", but it works. Basically 0280 solves this issue, especially when combined with 0198 and 0313 - the only situation where it breaks is if you connect with a legacy client that doesn't support carbons and set it a presence with higher priority than any other client. You'll still get the content of the messages on modern clients thanks to MAM though. So, basically, don't use legacy clients on your account, cause they will degrade your overall experience. Doesn't seem very surprising, especially given that you can't connect at all to most IM networks out there using outdated clients. My N900 with stock OS won't connect to Skype or Facebook at all, even though it tells me it can. It will to XMPP - with limited functionality, but it's better than nothing.

XMPP accumulated its great share of backwards compatibility weirdness, but it doesn't seem unmanageable and I believe that the way XEPs are handled made it better, not worse. I suspect that Matrix might be in worse shape when it reaches current age of XMPP. I would like to be wrong though, I wish them best.

So basically you reiterate what I said. And specifically:

> So, basically, don't use legacy clients on your account, cause they will degrade your overall experience.

Yes. As I said.

But let's not put a bouquet of roses on it: it's even worse than that, because you don't just have a degraded experience when using a legacy client. Using a "legacy client" -- of which there are many because of the XEP swamp -- will make all other clients behave wrong. And it is impossible[1] to tell which client (or server) to blame. Thus resulting in even the most modern and chose-the-right-XEPs clients regularly getting bug reports about lost messages... even if some other client is at fault.

If that's not a trainwreck I don't know what is.

---

[1] inb4: nothing is "impossible". But "practically speaking impossible for an end user who has other things to get done today and a limited time budget for debugging a problem which is a $n$-dimensional matrix of their clients and servers in use", yes.

I briefly worked on an XMPP client for an internal chat app. I ran into tons of issues (multi device problems being the main one I recall. This was over 5 years ago now.) It would've been simpler to build my own client and server with a custom protocol.

Someone else really wanted to build on XMPP though. XMPP has fundamental issues. No thanks.

Matrix is very extensible too; it's just that we aim to merge extensions as rapidly as possible into the core monolithic spec, rather than having them get fragmented into optional extensions.

I wonder what happens when there's 10 implementations of the matrix spec controlled by different groups. Being able to indicate where the implementation gap is seems like a strength rather than a weakness.

Hm, even when there are 10 different server implementations (or the ~50 client implementations there are today), we’d hope they would align to the same spec - just as with HTTP, SMTP, TCP/IP etc. There’s room for experimentation without introducing fragmentation, and any useful experiments would hopefully rapidly get merged into the spec. And if they don’t then yes, you risk a fork on that particular feature (the equivalent of only Exchange supporting email recalls, or browser vendor extensions), but it’s not the end of the world.

I'm not sure if there's really a place around for an IM network that's moving forward about as fast as HTTP, SMTP and TCP/IP are :)

it really depends, if the features are difficult to implement it may not be very quick at all, we see this with XMPP often where features take years to implement or never appear. If a homeserver targets a particular use-case then they may actively choose to ignore certain features that are not relevent. Really when people talk about the flaws of XMPP they often have in mind the problems of private e2e encrypted chat on mobile which is a hard problem to crack, but there's many more private servers inside organisations that don't even federate and work fine for how they are used. This flexibility perhaps makes it better to view XMPP as a toolkit for building networks out of, similar to how IRC networks which have a converged set of servers but outside that network all bets are off. The idea of viewing XMPP as a single network inevitably causes it to look fragmented but there are pockets within the network that interoperate just fine they just don't identify themselves as a group.

Reply on news.ycombinator.com