Easy XMPP: The Challenges | Hacker News Reader

Easy XMPP: The Challenges

xmpp.org

68 pointsge0rg60 comments

Improving UX helps somewhat but how do you convince users when most have unlimited texts with their plan and Facebook Messenger, Whatsapp, Viber, Telegram, Signal... to choose from?

I love the federation. I love the JID (so much better than a phone number). I love the handling of multiple clients connected to the same account. But it didn't take. Even my tech friends ditched it. IRC held strong against all of the above and even Slack to some extent but XMPP just faded away.

Signal has a gif search and SMS/MMS fallback. What does XMPP offer to normies?

Maybe the problem here is that we advertise XMPP as if it were a service, when it's a protocol that "normies" shouldn't ever have to hear about and care about. They should just be using the "ChatPro Company Fancy Name" app on their phone (which has fancy gif search), and not have to understand that it's federated under the hood. XMPP doesn't have to offer these things, the service using it does. That's the part I think the XMPP community is missing right now, a good public chat service that's federated and not strictly for businesses.

OK but now you're targeting the provider. Why should ChatPro use XMPP?

Whisper Systems wanted to get (their clever version of) strong end to end encryption in the hands of regulars users so they created Signal. Moxie has blog posts[1] and long discussions why XMPP was unsuitable in which he answers your idea: if vast majority of users will coalesce around ChatPro anyway, why not get the benefits of a centralized communicator?

Others want to silo their users, or offer this and that fancy feature. Why would they use XMPP?

And what is the benefit of using XMPP if it's really just ChatPro with a mostly unused s2s federation port open on their server somewhere?

[1] https://whispersystems.org/blog/the-ecosystem-is-moving/

I think that ChatPro (or any other SV startup) is motivated in silo lock-in of their users indeed. Normal people don't care much about it, they just move to the next silo at some point.

I see an interesting niche for XMPP in business deployments, basically what's currently covered by Slack, where you (as a user) often need to have multiple accounts just to participate in different teams of your BigCorp.

XMPP allows a company to host their IM infrastructure on-premise or to use a service like ChatPro, and there is a business interest in requiring federation to allow easier interop with suppliers and customers.

We just failed to execute on that mission so far... ;-)

We need a GitLab for XMPP.

GitLab is looking at Matrix https://matrix.org/

Hipchat is all xmpp. We didn't fail, it's just not advertized enough!

It looks like in some regions HipChat is a thing, while in others people don't even know what it is. I haven't seen it yet in my professional practice in Germany, whereas Lync (Skype 4 business) is almost everywhere...

> Maybe the problem here is that we advertise XMPP as if it were a service, when it's a protocol that "normies" shouldn't ever have to hear about and care about.

I don't think this is true. With email, SMTP is dominant, I do not know of any surviving legacy email service. To today's normie, email and SMTP are synonymous.

XMPP is an even more garbage name than SMTP, but if a standard united the market like it did with email, then there would be a word that is synonymous with that standard.

I think probably the best bet would be to try to turn SMTP into an IM protocol. Then people will make no (protocol-level) distinction between an email thread and an IM conversation. Threads and conversations can be both email.

But you're proving the parent poster's point here. Normies never hear of SMTP except when they copy and paste some incantation from their email provider's website. It's email, it works, and they have no idea how.

There's only literally 3 options. PLAIN/SSL/STARTTLS. I see one of XMPP's major problems that even people who liked, advocated, and worked (not extensively, but more than most I guess) with XMPP can't remember the name or details of a single XEP. In theory everything is compatible, in reality you're lucky if s2s works with the right crypto and you don't get gibberish because of OTR.

I think you hit the nail on the head. People simply use what's offered by companies. If we think people should be using applications that utilize certain protocols, then what we really need to do is convince/incentivize companies to build their applications to use those protocols.

This post comes at an interesting timing for us. Our product has a real time chat component, and currently is done over XMPP. We implemented using custom components on top of Tigase, a Java based XMPP server.

Due to a variety of reasons our chat services overall have fallen out of shape and we are in the process of considering a full re-write. Part of the consideration is ditching XMPP altogether. It seemed to me that for products where chat is only one of the features, XMPP feels like an awkward add-on, requiring its own set of IDs and auth protocols.

AFAIK many popular chat services these days are no longer based on XMPP, for one reason or another (e.g. scalability). Would like to hear any recommendations on what the best ways to do real time chat services nowadays.

If you're making a service that's primarily chat, just use XMPP (disclaimer: I used to work for a large chat system based on XMPP and got involved with the protocol development when I did so). It scales very well (if it worked for Google Talk and HipChat…), but it does require that you understand the protocol. Network protocols have tons of subtle edge cases that will cause problems; inventing your own is just a bad idea, even if the existing ones require a bit more work to learn and get going with.

Did you guys write your own implementation or use an existing server?

We wrote our own server based on Python/Twisted (which has an XMPP implementation in Twisted Words). The Twisted implementation of XMPP is a bit outdated now, but it wasn't too terrible to work with. There may be more up to date libraries; in Python land aioxmpp seems nice.

Thanks for the info!

The main alternative is probably Matrix currently (Disclaimer: i work on it). The advantages are that it's a pretty simple HTTP API, and there are good SDKs for JS, iOS & Android. The biggest disadvantage is probably that the UI SDKs (matrix-react-sdk, matrix-ios-kit and matrix-android-sdk) have ended up quite entangled in the Riot client codebase, and we need to decouple them so that they can be better used to implement custom chat/collaboration components.

Last I checked, the Synapse homeserver was heavy on resource consumption and therefore tough to run at home on, say, a RPi. How much progress has been made towards more usable homeservers?

I have a Synapse running (among other things) on a 1/1 VM. It's at the top of `top` (heh) when sorting by memory:

  PID USER      PR  NI    VIRT    RES %CPU %MEM     TIME+ S COMMAND                                                                                                                                                 
  486 synapse   20   0  510,1m  81,7m  0,0  8,2   3:03.73 S python2.7
  ...

There are six user accounts on that Synapse and few activity (less than 100 messages per day).

When I first installed it to play around with it, I joined #matrix:matrix.org and it brought Synapse to a grinding halt. Took quite some time to catch up with the thousands of peers it suddenly had to federate with, and I ultimately cleared Synapse's database to make it forget about all these peers.

For comparison, I also have a Prosody (i.e. an XMPP server) on the same box, which has about 10 user accounts and my own account is joined to a few moderately busy MUCs, so the number of messages is at least one order of magnitude higher. Yet memory usage is about one order of magnitude smaller (CPU usage is too infrequent to compare, but look at the "TIME" column for an initial comparison):

  $ psgrep 'prosody\|synapse'
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  synapse    486  0.1  8.1 522332 83664 ?        Ssl  Aug13   3:04 /usr/bin/python2.7 -m synapse.app.homeserver --config-path=/etc/synapse/homeserver.yaml --log-config=/etc/synapse/log_config.yaml
  prosody    574  0.0  1.0  52408 10476 ?        S    Aug13   0:27 lua5.1 /usr/lib/prosody/../../bin/prosody

Synapse does a lot more than Prosody though. If a Prosody server hosting a MUC goes down, the MUC is down. If the Matrix homeserver that created a room goes down, the room's still there as long as there are other servers participating. That decentralization does have its tradeoffs.

It's also the first Matrix server, so there's that. Its replacement is in progress: https://github.com/matrix-org/dendrite

Oh nice. That should have a lot less footprint indeed, both in terms of CPU usage and # of dependencies. (Not entirely sure about memory usage, but that should also be better for a compiled language.)

yup, synapse is heavy (as the rest of the thread says). this is mainly due to the DB schema being a bit naive, and it caching everything in RAM to speed things up. There are also some operations it does which spike RAM usage (which is then never reclaimed, thanks to Python2's malloc being a bit dumb). As others have said, Synapse is doing a lot more than something like Prosody. Dendrite on the other hand should be good for running on an RPi - we should have an idea this week, where the first monolithic Dendrite binary is due to land.

Recently I spent an entire evening trying to get Gajim and ChatSecure work together with OMEMO and delivery of messages to multiple devices. I'm a technical person, but I failed, I got into all kinds of weird error modes, but ultimately no success. What a frustrating evening that was, a waste of time.

I have stopped to recommend Jabber/XMPP to people. Too many times I did that over the past 15 years and even when I succeeded in making them curious to try out... in the end there were too many problems and things didn't work right, and ultimately I too often looked like an idiot for recommending what is to other people just a broken tool. In the end, people don't care about "it's the choice that gives you freedom". People use the things that just work. And honestly, I can't even blame them (even if I'm personally not willing to give in to the privacy compromises that come with the other tools).

After so many years fighting for XMPP, in my humble opinion it will be better to use matrix instead.

XMPP was mostly killed by Google.

Why would matrix adoption succeed where XMPP adoption failed?

They've done a few good things (Olm), and don't have to worry about backwards compatibility yet, but for the most part it seems like they're just reinventing the wheel and re-discovering problems XMPP fixed 15–20 years ago, except with a giant distributed graph protocol that doesn't scale very well (not that this matters for most people that aren't, eg. Google).

Matrix Riot is really good.

Does that have anything to do with Matrix though, or just that one company implemented a nice client? Conversations for Android/XMPP is also very good. There aren't any good desktop options, unfortunately, but I don't think this is necessarily an XMPP problem (although it is a problem that should be fixed), just that all the companies that write good desktop clients for XMPP make them proprietary or locked to their specific chat service.

https://github.com/dino/dino Shows a lot of promise. Would love a nice Windows client as well though.

With Mix coming, I think XMPP will become/remain pretty awesome. https://docs.ejabberd.im/tutorials/mix-010/

Thanks for the links; I wasn't aware ejabberd had an experimental MIX impementation! I switched to Prosody for my personal server a while back just because it was easier to setup/configure, but maybe it's time to look at ejabberd again (although I hate to run a full beam VM just for a single small server; as nice as beam is, it's still a lot of extra stuff to run on my tiny server).

I actually switched to Ejabberd from Prosody a few days ago! I hear you, but Ejabberd seems snappier now than from what I remember. Well worth a try, importing users from Prosody was a breeze.

There aren't any client's that can handle mix yet, but I think I saw something about it at Conversations GitHub.

Dino looks really cool, thanks for the link! Looks like it doesn't yet support MAM[0], but aside from that this may be the closest thing to a Conversations equivalent I've seen for the desktop.

https://github.com/dino/dino/tree/master/xmpp-vala/src/modul...

> or just that one company implemented a nice client?

That is a huge deal. Having a de-facto reference client for all platforms with most of the features is very important. XMPP never had that. Google could have done it but they didn't.

I used XMPP for more than a decade and tried everyone else to switch, but it has always been lacking in many ways in all platforms, without a safe fallback option.

> There aren't any good desktop options, unfortunately, but I don't think this is necessarily an XMPP problem (although it is a problem that should be fixed), just that all the companies that write good desktop clients for XMPP make them proprietary or locked to their specific chat service.

For the record, we have a working Desktop/Android client on the way, which supports file sharing, carbon, MAM, Jingle (for file transfer only so far, but voice/video is planed at some point), with unique interface, and more features to come. It is also working on web and console (TUI + CLI).

UI may need improvement, help is welcome.

https://www.goffi.org/blog/goffi/9ff9bb64-1e1f-4334-80e5-a86... and https://salut-a-toi.org/

Easy file sharing? Shared chat logs across devices? Easy encryption? Easy voice chats? Video chats? Group voice/video chats?

These are all things that I have on Matrix out-of-the-box and that I have some trouble getting on XMPP.

I run my own XMPP server, I have been doing it for a couple of years, but getting some modern features is very difficult, if not impossible. But with Matrix everything is quite easy.

I feel like clients and servers have fixed most of these. I've never had to do anything to have chat logs, all devices just have it thanks to MAM, file sharing is done with HTTP Upload, encryption is done with OMEMO (axolotl/Olm, although this one is still making its way into clients many support it already), voice and video is still a problem but lots of XMPP based services exist to make it easier (Jitsi Meet). It's only not a problem with Matrix yet because they have a single client; if they ever get as popular as XMPP and have many, they'll have the same fragmentation issues. What the XSF needs is one really good desktop client to compete; something with a feature set comparable to Convesations on Android, and good looks to boot.

>I've never had to do anything to have chat logs, all devices just have it thanks to MAM

Android has Conversations, which has pretty good XEP support. But I can count on one hand the number of desktop clients that support this, and none of them are Pidgin[0] or Adium, which are probably the two most popular desktop XMPP clients (though it's hard to know for sure). Gajim supports it but not for multi-user chats[1].

Chat history is certainly not a solved problem overall in XMPP.

>It's only not a problem with Matrix yet because they have a single client

Matrix has lots of clients, the difference is it has an official reference client that supports (almost) all the protocol's features.

[0] https://developer.pidgin.im/ticket/15653 [1] https://dev.gajim.org/gajim/gajim/issues/8036

File sharing, cross-device logs and encryption have been addressed in the last one-two years, pushed forward by Conversations on Android. However, getting that into other clients (e.g. on desktop/web) requires dedicated developers, and this is where XMPP currently lacks a little bit - not for lack of motivation but for lack of time.

"Addressed" as in "most of the XEPs are marked as experimental and are subject to change".

That might look a bit weird on the outside, but this is how protocol design works in the XSF. First you make a proposal (which is marked as Experimental), then you implement it and wait for other people to implement it as well, and then you look how well it works and whether it has rough edges that need fixing. If it is considered good, it moves on to Draft and then to Final [0].

This process could be faster for sure, but that requires independent developers to work on implementations and to share their experience.

[0] https://xmpp.org/extensions/xep-0001.html#approval-std

Is there a set of mobile and desktop clients that support all these things you said were "addressed" properly? People always parade Conversations, but if I want phone-only I already have to many messengers.

Because if have to look through tables and bug trackers to figure out what client/server combination I can use (and have to pester my friends to use(!)) you can't count something as "addressed" IMHO. I still use only a very basic set of features with XMPP, because in my experience attempts to get anything advanced are way to time-intensive to be worth it for the few buddies still on XMPP, and I can't really push new people onto it that way.

> WARNING: This Standards-Track document is Experimental. Publication as an XMPP Extension Protocol does not imply approval of this proposal by the XMPP Standards Foundation. Implementation of the protocol described herein is encouraged in exploratory implementations, but production systems are advised to carefully consider whether it is appropriate to deploy implementations of this protocol before it advances to a status of Draft.

Let me stress this:

> production systems are advised to carefully consider whether it is appropriate to deploy implementations of this protocol before it advances to a status of Draft

That's from XEP-0363: HTTP File Upload.

When incompatible changes are applied, the namespace used for feature negotiation is usually version-bumped, so implementations can choose to support and negotiate different versions of a given extension whenever that seems sensible.

Not sure this works as well elsewhere (e.g. the Matrix spec just states that it's "still evolving: the APIs are not yet frozen and this document is in places a work in progress or stale").

XEP-0363: HTTP File Upload

> WARNING: This Standards-Track document is Experimental. Publication as an XMPP Extension Protocol does not imply approval of this proposal by the XMPP Standards Foundation. Implementation of the protocol described herein is encouraged in exploratory implementations, but production systems are advised to carefully consider whether it is appropriate to deploy implementations of this protocol before it advances to a status of Draft.

It means that there will be one client that supports this (Conversations for Android) and maybe one server supporting this (probably ejabberd). Good luck navigating the minefield :)

Meanwhile I can install Slack on multiple devices and have the same functionality everywhere :)

Conversations is substantially inferior to WhatsApp.

No encryption by default. No sent and read receipt by default. User account and password hazzle. No audio chat. No video chat.

XMPP just doesn't cut it.

None of these things have anything to do with XMPP (which supports all of them). I agree though, a more feature-rich client would be nice.

Yes, they do have to do with XMPP. Conversations had for some time encryption enabled by default, but they needed to disable it again.

I think XMPP is doomed, because it doesn't enforce the things I listed, and I think it will never do so.

I like Matrix, but don't you think Riot has a really convoluted UI?

I was hoping to be able to adopt it for communications within my family and friends, but it did confuse me a lot. Especially on Android.

Riot's UI could be way better (i work on it), but we're constantly iterating to try to nail the UX. We changed it a lot on Android last month though (https://medium.com/@RiotChat/riot-im-unveils-the-next-genera...) so might be worth trying if you haven't recently.

Looks nice. I didn't try this version. My last attempt was on F-Droid last month, just before the upgrade.

Sigh…the struggles of dealing with Google Talk’s limited Jabber interface…

Google Talk is an interesting case; it was great having non-technical friends use a federated system (obviously they didn't know that, nor do they need to know that; this is one of the big problems with most small XMPP-based services, they only focus on technical people and use federation as a marketing point), but it was aweful dealing with, eg. Google refusing to follow certain standards, refusing to do TLS over S2S connections, etc. sigh indeed.

Another very important thing is some form of free hosting.

I used to have my own prosody server a long long time ago. Eventually, I got fed up, since very few people used XMPP. I just got rid of it and barely noticed any difference.

If I had some place where I could register my own user@domain that matches my own email, I'd use it for sure. But shelling out 5-10 USD a month for that isn't worth while, not for me, nor any other early adopter (especially when almost any other IM service is "free").

There are many free community servers, but you can't host your own domain with them. If this is a requirement, you can have a paid hosting for 1€/address/mo at https://account.conversations.im/domain/

That's exactly my point; having a paid-for entry barrier is already to high for something lacking adoption.

And since JIDs look email-like, people prefer them to BE exactly their email.

I am surprised this made it to the front page. XMPP is no longer relevant, please focus your efforts on a protocol that better meets your requirements. No matter what those requirements are, I would have hard time believing that XMPP is it.

Getting ejabberd set up to do encryption and file sharing was a huge pain in the butt. Worth it because I don't like SMS, but still freaking crazy hard to get running through a firewall.

XMPP: Captain Obvious edition

Add this article to the one above: https://gultsch.de/xmpp_2016.html

It doesn't help that most of the features expected of a modern chat are "warning: experimental" XEPs and supported by at most one chat client and at most one XMPP server.

Bazaar indeed

Yeah, frankly I think the technical specifics of XMPP are why I need about ten incompatible messaging apps with mostly the same features.

It simply has never been straightforward enough to do the right thing. Unlike with functioning protocols like HTTP and SMTP. XML was a fundamentally poor choice for wire and processing efficiency, XEPs don't get ratified quickly enough, and there's a chicken-and-egg problem between server and client software in implementing them even if they do.

XMPP is obsolete and irrelevant - please stop wasting your time with it.

Reply on news.ycombinator.com