Jabber/XMPP: How Do We Gain Traction?
gultsch.de
gultsch.de
People don't really sign up for protocols, they download apps. Like I often refer to the fediverse as "Mastodon" even though I know that's not entirely correct, because it's the fastest, easiest way to communicate what I'm talking about - microblogging. So you really need a single, good app you can just tell people to download.
It’s mostly whining that the world is too unfair because capitalism and people are dumb for choosing Signal, etc.
There are still a LOT of things very much up in the air, such as message deletion, which work inconsistently from client to client, but message history is not it.
I'm not sure what you mean by not handling encrypted messages fully, it handles encrypted messages fine.
as a user you cannot, even if you self host, because you are not the one controlling message storage
> I'm not sure what you mean by not handling encrypted messages fully, it handles encrypted messages fine.
you can load encrypted messages (including keys if they are needed to decrypt them if necessary) using a fresh login such that you can read them again?
"Because Snikket is designed natively as a tightly integrated suite of Docker containers built around the Prosody XMPP server, running it natively on FreeBSD is complex."
Holy shit
Snikket aimed to fill the gap for people who wanted an "out of the box" XMPP distribution kind of thing, and it's good at that. But shipping a bunch of software components with opinionated configurations doesn't really mesh well with distribution-level package management, and only really works with a container kind of ecosystem.
Both routes exist and are equally viable. Prosody has ports available for all the popular BSDs, and installing it should be a breeze. Just be prepared to then also set up a TURN server if you want calls, and potentially some web server stuff to if you want some of those features. After installing Prosody you can run 'prosodyctl check features' and it will tell you if you're missing any of the common stuff people want in a modern messaging app.
Hope this helps :)
As for apps, people simply prefer instant messengers purposefully designed for their platform. There is genuinely a lot of platform-specific quirks around things like push notifications, which a non-iPhone developer not have the motivation to figure out. Besides that, a lot of these predate contemporary cross-platform technologies like Flutter.
Plus the reality of the fediverse is if you want to use multiple platforms, you wind up needing multiple accounts anyway. Like if I want to upload to Peertube, I need to make a Peertube account, so we're back to where we started of having multiple places to follow people since I can't use my Mastodon account to upload to Peertube. I'm able to have some cross-platform interaction, like commenting on videos from mastodon - but there's issues there like, Pixelfed only showing people posts with images.
When talking to most people I just call the whole thing Mastodon and treat it like its a single app/service, because that's what people are used to. If I start trying to tout how the fediverse is a decentralized platform of multiple applications and you can kinda/sorta interact across platforms - people's eyes glaze right over. They just do not care.
For apps - I've never met a person that picks a messenger because its tweaked for their platform. They just prefer whatever their friends are on. iMessage is something of a odd case since Apple just rolls you into it automatically and presents it via the default app.
It just so happens it splits into well-maintained projects supporting the concensus features, and dead or dying projects that don't, modulo a couple contrarians, with a couple more features floating in limbo, having finalized or draft specs that nobody really implements.
That sort of problem isn't best fixed by centralization on one client/server but just certification, which Conversations/ModernXMPP already leads.
As for apps, yes normal people aren't very conscious of what a native app is, but they do feel when their app lags on their older phone, doesn't recieve push notifications, or just looks a little odd in places, which can happen in a one size fits all app. Some of the most popular messengers, like WhatsApp or Telegram, to my knowledge use platform specific tooling, and normal people genuinely appreciate how snappy they feel.
But the issue is more so about developer time. Yes a cross platform app lets you reuse the same work everywhere, but the lead developer still has personal blind spots, and might simply not consider or be able to test things like bespoke Apple workarounds for non-iCloud/encrypted chat push.
Plus it's a communication protocol, like email; we should be expecting people reimplementing it in the weirdest of places. For software like this, the duplication of effort is what drives you to make an elegant design in the first place.
If you want xmpp to gain traction you need an app that exists on both mobile platforms. It can (and should) have per-platform tweaks and everything to be performant. But a person needs to be able to just tell their friend a single app, they need to be able to find it in their app store.
The question is "how does xmpp gain traction?" and the answer is having a single flagship app that runs on everything.
I can expect this from our parents and grandparents, but not from younger generations that were born with PCs and smartphones. They need to learn; we need to educate them. Otherwise they will stay being the perfect fodder for Big Corpos and their monopolistic tactics.
We need to start focusing on the protocols, not the platforms.
People don't use single browser and single email client, why IM should be different? It's a deficiency when you're forced to use a single "official client".
You sign up for Gmail, you get the Gmail app. You sign up for Outlook, you get the Outlook app. 1 service = 1 app.
For browser - on mobile most people use whatever the default browser on their device is. On desktop - Chrome. Pretty much everybody just downloads and uses Chrome.
But most people seem to use Chrome on every device, no matter how much I wish they didn't.
Google has since spent the time making Chrome more locked down and user-hostile from a privacy perspective but the conventional wisdom for years was “just use Chrome” (for good reason) and that inertia continues :/
Also last I checked - on iOS devices even if you use Chrome, you're really using Safari - I believe apple forces all web content to be rendered using the built-in WebKit.
This is both true and also if followed would fly in the face of what XMPP (and the Mastodon-flavored ActivityPub-powered fediverse of microblogs-and-more) is supposed to be about. And it's also both true and yet people understand what "email" means and how to use it without constantly and consistently running into issues trying to bang out a reply to alice@gmail.example even when the sender's inbox is hosted @yahoo.com—nor do folks spend much time thinking about how and why AT&T's SMSes (for example) are able to make their way to their friend's device even though they use Verizon (to give another example—and if they even know which provider their friend uses at all).
One thing that Jabber (and, later, Mastodon) did wrong was to take the unfortunate stance that it wouldn't be too big of a deal to adopt email-like identifiers without actually being email or implying that Internet-standard email services are available; it was felt that users would just be smart enough to adapt to it. This was a mistake.
The confusion with email, though, can be exploited as useful momentum—something that the network has going for it, instead of a flaw.
If the author and the rest of the Jabber/XMPP community wants the public to "give Jabber/XMPP a shot", then it probably does need (a) a flagship instance (a la mastodon.social) that controls multiple domains (the way that many email providers like Runbox or Fastmail do) and requires the user to pick which one they want their handle to be associated with at signup, in order to introduce email-like decentralization to the userbase as early as possible, and that (b) raises the bar by setting a standard among Jabber/XMPP instances and actually offering email services to that userbase (prior art: Google married (XMPP-based) Gtalk with Gmail in the early days).
While undertaking all of this, an effort to update the XMPP protocol (a la JMAP, but in a backwards-compatible way) while simultaneously reconciling it with legacy email (also in a backwards-compatible way) wouldn't hurt—where "backwards-compatible" here means "to gracefully degrade and provide a fallback" (prior art: Delta Chat).
They've been working on a version 3 for years but no idea if that will ever happen.
As for version 3, yeah turns out that things take a very long time a tiny team that has zero financial support.
In short, I wouldn't recommend Pidgin for someone who wants a modern messaging app.
It just needs feature versions - you support this one, means you support all of the features deemed important for users, no partial mishmash.
Also sending media (or even rich text) never really worked as each client implemented it differently. While whatsapp just works...
Many Android apps give the option of using either FCM (Firebase Cloud Messaging, Google's push service), UnifiedPush or polling.
That is how you gain traction: by having people use it, realise the user experience is comparable or in some cases superior to that of other messaging systems - the party chat thing I mentioned above being a good example - and then realise they dont need to hand over their data to an ad broker (Google, Metafacebook, the fruit factory) or the government (all of them through some "chat control" type law).
If I understand correctly, Google and Apple have a way for clients to request push tokens, but in order to use the token, you have to authenticate as the app owner, so a client unaffiliated with the server can't simply send a push token to a server and have that be used ... Instead you'd need that push proxy server.
That proxy server comes with concerns about reliability and privacy and etc, but if you do pushes without cleartext content (and you should!), the footprint is minimal and you gotta do what you gotta do.
I don't follow xmpp/jabber, but I'd hope there's already an XEP for this. It's a pretty apparent need for the last I dunno 15 years and it has a clear solution.
With the right agitation, maybe you could get sharable platform push tokens and skip the intermediary app developer pusg server, but that seems unlikely.
You probably also want something in the push system to help trace push problems. For small chat servers and smallish client push proxy servers, I expect minimal operations staffing, but when messages are being delayed because of push problems, you want to be able to let affected clients know that they need to be more agressive with periodic/background connections until push works again. You may also want to let users or the administrators of the proxy push servers know as well. Pushes might be initiated late on the origin, delayed between the origin and the proxy, queued for a long time at the proxy, delayed between the proxy and platform push, or delayed within platform push (including on device delays from Doze, etc).
You've got to be able to detect that so you can make adjustments and reduce delay. Messaging delay reduces user satisfaction.
Yes, and it has been solved long time ago already and is now widely supported by both servers and clients.
> I don't follow xmpp/jabber, but I'd hope there's already an XEP for this.
See XEP-0198 from 2004 and XEP-0357 from 2015.
Taking a quick look, this one is marked deferred; I don't know what that means, but it sounds like it's not an accepted standard?. It requires the application push server to be an XMPP server (which I suspect is challenging for client developers, compared to having a https url that takes a POST or similar) and doesn't include a method for a client to request a test push to validate the setup or discuss feedback to the client for push failure.
This doesn't feel solved to me.
I didn't review XEP-0198 closely, based on perhipheral ecosystem interaction, I do think that one is solved.
The client phone app tells your XMPP server which gateway to use (or maybe it's vice-versa, the gateway contacts your server. I forget). In any event XMPP servers like Prosody support this out-of-the-box.
>Also sending media (or even rich text) never really worked as each client implemented it differently.
Everyone use HTTP upload nowadays, it works all the time.
No it doesn't. Firstly, an out-of-band upload is ridiculous. Secondly, it doesn't support upload resumption in case the network is bad.
It is a bit, true, but the alternatives are worse. XEP-0231 (Bits of Binary) Base64-encoded data inside the text message is more ridiculous idea. Direct P2P connections, which were widely used before HTTP upload, are mostly not working in our day.
>it doesn't support upload resumption in case the network is bad.
There's a draft for HTTP for it: https://httpwg.org/http-extensions/draft-ietf-httpbis-resuma...
It's not that "alternatives are worse, it's that xmpp is a broken protocol, which doesn't include support for anything, even such a basic thing"
>There's a draft
LOOOOOOOOL
Which means that it will never work for human messaging, due to all the IoT cruft.
>Thanks to Daniel, the author of this post, we now have communication profiles which the IM-oriented servers and clients follow.
LOL, he only implemented p2p transfers when it bit himself in the tail on a plane flight.
Conversations is the best of them all, but it doesn't mean that it is good.
HTTP may not have native resumption, but it's not hard to do partial upload and resumption over HTTP, you just need agreement on parameters / a protocol to determine where to resume from.
What for?
>t's not hard to
LOOOOOOOOL
To prevent head-of-line blocking.
Of course using TCP has nothing to do with head-of-line blocking.
Only if you blindly do write(socket, fileptr, sizeof(file)), it starts to matter, but, you know, people did not start writing for TCP/IP yesterday.
If you have a decent protocol, it splits files (and texts as well) into chunks, and sends them in prioritized order, text chunks having higher priority than binary chunks. The server re-assembles the chunks then.
The fact that such a simple way of multiplexing data is a discovery for XMPP fanboys is very revealing.
But it's easy to end up with low throughput, because you limit how much unacked bulk data you send, because you don't want to overqueue bulk data and not be able to immediately write interactive messages.
And it's easy to end up with high latency, because despite the limits above, you queued too much and interactive messages need to wait. Or you sent bulk data and there was a burst of packet loss and new data can't be received until the missing packets are resent and received.
> people did not start writing for TCP/IP yesterday.
And yet you find resuming uploads over http to be too hard?
>And it's easy to end up with high latency,
And yet the OS is doing just that when you open two TCP streams. It packs them into ethernet frames and keeps track of the acks. Only the OS does not know the context, so it prioritises both streams equally. (Unless some evil magic is involved.)
>And yet you find resuming uploads over http to be too hard?
Me? No. Xmpp developers? Apparently yes, as it's neither in any of the XEPs nor in any of the clients.
Of course there's prioritization options on socket APIs since forever (DSCP), but usage is limited, because it requires coordination and coordination is hard.
Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times. If you're lucky the chat connection didn't see any of the loss, and even if you're not, you've likely got a smaller queue on your chat connection than your media connection.
This also doesn't prohibit client driven prioritization --- when connecting, get caught up on chat before processing media. Maybe pause media while sending chat messages or if server acks are slow. Server driven prioritization is hard, because probably your http media servers don't have information on the chat connection --- but if a client stops reading from the media connection, the server will get the message eventually.
Well, again, it's merely an implementation detail. You could just open a second TCP connection using the in-band IM protocol, not some external HTTP. Or make your IM protocol UDP based.
But really, that's what quic was designed for, which is also not new.
> LOOOOOOOOL
? I've done it a few times. It worked fine on Nokia S40 and the version of J2ME they run can't even seek backwards in files.
It's a three part recipe:
a) send a request to upload with whatever auth you need, some stable identifier for the file and the file size. Ideally a nice checksum to confirm the file was not corrupted in transit; TLS should protect you, but I've seen things, a 32-byte sha256 checksum offers protection from a lot of things.
b) if the upload is unfinished, you'll get a url (or whatever) to post to and a starting offset (0 on the first time); if the upload is complete, you'll get a download url to send to your correspondent. (or an error like file too big, try again later, go away, whatever)
c) upload the file to the url. Maybe get a status if the upload finishes and you're still online to receive it. Status could indicate error or a download url. If you timeout or get a retriable error, go back to part a.
It's not rocket science or anything. Maybe it would be hard to get done in the XMPP ecosystem, but it's simple enough to do with any stack that's got big enough media that resuming uploads is relevant.
Now tell your girlfriend's mother to do the same.
On Android, I've never had any connection issues with either Signal or XMPP, so I would say that this problem is solved.
The thing about these notifications seem to be that apps that regularily get notifications from the the app get them reliably.
This is based on the observation that xmpp-contacts I'm in regular contact with seem to get notifications quickly and contacts I talk to less regularily not so much.
Sadly this dependent on apples whims
Also huh, I've been to Yerevan but never heard of Radio Yerevan jokes until now, that's pretty funny.
Same for media, it just works, you should try it? And funny you mention WhatsApp, it's effectively a dialect of XMPP running on a fork of ejabberd.
The Telegram client leads in every aspect by far. It's just easy to use.
[1] $ curl -s https://updates.signal.org/desktop/apt/dists/xenial/main/binary-amd64/Packages |
awk '/^Package: signal-desktop$/{p=1} p{print} /^$/{if(p) exit}'Also, I would not advise to add proprietary repositories as you grant them root access to your system which is against security practices such as the principle of minimum privileges and defence-in-depth.
I really like Telegram, it's the closest thing to what I wish IM should be, but still it's so far from it and to be honest its apps suck now trying to shove the "premium" features down your throat. I prefer using TelegramX (or Forkgram to still be able to see statuses)
Frankly speaking, Telegram is also shit. It just sucks a bit less than the rest of them.
This is what the FOSS services kinda tend to glaze over, SO MUCH of my personal and professional life is managed by chatting with services that are represented as bots in a chat.
What does the Telegram Bot api have, what Matrixes API doesn't have?
Bots are, on a platform level, presented as bots. It's not a courtesy rule of "please name bot accounts as _bot" or something.
You can do actual payments with the API, it's possible to earn money with a bot.
Custom inline keyboards you can use to operate bots instead of parsing commands like ye olde IRC bots. Like a home automation bot could reply with a list of rooms when you ask for a room temperature, then you just tap the button for that room.
Closed source, walled garden yadda yadda, but it just works.
Discord bots are a bit less robust, but can still do a ton of things "natively" Matrix bots can't.
1) With OMEMO, each client/device of a user has its own key, but there is no trust relationship between them or some common master key. I can verify and trust counterparty key, but then counterparty adds another client/device and i have to do manual verification again.
2) Group chats do not have group key, but one encrypt messages for each group member (pairwise), which means that everybody has to do key exchange with everybody. This does not scale, especially with 1), which means that even if membership is stable, new keys may appear. Some clients (Conversations) do not allow to send message to a group when they do not know some keys of some group members/devices, which is fail situation that happen pretty often.
2) That's where MLS¹ kicks in, essentially a new encryption scheme that's suitable for large group chats. There are projects² for its adoption in XMPP. I have no strong opinion about MLS, but I tend to think that it's more for the gimmick and having XMPP score well in random protocol benchmarks: if you've got such a large groupchat that OMEMO doesn't scale anymore, you have a groupchat with so many participants that it is only private in name and more or less encryption doesn't matter.
Sorry to be a pain, and with full recognition that I don't probably represent the majority, but I care about emoji reactions, reply support, ability to send & receive images and video, ability to see that video later, a reasonable set of access control features for group chats; reply-privately for group chat messages, and a dozen other things. And I care about not making someone download an app I've never used that talks the protocol, because I can't support them getting started. So a single cross platform UX is needed too before you can hope for traction.
Some of this is UX; some requires infra (eg. storing & forwarding media)
But to produce a "complete" messaging tool is harder than it looks and the protocol layer, while important, should not be the focus of discussion. There needs to be a holistic conversation that includes UX, infrastructure capabilities, branding, and network effects.
And it can't just be an optional feature in some clients, it needs to be something that just works everywhere.
Some use cases like business chat actually can't have E2EE, and that'd probably be Jabber's foot in the door because no other open solution has taken off in business, and businesses don't need so much network effect. This was a main Jabber use case back in the day too.
Make decent official Jabber clients, called Jabber and not something else like a bird, and server that implements the right spec with default settings. Now is the time, you can AI slopcode entire native apps if you want to. If someone really want to put extensions on a server, it's called XMPP and isn't compatible with Jabber; the X stands for excommunicated.
So, Cisco Jabber (now Webex)? You can't really compete against Microsoft and its bundling strategy. Even Slack is dying, unfortunately.
Webex can't be run on-prem, can it? I vaguely know about the old Cisco Jabber.
But you're right, the best competitive advantage of Teams is that it's "free"[0]. But that's about it, it's objectively bad in all other ways - anyone claiming otherwise has Stockholm syndrome and is in denial =)
Just the fact that, as a consultant, it's literally impossible to connect to multiple teams workspaces with the same client. The OFFICIAL Microsoft advice is to use Private Browsing in separate browsers...
Meanwhile on Slack I can just hit cmd-number and swap between clients. AND we can have shared channels over instances, like $vendor having a support channel on our Slack with their actual people with their own accounts.
[0] If you have M365 anyway.
Thankfully we use Slack where I work. The cross-org channels are cool.
Probably the Facebook chat is still XMPP internally even today.
https://www.process-one.net/blog/whatsapp-facebook-erlang-an...
If reviving XMPP is the battle you want to fight, best of luck but I'd encourage you to take a step back, face reality, and spend your energy on something with a better chance of success. And I say that as someone who used Jabber all day long 20 years ago.
For anyone paying attention, it should be the clear "safe bet" for what to try next/keep at hand somewhere.
but what ?
All we need is for a billion dollar company to build their own implementation and XMPP can be back once more. Exactly the way it gained popularity back in the day.
This is (to me) a big issue when using bots, for example. I need a non-ambiguous trigger for them.
I wanted to self-host a chat service where I can control my bots and AIs, but with no native tagging functionality makes it really janky to work with.
Discord and Telegram Just Work, but I'd much rather have the whole thing in my control.
Buzz[0] seems nice, but for one person an 5 bots having a full-ass postgres+redis+s3 setup seems completely bonkers.
The fact that 25 years later it's still being described as two completely different names with a slash between them ("Jabber" SLASH "XMPP" ) makes me pretty confident it's not coming back.
Don't get me wrong...I'd love open, interoperable chat that everyone is reachable by as much as anyone, but it's important to recognize when one approach isn't working and why it isn't.
There's a lot of lessons to be learned from the death of jabber which I outlined in a post back on 2019: https://larrysalibra.com/the-death-of-jabber/
It _works_, but the limitations of Jabber without 42 different optional extensions became apparent pretty quickly.
It's a little like asking "Carburetors: How do we gain traction?"
XMPP is not a well designed protocol in many ways. There's no particular reason for anyone to care.
My most-used messaging app is SMS because if you have a cell phone, you have SMS. Example this morning is I made a connection with a handyman recommended by a friend. Text message, and it just worked.
So what's the compelling reason to move to Jabber? Why should my 92-year-old mother use it?
I am using Whatsapp and I MUST have some SMS app in phone, if your app will support SMS and on top of that will provide Jabber/XMPP messages over data I will be more than happy to replace my SMS app with your app.
Sadly seems there is pretty much NONE apps where dev would bother to implement support for simple SMS. And if your app doesn't support SMS I am not going to install any other obscure app since I need SMS to receive 2FA codes/delivery notifications/doctor appointments, etc.
Signal used to be liek this until 2022, not many people used it, but at least you could use it for SMS and if there was Signal contact you could use their network, in their wisdom they removed SMS feature, so Signal was removed from phones of my whole extended family since we need SMS and not going to have extra Signal without SMS, if anyway majority use Whatsapp.
It'll ensure the wrong people don't end up in your chats.
For use on mobile, run it in a tmux or screen session and reconnect over ssh.
Businesses want to analyze internal company messages. Chat protocols / platforms need to make inroads with one of the core audiences for online chat. Those are largely taken right now, (slack/teams business, discord gaming/oss). If it is personal/small group messaging, several E2EE options already exist, though I understand Jabber/XMPP is trying to be in the chatroom genre
chat is network effect, so a chat app needs to answer "which network of people" if the creators about traction?
I can easily imagine a future where people switch between three messengers, one for general communications with acquaintances and family, one for work, and one for their social circle. Many already do this with social networks.
The way businesses handle this is by installing a certificate they manage and deploy to their proxies.
I would rather want a feature, like Telegram has, where you can set auto-delete of all messages older than N days.
Everyone who has to, say, prove that they kept paying their rent for a year without interruptions. You sent a photo of your money transfer from the bank app to your landlord, and he sends the "received" screenshot from his bank app.
Moreover, a lot (even most) people make personal notes by sending messages to themselves.
Most people trust the police. This might not be justified in your state, but most people still do.
"Where were you during the event X? I was very far from the place you are interested in, as can be proven by this photo, sent to my grandma, have a look at the history in her phone."
If message history is important it should be backed up on the receiver's end after decryption, like WhatsApp and signal do.
The beginning was complaining about Signal! What is XMPP needed for then? Also, doesn't seem like many people have friends on either of them.