Hacker News Reader

Comment by harvie | Hacker News Reader

Jabber/XMPP started to suck at the moment we've started using smartphones that keep reconnecting between wifi and mobile networks as i move around. relying on that single persistent TCP connection does not make sense anymore. This is huge pain with IRC as well... And no, i will not run server-side proxy irc client that is supposed to keep me reachable while im reconnecting.

Also sending media (or even rich text) never really worked as each client implemented it differently. While whatsapp just works...

View on HN

XMPP has brilliant session management, it reconnects more seamlessly than many other protocols on mobile. It literally reconnects (rejoins the same session within different TCP connection, not replaces it with a new session).

>Also sending media (or even rich text) never really worked as each client implemented it differently.

Everyone use HTTP upload nowadays, it works all the time.

>Everyone use HTTP upload nowadays, it works all the time

No it doesn't. Firstly, an out-of-band upload is ridiculous. Secondly, it doesn't support upload resumption in case the network is bad.

>Firstly, an out-of-band upload is ridiculous

It is a bit, true, but the alternatives are worse. XEP-0231 (Bits of Binary) Base64-encoded data inside the text message is more ridiculous idea. Direct P2P connections, which were widely used before HTTP upload, are mostly not working in our day.

>it doesn't support upload resumption in case the network is bad.

There's a draft for HTTP for it: https://httpwg.org/http-extensions/draft-ietf-httpbis-resuma...

Re: P2P, it's kind of making an impromptu comeback. If you want some trivia, IIRC, the developer of Conversations was on some trans-continental flight few months ago and realized that XMPP works just fine on the free onboard internet plan (being considered by the firewall as legitimate "social-usage"), except for HTTP upload. So now, when your HTTP upload fails, Conversations asks if you want to try again with P2P, so you can send your images for free from the plane :-)

>but the alternatives are worse

It's not that "alternatives are worse, it's that xmpp is a broken protocol, which doesn't include support for anything, even such a basic thing"

>There's a draft

LOOOOOOOOL

XMPP was not designed for human chat only, and XMPP foundation promotes it as an IoT communication protocol, like MQTT. Thanks to Daniel, the author of this post, we now have communication profiles which the IM-oriented servers and clients follow.

>XMPP foundation promotes it as an IoT communication protocol, like MQTT

Which means that it will never work for human messaging, due to all the IoT cruft.

>Thanks to Daniel, the author of this post, we now have communication profiles which the IM-oriented servers and clients follow.

LOL, he only implemented p2p transfers when it bit himself in the tail on a plane flight.

Conversations is the best of them all, but it doesn't mean that it is good.

I use XMPP almost exclusively for the last 10 years and don't see any major issues, not with the protocol, not with the clients. Everything is pretty reliable and works fine.

Yeah, me too. Even my mom agrees to use it, because she loves me. But it's a "worksforme" fallacy.

Having media out of band causes some problems but eases others. Having a separate tcp window for media and messages is pretty valuable. Additionally, if media is separate, you might run it in different locations than chat.

HTTP may not have native resumption, but it's not hard to do partial upload and resumption over HTTP, you just need agreement on parameters / a protocol to determine where to resume from.

>Having a separate tcp window for media and messages is pretty valuable

What for?

>t's not hard to

LOOOOOOOOL

>What for?

To prevent head-of-line blocking.

>To prevent head-of-line blocking.

Of course using TCP has nothing to do with head-of-line blocking.

Only if you blindly do write(socket, fileptr, sizeof(file)), it starts to matter, but, you know, people did not start writing for TCP/IP yesterday.

If you have a decent protocol, it splits files (and texts as well) into chunks, and sends them in prioritized order, text chunks having higher priority than binary chunks. The server re-assembles the chunks then.

The fact that such a simple way of multiplexing data is a discovery for XMPP fanboys is very revealing.

You can certainly chunk things and interleave/multiplex interactive and bulk data.

But it's easy to end up with low throughput, because you limit how much unacked bulk data you send, because you don't want to overqueue bulk data and not be able to immediately write interactive messages.

And it's easy to end up with high latency, because despite the limits above, you queued too much and interactive messages need to wait. Or you sent bulk data and there was a burst of packet loss and new data can't be received until the missing packets are resent and received.

> people did not start writing for TCP/IP yesterday.

And yet you find resuming uploads over http to be too hard?

>But it's easy to end up with low throughput,

>And it's easy to end up with high latency,

And yet the OS is doing just that when you open two TCP streams. It packs them into ethernet frames and keeps track of the acks. Only the OS does not know the context, so it prioritises both streams equally. (Unless some evil magic is involved.)

>And yet you find resuming uploads over http to be too hard?

Me? No. Xmpp developers? Apparently yes, as it's neither in any of the XEPs nor in any of the clients.

> Only the OS does not know the context, so it prioritises both streams equally.

Of course there's prioritization options on socket APIs since forever (DSCP), but usage is limited, because it requires coordination and coordination is hard.

Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times. If you're lucky the chat connection didn't see any of the loss, and even if you're not, you've likely got a smaller queue on your chat connection than your media connection.

This also doesn't prohibit client driven prioritization --- when connecting, get caught up on chat before processing media. Maybe pause media while sending chat messages or if server acks are slow. Server driven prioritization is hard, because probably your http media servers don't have information on the chat connection --- but if a client stops reading from the media connection, the server will get the message eventually.

>Even with equal prioritization, when you overuse on the media connection and that results in packet loss and congestion window reduction, chat messages don't have to wait for all the dropped packets to be resent over multiple round trip times.

Well, again, it's merely an implementation detail. You could just open a second TCP connection using the in-band IM protocol, not some external HTTP. Or make your IM protocol UDP based.

But really, that's what quic was designed for, which is also not new.

>>t's not hard to

> LOOOOOOOOL

? I've done it a few times. It worked fine on Nokia S40 and the version of J2ME they run can't even seek backwards in files.

It's a three part recipe:

a) send a request to upload with whatever auth you need, some stable identifier for the file and the file size. Ideally a nice checksum to confirm the file was not corrupted in transit; TLS should protect you, but I've seen things, a 32-byte sha256 checksum offers protection from a lot of things.

b) if the upload is unfinished, you'll get a url (or whatever) to post to and a starting offset (0 on the first time); if the upload is complete, you'll get a download url to send to your correspondent. (or an error like file too big, try again later, go away, whatever)

c) upload the file to the url. Maybe get a status if the upload finishes and you're still online to receive it. Status could indicate error or a download url. If you timeout or get a retriable error, go back to part a.

It's not rocket science or anything. Maybe it would be hard to get done in the XMPP ecosystem, but it's simple enough to do with any stack that's got big enough media that resuming uploads is relevant.

yeah, yeah, yeah.

Now tell your girlfriend's mother to do the same.

XEP-0198 is great for session management. But, Android and iOS don’t really want apps maintaining persistent sockets. WhatsApp solves this by leveraging the push infrastructure when the app is suspended.

XMPP also supports push notifications using Android and iOS infrastructure. But it also works with persistent connections, and the client notifies server when the app is not in foreground, for server to push only important data (such as direct message) to conserve battery.

XMPP also leverages the push infrastructure, because otherwise there is no way it would work on many modern mobile platforms.

I have no problem with that with XMPP. Conversations automatically reconnects after each switch within few seconds, so it is transparent for users. On IRC that is huge pain, true.

Conversations also supports Android's/Google's push notifications.

XMPP is more reliable than Signal on iOS. My wife regularly doesn't get Signal notifications for messages that I sent when she was in a marginal service until hours later when she opens the Signal app to send me something.

On Android, I've never had any connection issues with either Signal or XMPP, so I would say that this problem is solved.

Does XMPP reliably get push notifications on iOS?

My impression is that it does, but I daily-drive an Android phone, so maybe someone else here can comment?

Bit of a Radio Yerevan Situation. Not even the various emergency broadcast app reliably get timely push notifications.

The thing about these notifications seem to be that apps that regularily get notifications from the the app get them reliably.

This is based on the observation that xmpp-contacts I'm in regular contact with seem to get notifications quickly and contacts I talk to less regularily not so much.

Sadly this dependent on apples whims

I don't think Apple is the weak link here. Somewhere there must be a gateway between an XMPP server and the APNS. Maybe the server isn't handling it well. I do know APNS was super annoying to deal with before they added the HTTP API.

Also huh, I've been to Yerevan but never heard of Radio Yerevan jokes until now, that's pretty funny.

So, my understanding is that while XMPP has historically sucked on mobile, it's gotten better? Does anyone know about the current experience? I don't know if it's due to new XEPs or if it's just better client implementation...

If you're talking about the push notifications experience, then XEP-0357 solves this, but app developers have to host the infrastructure which the XMPP server calls back to and dispatches the notifications to Apple or Google. Last time I used Monal on iOS this was solved and worked well. On Android I think most apps still prefer to keep their own background connection open.

https://xmpp.org/extensions/xep-0357.html

> On Android I think most apps still prefer to keep their own background connection open

Many Android apps give the option of using either FCM (Firebase Cloud Messaging, Google's push service), UnifiedPush or polling.

Of note, Conversations and some other XMPP clients can act as UnifiedPush Distributors (i.e. they use their own reliable persistent connection to proxy notifications for other apps). Some mastodon (tusky, fedilab, …), matrix (fluffy chat, element x, …) clients use it¹

¹: https://unifiedpush.org/users/apps/

I've been using it for years, so has my family and recently a group of some 60 friends with whom we communicated about a large party ont of my friend and me organised. I made an invitation site with an instance of converse.js at the bottom of the page, pointed at an anonymous group on my (Prosody) server. We just mostly Conversations or one of its forks on Android - no fruit phones among us - and Converse.js on web, higher on the same server which hosts all our other services (mail, "cloud", media, etc.). Once installed and configured it is maintenance-free, lightweight and reliable.

That is how you gain traction: by having people use it, realise the user experience is comparable or in some cases superior to that of other messaging systems - the party chat thing I mentioned above being a good example - and then realise they dont need to hand over their data to an ad broker (Google, Metafacebook, the fruit factory) or the government (all of them through some "chat control" type law).

XMPP is so bad on mobile that this persistent connection thingy and the whole XMPP protocol itself are effectively what prop up every single push notification on Android and Nintendo switch. True, Android vendors made a habit of killing background apps for misguided battery optimization reasons (less the case nowadays), which XMPP taught itself to work around in 2014/2015.

Same for media, it just works, you should try it? And funny you mention WhatsApp, it's effectively a dialect of XMPP running on a fork of ejabberd.

This should be solvable, although maybe a little meh. Client developers on platforms without unrestricted background connections need to run a server for push notifications. And server developers need to accept and use a push url from the clients.

If I understand correctly, Google and Apple have a way for clients to request push tokens, but in order to use the token, you have to authenticate as the app owner, so a client unaffiliated with the server can't simply send a push token to a server and have that be used ... Instead you'd need that push proxy server.

That proxy server comes with concerns about reliability and privacy and etc, but if you do pushes without cleartext content (and you should!), the footprint is minimal and you gotta do what you gotta do.

I don't follow xmpp/jabber, but I'd hope there's already an XEP for this. It's a pretty apparent need for the last I dunno 15 years and it has a clear solution.

With the right agitation, maybe you could get sharable platform push tokens and skip the intermediary app developer pusg server, but that seems unlikely.

You probably also want something in the push system to help trace push problems. For small chat servers and smallish client push proxy servers, I expect minimal operations staffing, but when messages are being delayed because of push problems, you want to be able to let affected clients know that they need to be more agressive with periodic/background connections until push works again. You may also want to let users or the administrators of the proxy push servers know as well. Pushes might be initiated late on the origin, delayed between the origin and the proxy, queued for a long time at the proxy, delayed between the proxy and platform push, or delayed within platform push (including on device delays from Doze, etc).

You've got to be able to detect that so you can make adjustments and reduce delay. Messaging delay reduces user satisfaction.

> This should be solvable

Yes, and it has been solved long time ago already and is now widely supported by both servers and clients.

> I don't follow xmpp/jabber, but I'd hope there's already an XEP for this.

See XEP-0198 from 2004 and XEP-0357 from 2015.

> XEP-0357 from 2015

Taking a quick look, this one is marked deferred; I don't know what that means, but it sounds like it's not an accepted standard?. It requires the application push server to be an XMPP server (which I suspect is challenging for client developers, compared to having a https url that takes a POST or similar) and doesn't include a method for a client to request a test push to validate the setup or discuss feedback to the client for push failure.

This doesn't feel solved to me.

I didn't review XEP-0198 closely, based on perhipheral ecosystem interaction, I do think that one is solved.

Both Monal and Conversations support XEP-0357. Each project runs their own XMPP-to-Apple/Google gateway for push notifications. It has to be that way because the gateway server and client phone application have to be signed by the same developer account.

The client phone app tells your XMPP server which gateway to use (or maybe it's vice-versa, the gateway contacts your server. I forget). In any event XMPP servers like Prosody support this out-of-the-box.

Reply on news.ycombinator.com