Modern messaging: Running your own XMPP server | Hacker News Reader

Modern messaging: Running your own XMPP server

codedge.de

224 pointscodedge148 comments

Last time I tried this I couldn’t even find an iOS client that supported reactions. Hate on the complexity of Matrix all you like, but at least when you ask your family to use it, they get a modern experience and don’t feel like they’re giving up everything vs every other messenger they are used to.

I recently tried to move people over, and the iOS situation is bleak. It seems like notifications don't work at all in any app, which is a big problem for chat. Also, one of them (I think Siskin?) doesn't display sender name/avatar in group chats, so you can't tell who sent what.

There's also some weird behavior with how E2EE works, which is causing me problems in group chats. Initially it worked, but now people are unable to read some messages with errors like "this message was not encrypted for this device"

FWIW, everything works great on Android with the Conversations app, and also I admit I haven't really RTFM'd too much so it may be my fault as a lazy admin.

Edit: this is with ejabberd btw

Monal works great on iOS in my families experience. They've never missed a notification and E2EE (OMEMO) never failed. We are using the conversations.im server however.

My experience with XMPP server hosting is that the defaults are not what you actually want, and many bare minimum features (not even talking about reactions) are bolted on.

For (repliable) notifications on iOS, you need mod_cloud_notify server-side.

I’m using Monal and it’s decent. Agree reactions are a big thing missing, but notifications etc have been flawless which was a harder requirement for me

when my family used matrix for a couple years the e2ee features would randomly fail to decrypt history and would present my non technical family with that "match these emojis to unlock your account" screen all the time.

between that and the unwanted features (federation, spaces) that aren't useful in a texting context, no way to search message logs, and a server to maintain.

...

I went back to RCS

fwiw, the vast majority of Matrix encryption errors got fixed by 2024 (https://www.youtube.com/watch?v=FHzh2Y7BABQ etc).

Search on mobile (in Element X) is on its way and is making good progress: https://youtu.be/Q6NSmptZIS4?t=1297 from a demo from a few days ago.

Finally, self-maintaining servers now exist (https://element.io/server-suite, complete with AGPL distro: https://element.io/server-suite/community)

(On the other hand, notification reliability on Android is still a known problem on Element X for some users that we're working on.)

Noticed Facebook Messenger added E2EE which has made it way more glitchy and annoying. Used to be solid for things where you don't need super security.

And there's no way it's more secure anyway, cause your secret is just a 4-digit pin.

There isn't a single XMPP client for iOS that properly follows the iOS HIG and isn't ugly or unusable. There are very few clients that are even fully featured. The XMPP landscape on macOS is even more bleak.

Isn’t this true of most open source software in iOS?

The intersection of developers who want to develop on and for a proprietary walled platform and also want to work on open source clients for open standard, descentralised protocol must be pretty small.

As an open-source XMPP project, we tried hard to find iOS developers willing to help with development (as volunteers, or even paid) and it's just so hard to find people in this intersection of interests.

If anyone reading this thread has iOS dev skills and cares about improving open-source messaging on iOS, feel free to get in touch.

I'm not even against the walled garden, it's just that developing on iOS is really painful compared to, say, web.

Oh and I have made an iOS app that uses XMPP before, but it wasn't general purpose chat.

And alas web push notifications on iOS remain very critically buggered beyond measure. https://webventures.rejh.nl/blog/2024/web-push-ios-one-year/

I get that not everyone would accept a web-app replacement, but it sure would be nice if iOS wasn't actively obstructing the possibility of some folks using a web-app messaging service.

> Last time I tried this I couldn’t even find an iOS client that supported reactions. Hate on the complexity of Matrix all you like, but at least when you ask your family to use it, they get a modern experience and don’t feel like they’re giving up everything vs every other messenger they are used to.

Now if only it actually worked consistently and reliably.

I'm not being snarky, the app is flakey as hell. Reactions are nice and all but pointless if they don't have the send a message part down.

> Now if only it actually worked consistently and reliably.

It does. I use a self hosted Matrix server for chat with friends (multiple servers federated, even!) and it is rock solid.

I dunno, it works great for me. I've been self-hosting a Matrix server for 5 years and I'd not describe it as "flakey" at all.

XMPP lacked good, modern clients, so folks designed a terrible protocol, and then made modern clients with this kind of feature for that protocol.

There’s an extra step that could have been skipped entirely there and we’d be better off.

It would be cool for clients and protocols to be separated by an abstraction layer, it might change the n times m to n plus m but I suppose it's difficult

what do you mean?

client - server - server like?

We have precedent of exactly this.

Pidgin implemented support for various networks under an abstraction layer and had a unified UI for all of them.

XMPP implements this via gateways. I used to talk from XMPP with folks who used MSN or Yahoo on their end. We have gateways for Telegram, WhatsApp and Signal nowadays.

The Nokia N900 used the Telepathy framework to expose a local D-Bus messaging service which itself connected to various networks (SMS, MSN, XMPP, etc). The UI was a single unified one for all these networks.

> designed a terrible protocol

Have you looked at the protocol? Have you built things in both? I have, and I think the Matrix protocol is pretty good. The goals are different. It's not a send once and forget kind of thing, like XMPP, Signal and email. It's about keeping your chats synced on your server, and with any other server you happen to be federating with. It's Discord or Slack, except federated. That's not easy, and it's not simple, but it has huge advantages too.

> It's not a send once and forget kind of thing, like XMPP, Signal and email. It's about keeping your chats synced on your server, and with any other server you happen to be federating with. It's Discord or Slack, except federated. That's not easy, and it's not simple, but it has huge advantages too.

And huge disadvantages. I don't understand why everything has to be centralized/ routed through an intermediary. Well I do understand it, modern big corps wants to be that intermediary for various reasons, but thats a business reason not a technical one.

at least when I help someone set up XMPP it doesn't have a single blessed client that tries to convince them to create an account on matrix.org

iirc, matrix started as attempt by Amdocs (in same org that I worked at) to give to telecoms their own messaging client that will be better than SMS in order to compete with OTT clients that they saw as "unfair" and "eating their revenue"

hence, matrix by definition was born due to business reasons and not technical.

you may be mixing up Amdocs UC (which i ran) which was explicitly OTT-messaging-for-telcos… with Matrix, which is what the UC team did next. The point of Matrix is/was to replace the PSTN with a decentralised alt on the internet - so everyone (including Amdocs) could then go build cool stuff on top.

isn't matrix is direct continuation of amdocs uc with same team, your including ?

nope, Matrix has zero code or philosophy in common with Amdocs UC (which was proprietary, centralised, unencrypted XMPP + SIP held together with HTTP). Instead, after we gave up on trying to persuade telcos to roll out OTT messaging apps built on the Amdocs UC stack, we started Matrix as an new project to instead try to disrupt the PSTN (decentralising & disrupting it much as cryptocurrencies try to disrupt legacy payments networks). The reason Amdocs funded us to do it was in case we were successful enough that Amdocs could then sell Matrix solutions alongside PSTN solutions in the long term future.

In other words, this wasn't really a "hey we need to sell messaging apps to telcos" play - it was a long-term R&D experiment, more like Bell Labs funding UNIX.

ok. now i remember things better let me rephrase it. matrix was initiated by/inside amdocs. it (given that it was amdocs) was meant to be sold to telecoms to compete with OTT offerings (open source wording, etc - amdocs was adding it to everything back in this timeframe as it was trendy).

I was sufficiently "high up" in organization to hear this pitch. For record I said that they (telecoms) won't buy it and I didn't like project technically

Later when Amdocs saw that it's a no go they span you outside and later cut the funding.

> matrix was initiated by/inside amdocs.

it was initiated very much by me & the UC team while inside Amdocs.

> it (given that it was amdocs) was meant to be sold to telecoms to compete with OTT offerings

in the 5-10 year horizon, yes. And indeed eventually (as Element) we have ended up with a bunch of telco customers. However, this was not the immediate goal at the time - it's not like Matrix was created to improve EBIT for Amdocs UC; it was a long-term R&D play.

> Later when Amdocs saw that it's a no go they span you outside and later cut the funding.

It was the opposite. Amdocs saw that Matrix had legs - e.g. Ericsson started selling Matrix-based solutions (stuff like https://matrix.org/blog/2016/11/23/when-ericsson-discovered-...) - but also saw that it didn't fit inside Amdocs. The whole idea of Matrix is to be an open standard for everyone, just like XMPP or SIP or HTTP. For it to succeed, it obviously couldn't live inside Amdocs.

So, they agreed to both cut funding and span us out; we then raised funding independently and set up The Matrix.org Foundation as non-profit to look after Matrix for everyone, and separately set up Element as for-profit to fund our work on Matrix. It's not exactly been a smooth journey (see https://youtu.be/lkCKhP1jxdk?t=363 for my FOSDEM talk trying to explain the route so far), but I can confidently say that Matrix was not borne out of trying to scratch an immediate business itch for Amdocs, but instead a longer-term experiment in building something better for everyone (including Amdocs).

But what do I know :D

>it was initiated very much by me & the UC team while inside Amdocs.

in other words by amdocs.

ericsson starting to sell matrix based solution while amdocs not selling any (or none of traditional amdocs clients like AT&T, Comcast, T-Mobile, etc expressing interest in buying one), it's the definition of "no go" and "no fit" for amdocs. At this timeframe, amdocs could sink tens of millions of dollars into open source project if they thought that they will get some money back. ONAP will be prime example of this.

internal presentation in amdocs that were circulated on VP+ level most definitely talked about scratching immediate business itch for Amdocs. It was talking about how telecoms are upset that OTT messaging killing SMS and profits and that matrix is attempt to give to telecoms something to compete with it. I said that idea that telecoms can get this market back is bananas, but management had shiny new toy to play with and they were excited about it. Hence they allowed it to run till the moment that they saw that no profit can be done there.

There are technical advantages to a dumb client. The more you outfit an XMPP server with basic things modern users want like message history and push notifications, the more state and responsibility moves to the server.

Disclaimer: I have a lot of XMPP experience but have never used Matrix

take a look at the way https://delta.chat solved for that very modern/looking/feeling - just email.

> I don't understand why everything has to be centralized/ routed through an intermediary. Well I do understand it, modern big corps wants to be that intermediary for various reasons, but thats a business reason not a technical one.

centralized? no

If you sign up for messaging on let's say Signal, do you really want your client to talk to Facebook, Google and dozens of other services?

And do you want the users of your chat service depend on "random" other services ? Not being able to access chats, media, because an outage or even shutdown of a random service over which you don't have control?

My idea is basically person to person comms. Why can't they just send messages directly to each other? I guess mobile can't do this and relies on polling (i.e. you can't expose a service running on the phone for security reasons). And when the group gets large enough, beyond say 3-4, sending messages to each recipient gets unwieldly

Check out delta.chat thy have solved super simple p2p messages

https://delta.chat/en/2024-11-20-webxdc-realtime

Directly doesn't work over the internet, because the internet doesn't work like that. You'd be restricted to bluetooth, wifi direct or similar.

If you allow for forwarding in between, there are p2p messengers. Messages are stored - if at all - using store & forward techniques. They can use the internet, but also bluetooth and other communication technology to transmit messages. The most popular ones would be Briar and Jami. Matrix P2P exists at an experimental stage as well (moving servers onto devices).

The problem with them you give up convenient things, even though there are often are workarounds. For instance, plain p2p you cannot communicate with offline devices, but store and forward techniques exist, either storing messages on random people or dedicated store and forward servers.

Battery drain, lack of reliability. It's just extremely simple, quick and reliable to send messages to a defined server and receive Push Notifications via push service.

Want a reliable storage of your messages (encrypted) such that you can always access them when you wish? well...

> Directly doesn't work over the internet, because the internet doesn't work like that. You'd be restricted to bluetooth, wifi direct or similar.

That is how "the internet" has worked forever.

People have been running servers on their own home machines, for a long time. If you really want you can setup a webserver/sshd/etc. on your own machine and access it from the outside if you don't care about DNS and static IP. Will also probably need to port forward at the router, though.

Its less common nowadays due to the cloud, and security issues, etc. sure. And I'm not sure to what extent its possible over mobile.

But for two people who are on PCs/Macs, it should definitely be possible. And it would be much lower latency because you don't have to bounce it off another server.

XMPP is a client/server model too, that needs to store messages for some configurable amount of time. What distinction are you trying to make here? There are very few peer-to-peer messengers.

Yeah peer-to-peer would be my idea. Send directly to each participants device, no third party involved, at least for the messaging part. So one less vector for attack. You'd probably want a central service for determining who's online.

Wouldn't work well for more than a few people, but not every conversation has group sizes that large.

Also very difficult because:

- direct connections are really hard (Tailscale built a whole company on solving this one problem)

- even Tailscale can't establish direct connections without a coordination server

- even if you can reliably, and always, establish direct connections, it doesn't matter if someone is offline

- push notifications don't work without a server, on Android or iOS, so even if you're online, you're out of luck (won't ever get a new message because there's no push notification to tell the client to connect, and you can't just leave a TCP connection open forever on a mobile phone)

My take is that it's fine to have a server in the middle with E2EE. That's the whole point of E2EE.

You're assuming negligence or bad intent when you could be looking for a legit reason behind this.

i bet you could write an iOS client with support for reactions before matrix implements custom emoji

I’ve been running Prosody [0] for many years now. Don’t ask me why I chose it over another server. I used to have a jabber.org account that I used before that, already back when I could use jabber to communicate with gchat and facebook messenger users. Alas, nowadays, it’s just my wife I use it to communicate with :(

[0]: https://prosody.im/

Prosody is sweet. I forget why I switched over from ejabberd, but it it was quite reliable and had such a huge breadth of extensions it supported. Ejabberd has been intimidating to setup but really wasn't hard, but Prosody was a snap.

Prosody was a breeze to set up and it worked pretty well for me. I set up a simple bot to get notifications on my phone. The whole set up works great until you get to the notifications. Super unreliable. I believe that's an issue of the iOS client I am using.

I still remember when Google (and Facebook?) used XMPP for their chat functions. You could log into any XMPP client and chat with people using Google infrastructure.

Good times, I feel old now.

Yeah I used to use Pidgin to chat with people on Facebook! I miss those days.

Yeah, I had iChat logged into 4 different things, one of them being AIM.

One potential issue with XMPP is the default port is commonly blocked on public wifi.

There is an nginx-xmpp to proxy it, but it is archived. https://github.com/robn/nginx-xmpp

For that reason http based protocol just seems much easier on the network or something that can be easily reverse proxied without extensions will be easier to self-host and have wilder internet connection accessibility.

At least Prosody implements BOSH (xmpp over http) and communication over Websocket.

https://prosody.im/doc/setting_up_bosh

https://prosody.im/doc/modules/mod_websocket

But I never tried it myself and from a quick search the popular non-browser XMPP apps/clients don't seem to use it.

with chat control on the horizon, they should probably consider implementing it

I actually enabled BOSH on my Prosody setup.

For reference: https://www.someodd.zip/phlog-mirror/xmpp-server.gopher

BOSH still has some interesting trade-offs even today. It can help with some NAT headaches and rides over plain HTTPS. I like this old post:

https://metajack.im/2008/07/02/xmpp-is-better-with-bosh/

Curious who here uses BOSH in production and/or WebSocket (RFC 7395) these days.

Not for 13ish years but we built the chat feature in a social networking platform with BOSH. Looked just like FB Messenger, really. The thing never took off, so I don't remember any talks about outside federation, I think it was not on by default.

The default ports are often blocked on such networks (public wifi, corporate firewalls, etc.), but also often open. 5222 is used by e.g. WhatsApp, 5223 is used by e.g. Apple push notifications. So it's not as bad as it could be.

But it's also totally possible to run XMPP over 443, and this is a feature of many popular XMPP deployments. If you're self-hosting, there are some guides for different deployment approaches here: https://wiki.xmpp.org/web/Tech_pages/XEP-0368

> 5222 is used by e.g. WhatsApp

I left in 2019, but when I was there, WhatsApp used port 5222, but the client would try port 443 if port 5222 didn't work. After it had tried those enough, it would try on port 80 with HTTP wrappers.

Really, the right model for a public service is what AOL did for AIM. Listen on all the ports. Clients should try on the 'proper' port, then 443, then 80, then random permutation. Skip certain ports because nobody likes it if you probe on smtp, smb, or chargen (etc)

Yeah. I'd be surprised if Apple didn't use similar logic. XMPP can do multiple ports (a deployment can specify the recommended order in DNS). I've not heard of an all-ports deployment, but it does sound like an interesting experiment :)

Apple has a bit of an easier time, because they can (and do) lean on cellular carriers that restrict access to their push servers. And they have 17.0.0.0/8

But they say port 5223 and 2197 with fallback to 443 [1]. Google says 5228-5230 and 443, with a couple outliers [2].

[1] https://support.apple.com/en-us/102266

[2] https://support.google.com/work/android/answer/10513641?hl=e...

You don't need any third party modules and can proxy based on ALPN (https://wiki.xmpp.org/web/Tech_pages/XEP-0368#nginx) thus running everything on port 443. Note that ALPN is not encrypted AFAIK but public wifi services don't care.

"Since 3 years the European Commission works on a plan to automatically monitor all chat, email and messenger conversations.12 If this is going to pass, and I strongly hope it will not, the European Union is moving into a direction we know from states suppressing freedom of speech."

If this come to pass, there will be two approaches:

1. People will not share anything important online, only in person

2. Every friends group will have a more technical guy/girl who will ran chat infrastructure for them

Interesting article though :)

3. Normal people accept surveillance and techies are tracked down / silenced.

Given our current surveillance state with social networks, I don't see (1) or (2) as real options.

I hope not; I hope there is a culture change; but, hope is not a strategy so it's better to build alternative tools and learn how to use them

Ejabberd has been around since forever, and people have been saying that forever. Why would it suddenly happen now?

One thing I do agree with is that collaboration tools have to come first. We've become unable to do anything without a company/boss, completely atomized. Three or four programmers/designers who hang out should be able to do anything together. Put them in a downtown office with some MBA prick alternately yelling at them and kissing their asses, and they can build empires that they get to share 5% of.

Yeah that made me laugh, aint no way average people are going to care. They already assume things are maximally bad. They assume things aree completely public and still choose to overshare. They are past the point in caring and I dont blame them, its near impossible to actually take serious without going down a schizo rabbit hole.

On #2, even though I and most of my friends/family are in the US, likely going to offer such to friends and family... I'm about to move from a /29 to /28 subnet to run a few extra services on my hosted server.

I've got a nice mailu config and wanting to expand with Nextcloud (or alternative) and likely xmpp services... I mostly use a pretty light host VM and docker compose configuration to make up/down/backup/restore pretty smooth... I'm not currently running across multiple servers, but do want to be able to have a slightly more consistent config... I've got a combination of Caddy and Traefic on the different servers for TLS and all my apps are /apps/appName/(data|docker-compose.yml) on the server(s). Which keeps my maintenance chores relatively simple from a couple remote ssh commands and rsync.

Mostly been a bit lazy in terms of getting this all done.

Been running jabberd for probably >10 years. Horrible experience (for the users) and terrible clients. Sometimes you'd just appear online while not sending/receiving anything.

I switched to Nextcloud Talk after Skype shutdown and migrated all my family there. They love it. We have a private cloud, we can share photos and other files, great mobile support... The only issue at the moment is relatively long delay before receiving a message (up to 30secs?) since I've been too lazy to setup redis.

I see these comments a lot from people who have very old deployments and didn't keep up with changing best practices. The users of these deployments also tend to be using out-of-date software such as Pidgin to access their account.

There is zero reason for long delays or lost messages in XMPP.

I'm running ejabberd in my home network and together with Conversations (Android app) it is my solution to have all devices report their status to my phones, tablets and pc. Easy to upgrade (when used with Docker), never complains.

It's a bit the same feeling like RabbitMQ, which I'm no longer using that much, but I also always thought that it is a reliable software. I've wondered if it is related to them being written in Erlang.

Take a look at delta.chat Your server can be a traditional email server you set up or one of their optimised devocot, smtp relays: https://github.com/chatmail/relay

The clients are great (they have reactions!)

there is even a really good app store associated : https://webxdc.org/

I actually messed around with Delta.chat this past weekend for the first time.

It's kinda cool. If or when, I need to setup new communication needs, I might use this. (Currently do not have a need for any)

I love XMPP. I've never missed a notification with it and I wished I switched back to it sooner rather than sticking with Matrix, I do wish it gets more love these days. I've been tempted to make my own desktop client and integrate VoIP with something like Mumble to achieve a Discord like experience with voice channels, but that's way beyond my pay grade.

Movim and Dino do multi-party jingle, aka voice and video group calls. Maybe you can contribute to improving it ;)

Jingle is P2P with an optional server middleman for firewalled connections only no? I haven't seen any support for actually hosting a voice server with XMPP, only just allowing clients to figure it out themselves. I'll give it a look either way.

Indeed, Jingle is for establishing connections, P2P when possible, but there are lot of extensions around it.

I've proposed a specification for SFU hosting (check https://bloggeek.me/webrtcglossary/sfu/ if you don't know what's a SFU), and wrote a component based on the excellent Galène SFU, as well as client implementation (in Libervia) as part of a NLNet/NGI grant (https://nlnet.nl/project/Libervia-AV/).

XMPP council (disclaimer: I'm a council member for the current term) asked me to some modifications and to re-propose, which I'm about to do. I couldn't find the time so far (cause I'm working on ton on stuffs), but will go back to it very soon.

To sum-up: this is very much worked on.

And Libervia (disclaimer: I'm the lead dev). It also implements SFU including components (based on Galène), but I'm reworking design on this part.

Also note that Libervia is using a backend/frontends architecture with a D-Bus API, you can use it to make your own frontend with any language you like.

I went looking for more info on Libervia and noticed your site is down.

Yes, I'm having a DDoS attack these days, no idea why somebody would do that to my small server. I've deployed counter measures so the site is more or less usable, but the attack is still going on.

Good stuff. I run my own ejabberd server and it's been great. Installation is relatively straightforward, configuration isn't terribly convoluted, and it "just works" once up and running. Now my server doesn't have a ton of load on it, as it's used mainly for development / experimental purposes at the moment, so I can't say much about it's scalability or anything. But generally speaking, I installed it, did the initial setup, and other than adding/modifying users, I've rarely had to think about it since.

I've been homelabbing my own XMPP setup for a while. I went with Prosody and I enjoy its Lua flexibility and clean config model.

For anyone curious, I documented the full process (including IRC bridge + more!) here:

https://someodd.zip/phlog-mirror/xmpp-server.gopher

I also covered audio/video calls using a companion STUN/TURN server:

https://someodd.zip/phlog-mirror/xmpp-server-video-audio-cal...

Curious how others here are handling federation and mobile clients these days. I use Conversations (F-Droid) and Gajim.

For TLS I would recommend https://github.com/pingooio/pingoo instead which has automatic certificate management and is really easy to setup.

Never implemented this by myself, but I've read in the past this post on using Prosody, by the guys helpong the Low-Tech Magazine blog to become a self-hosted solar-powered site.

https://homebrewserver.club/category/instant-messaging.html

If you plan to communicate with people not in the techsphere, Signal is probably the best bet to convince people to switch to.

Signal is a walled garden and might even leave the EU soon. Of course if that happens, XMPP clients will also be harder to get, but at least I will always have full control over my XMPP server.

Signal is not even private since it requires a phone number

Its still private, just not anonymous

If you're looking for something that's a little less hassle and has some very sane defaults, try https://snikket.org/

Has anyone done a security review of their source code?

It is simply Prosody + Conversations + Siskin [1], so I'd say that many people have had their eyes on their code.

Specific security audits would have to be searched for, though.

[1]: https://snikket.org/open-source/

They're really lagging on their mobile client updates.

You can use upstream apps instead of their forks, as I do. Zero problems.

I'm missing the actual self hosted guide where I can use my own hardware instead of a VPS.

Can't you install docker compose on your own hardware and follow this? https://snikket.org/service/quickstart/

Hellz yeah. I'm at ~10 active users on my XMPP server and going strong. Finally something that won't unilaterally lock us out if we don't update every week, and we can chat from using any OS. We really do not care about "reactions" or stickers or all that other shit people complain about being lacking. "It doesn't look nice enough".. We just want to talk in private with a secure protocol that lets us actually use whatever we have, up to and including no mobile device at all.

Any idea what kind of server we’d need to handle 40M users if it’s just text chat (no files or audio)?

If you’re running a single machine, you’ll be limited by the number of available ports. It’s a TCP limitation and nothing to do with XMPP. You’d need a cluster of XMPP servers to handle 40M users. Even for just text. Port limits are port limits.

40M users don’t all simultaneously send messages, or does XMPP need to sustain an open connection to each user?

Yes, you need to have open connection to receive messages.

Or push notification support (which is the same, but basically the OS (Android/iOS) is the thing holding an open connection :) )

You're thinking of outbound port limits. Inbound connections all come in on the same port and there's no port-related limits there.

The real limits are going to be on hardware. If we want 40M concurrent connections, the server will need a few hundred to a few thousand gigabytes of memory.

Unless you're doing multicast or anycast, there's a port bound IP handshake that happens. You have your listener (your server port) and the connected TCP/IP client (client port, on the server machine). You're limited to 64533 clients (0-65535 but the first 1000 are reserved). If there's a way to get more client on a single machine, I'm all ears - but that's baked into IP protocol. TCP/UDP doesn't matter. It's a limitation of IP.

Assume 10% of 40M users are active at once. That's 4M clients to deal with. You would need 62 servers (and probably a few for cache) to handle the load. But those could be small core cheap servers.

The uniqueness tuple for IP is (source IP, source port, dest IP, dest port). You're limited to 65,535 connections from the same two IPs on the same port, but that's not relevant to XMPP which uses only one and some transient things for file transfer and such. At worst having that many people behind one NAT will be a problem... which at this scale could be an actual problem, but there are still solutions (multiple ports being the easiest, and the fact this cluster will probably be on multiple public IPs anyhow).

See https://ats1999.github.io/blog/65535/ or similar pages.

You should be able to deal with 4 million clients on one server in 2025; we did 2 million on one chat server in 2012 [1]. I can find documentation of 2.8M in Rick Reed's Erlang Factory presentation [2], page 16. That was part of a clustered system with chat connections on one group of servers and several other groups of servers for account information, offline messages, etc. Also, the connections weren't Noise encrypted then, and there were a lot more clients in 2012 that didn't have platform push, so they would try to always be long connected ... it's not very hard to manage clients that just need to be pinged once every 10 minutes or so.

But this was with only 96G of ram, and dual westmere 6-core processors. You can put together a small desktop with consumer parts that will be way more capable today. And if you run everything on a single machine, you don't have to deal with distributed system stuff.

When I left in 2019, we were running on Xeon-D type hardware, with maybe 64 GB ram and IIRC, only doing about 300k connections per chat machine. Chat scales horizontally pretty well, so run a couple machines of whatever, find your bottlenecks and then scale up the machines until you hit the point where it costs more to get 2x of bottleneck on a single machine than to get 2 machines with 1x of the bottleneck. I suspect, if you can get quality AM5 servers, that's probably the way to go for chat; otherwise likely a single server socket would be best; dual socket probably doesn't make $/perf sense like it did 10 years ago. If you get fancy NICs, you might be able to offload TLS and save some CPU, but CPU TLS acceleration is pretty good and there's not a ram bandwidth saving from NIC TLS like there is for a CDN use case.

IMHO, getting a single server to support 4M clients shouldn't be too hard, and it should be a lot of fun. Mostly all the stuff that was hard in 2012 should be easy now between upstream software improvements and the massive difference between CPUs in 2012 and 2025. The hard part is getting 4M clients to want to connect to your server. And trying to setup a test environment. Elsewhere on this thread, there's a like to the process one blog from 2016 where they ran 2 M clients on a single server (m4.10xlarge (40 vCPU, 160 GiB) with a single same spec server as the client load generator; the impressive part there is the client load generator --- I've always needed to have something like a 10:1 ratio of load generators to servers to load down a big server. And they managed to get that to work with all their NIC interrupts sent to a single cpu (which is not what I would have recommended, but maybe EC2 didn't have multiple nic queues in 2016?)

> If there's a way to get more client on a single machine, I'm all ears - but that's baked into IP protocol. TCP/UDP doesn't matter. It's a limitation of IP.

As others have said, the limitation is the 5-tuple: {protocol, SrcIp, DstIp, SrcPort, DstPort}; if you're a server, if you hold SrcIP and SrcPort fixed, and for each dest ip, you can have 64k connections. There are a lot of dest ips, so you can host a lot of connections, much more than you can actually manage on a single system. If your clients are behind an aggressive CGNAT, you can actually run into problems where there's more than 64k clients that want to connect to your single IP and port ... but you can listen on multiple ports and address it that way, and most aggressive CGNATs are only for v4; if you listen on v6, you'll usually see the client's real ips or at least a much more diverse group of NAT addresses.

If you listen on all the ports, you can have 4 billion connections between you and any given IP. That's not going to be limiting for a while.

[1] https://blog.whatsapp.com/1-million-is-so-2011

[2] https://www.erlang-factory.com/upload/presentations/558/efsf...

You can put multiple IPs on a computer. (Or VMs, LXCs, etc. which would each get an IP)

Now what size of machine?

(I don't really care, and the original question does have a whiff of malicious intent, but scaling discussions are sometimes interesting...)

Here's a blog post on scalability from the front page (well, sort of, see last paragraph): https://www.process-one.net/blog/ejabberd-massive-scalabilit... This includes not just textual claims about scalability but a lot of hooks you can use to followup on, like a reference to the Tsung benchmarking tool you can use. If you're asking about this for serious reasons, at this scale you're obviously running your own tests anyhow. You may also want to speak directly to Process One about this because it sounds like you're at a scale where you should probably be looking at paid support anyhow.

I'm not necessarily endorsing this, just giving it to you as some first stabs at answers and some ways to follow up.

If there's anyone reading this from the ejabberd project, note that the link to this on your front page under "Massively Scalable" is completely broken; that links to "blog.process-one.net" but that domain is completely dead, so it doesn't redirect to the link I gave above. (It's also part of why I posted this; my post here is not a "just read these docs" because I had to do non-trivial work to find them.) I had to pull that out of archive.org. Should probably check for any other links to that domain too.

Thanks, I've forwarded it to their chat channel.

Thanks for noticing, it should be fixed now.

I'm past my edit window now so I can't remove the claim, but I will verify and agree that it is fixed now. If you (the reader, not Metalhearf) previously visited the home page you may have to manually reload to bust the cache to see the correct link(s) now.

> Any idea what kind of server we’d need to handle 40M users if it’s just text chat (no files or audio)?

IRC.

Spin up multiple VM/Docker/whatever instances with an IRCd daemon, add each IRCd as a leaf.

If you want 40M connections, on FreeBSD, the maxfd count is physpages / 4; you could edit the kernel to change this, but are you really going to serve 40M connections with only 16k ram for each user? If my math is right that puts you around 640GB of ram. (40M * 4096 * 4 = 40G * 4 * 4 = 640 G), which is totally doable on a server socket. Probably you don't have everyone simultaneously connected, but probably you also need more ram per connection, so kind of depends. By the time you find 40M users to connect to your server, you should be in tune with your hardware needs ;)

Maybe Tigase (https://tigase.net/; https://github.com/tigase/tigase-server/) would work for you?

It does offer almost-linear horizontal scalability.

(disclosure, I work for Tigase)

> support clustering (for high-availabilty purposes)

That reminds me of one idea I had back in the day. You see, not everyone has the skill nor time to set up their own server, so clients rely on 3rd party servers. But sometimes these servers either experience intermittent connectivity issues or just die out altogether, which means users now have to set up second account and re-add everyone, while also scattering chat history across multiple accounts. What I'd like to see is some way of linking accounts across multiple servers, so that:

- message delivery could have transparent fallback (it gets delivered to me@bar.com if me@foo.com is unavailable)

- you don't have to add and authorize multiple accounts

- rosters and chat histories also get synchronizes between the two

A lot of that could probably be hacked around just on the client side, and I don't have good answers to questions like "what happens if I want to unlink two accounts for some reason".

(And going a bit off-topic: I also wish I could use multiple email addresses while registering in other places, because email provides too can fail, close altogether, or even just ban you for no good reason while also having crappy bot-driven support@.)

Thanks! Same for me, it just works. Even regular updates run smoothly with ejabberd. I also do audio calls with it, and it is just a pleasant experience.

Does anyone use ejabberd inside a wireguard network? I'm hoping to set it up on the main server, and connect to it with aTalk on Android - but I'm worried ssl is going to be a pain if it's not exposed with a proper domain name. I don't know if aTalk will accept a self-signed certificate (or even better, just use non-ssl with an address and port without a domain name)?

ejabberd should work fine, it won't care. But you're right - whether apps will allow you to accept self-signed certificates can vary. Some are strict and don't allow bypass, others may just issue a warning prompt that you can dismiss. I haven't personally tried aTalk.

I was thinking of setting something like this up on a Tailscale network, and figured I'd just get real certs for the servers in question using DNS challenges, which I've been able to do with my tailnet (driven by Headscale) for a while now. But even if not, if your root cert is in your device's trust store, then an app would have to go out of its way to only trust well known CAs.

I have this setup, but I haven't tried using a self signed cert. I just have a public domain with the DNS pointing to the ejabberd internal IP. Setting up LetsEncrypt with DNS auth isn't that difficult, and I'm using Digital Ocean for their free DNS API to automate renewals.

Although the big issue with this is that clients need to have wireguard enabled at all times, otherwise they can't even receive notifications. It's kind of a PITA for non techies to understand, and also kind of a PITA for techies who may already be using wireguard for something else, as Android/iOS don't let you have multiple VPNs active at once.

It may be better to expose my ejabberd service to the internet directly. Even if it gets hacked, messages are E2EE at least.

You should be able to use a DNS provisioning through Let's Encrypt assuming you're on a supported host for your DNS based provisioning. Traefik may even have an easy button option depending on your configuration.

> I assume the server is going to be run under xmpp.example.com and you all the following domains have been set up.

> [multiple A records]

"A" DNS records may be used for a fallback, but SRV records are the primary way to configure those [1, 2]. Also some of those can reuse an existing domain name, and some may not have any DNS RRs, but only be used as an internal JID.

> ejabberd is a robust server software, that is included in most Linux distributions.

Prosody [3] is another nice and popular option.

> Install from Process One repository

> Install from Github

Both ejabberd and Prosody are available from regular Debian repositories as well.

> Make sure the fowolling ports are opened in your firewall, taken from ejabberd firewall settings.

A port range is also needed for TURN, to use for relaying. And there is a typo.

> Clients I can recommend are Profanity, an easy to use command-line client, and Monal for MacOS and iOS.

Among relatively feature-rich and user-friendly ones (quite polished, supporting more recent standards, including voice calls with DTLS-SRTP, OMEMO), there are also Conversations for Android, Dino for Lignux (GUI), poezio for TUI (though that one has no voice calls). Setting converse.js (a Web client) may also be convenient (and done rather easily, at least with Prosody).

[1] https://www.rfc-editor.org/rfc/rfc6120.html#section-3.2.1

[2] https://xmpp.org/extensions/xep-0368.html

[3] https://prosody.im/

As much as I like the idea of XMPP, I don't have good experiences from interacting with it. Neither clients nor protocol/server level.

I've written multiple parsers along the way, back in the days when there was nothing else and more recently for use in very constrained embedded contexts.

I don't know how much has changed, but it was more complicated than I would have wished, seemingly designed more to check theoretical boxes than for ease of use.

I was also part of a project where the backend was implemented as a bunch of services communicating via XMPP, custom server etc. And it was a total mess, we spent a lot of time on manual intervention just making sure messages weren't dropped.

Is this a common experience with XMPP or did I just hit all lemons?

I have been running a family and friends XMPP server on a cheap VPS for almost 10 years and the only downtime we had was when the datacenter burned down (OVH, true story).

The problem is not the server. The clients all suck.

I couldn't get iOS notifications to work properly when I tried to set up ejabberd.

Ended up using Hey, and it works pretty well, I guess, but a rails PWA is a little heavy duty for my taste, I would prefer XMPP

I cannot say, that I have problems with it. I use monal on iOS and whenever someone mentions me, I get a notification.

For people who want an easier approach, both NethServer and Cloudron have this packaged as an app for people who self host. The Cloudron one is not official yet and is a community developed one.

XMPP has very nice server implementations, and the protocol is OK regarding complexity.

But the clients are lacking. On linux there is gajim that is "okish" but it lacks calling capacity with mobile clients. On mobile there is conversation and derivatives on android which are "nearly there" and monad on iOS.

Globally, the main lacking features are:

- voice and video calls cross platform

- gif integration, tenor/giphy/imgur...

- fast sync (if you open gajim after being offline for a week, it takes ages to catch up)

I kind of view the lack of gifs as something positive? Like, it forces one to use actual words and think out a response.

Plus, you're not leaking all the tracking associated with those widgets.

However, I understand that people have come to expect having fun experiences in their IM clients, and that usually requires reacting with animated GIFs.

I agree but it's a required feature for many people.

> On linux there is gajim that is "okish"

In my book it only got worse. Way worse. Sure, it looks more familiar to those who is used to iMessage/Whatsapp/Telegram, but I bet it would still look quite alienating for that audience. And for those who remember Gajim 1.x, disastrous UX doesn't outweigh introduction of reactions, history syncing and whatnot. Last time I checked it a couple of months ago:

- It was not possible to close group chat without leaving MUC.

- I had to constantly open separate search dialog to write to someone who's already in my roster but is not in the list of active chats.

- Speaking of active chats: what annoys me the most about modern IM clients (and that includes Gajim 2.x) is that list of chats is sorted by the last activity date. I don't even know how Whatsapp/Telegram users live with this, I got so fed up with hovering my mouse/finger over one chat and tapping it only for the whole thing to reorder in the last jiffy and opening something completely different, I just dropped my account from one of those centralized mass-market services altogether. It is that annoying.

- It had lots of smaller warts like nickname autocompletion requiring way more key presses (especially after someone mentions you).

Maybe try Movim, it has everything listed https://movim.eu/ :)

(I'm the author)

the only problem, as usual, is the network effect

what about

- single point of failure of rooms (server creating the chat room)

- media access via 3rd party servers

- gossiping encrypted message access across multiple devices

- encrypted audio-visual (group) communication

- good clients for iOS

- reliable server-side storage of messages (of all(!) chats)

- realistic verification concept for e2ee with multiple devices

Not my project, but I was hoping to see more stuff like https://prose.org/

As others have pointed out, for iOS Monal is good.

> for iOS Monal is good.

Monal looks like something that would be good in 2005, not 2025. There is no way you'll convince people nowadays to use it unless you force them to.

Maybe for friends and family? But then, if their phone might be compromised by the state...

And telling friends and family they need to setup a Jabber connection to talk you will make you "That guy/girl who...". And if the idea spreads, then talking to n people requires n accounts on n servers. At least Pidgin (last time I used it, which was last decade) supports such a configuration. I wonder if the mobile apps can do the same.

> And if the idea spreads, then talking to n people requires n accounts on n servers.

No, it doesn't.

XMPP is like E-Mail and Matrix a protocol which supports federation, i.e. a protocol which specifies how many service providers can cooperate, specifically forward messages to other service providers to reach their users.

Anyone considering wiring Agents to XMPP? Seems like a convergence of A2A and human chat, and supports all the human-interface pieces of agents like slash commands. You can build many topologies with XMPP, and there is a wealth of extant source code.

Of possible interest: https://xmpp.org/extensions/xep-0384.html

I've forgotten how much hassle installing applications can be since docker.

Here are the official docs for deploying ejabberd using containers: https://docs.ejabberd.im/CONTAINER/

Part of my thoughts... though if you're familiar with Ansible, the automation isn't so bad in that ecosystem. I mostly run my personal stuff single instance, so deploying /apps/app-name/docker-compose.yaml is my general approach to most things along with either Caddy or Traefik.

On a tangent, what’s the simplest and best self hosted discord alternative?

yunohost installations include the metronome XMPP service enabled out of the box. it's similar to prosody

I'm sorry but there's no way you can call xmpp modern messaging. Matrix with all of its shortcomings looks like 2050 compared to xmpp. Xmpp doesn't even have any half-decent mobile client!

Conversations is fantastic, not sure what you're on about.

Although material like this is extremely important, instructions like 'Fill in the IPv6 addresses accordingly.' are a roadblock to anyone who isn't particularly knowledgeable about networking. You could argue that those people should go away and learn, but not everyone has the time or frankly the ability to do so.

Take a look at https://snikket.org .

Thanks :)

Reply on news.ycombinator.com