This Year in Matrix
matrix.org
matrix.org
- Synapse is very resource hungry even for a small server
- Synapse creates a gazillion of TCP connection and keep them open, it was a real problem for my ISP router (SOHO router for 1gbit/s) and it took a very long time to debug, be sure to limit the number of TCP connection on the server
- Synapse is hard to get a good idea of the health of the server, the federation can break without warning
- If the federation broke for some reason, when submitting messages, they will show as delivered, this was one of the major problem for us
- Encryption is very cumbersome to get right for non technical users, there should be a server wide option to make it "easier but less secure". Even for me, it was hard to get rid of a warning popup about some device error.
- Group VOIP is hard to setup, it should be as easy as discord to be viable.
- Webhook is really hard to setup too, I want to be able to right click a channel and "copy webhook URL" to be able to post to it with CURL (discord has that)
- Globally it is very slow when joining federated rooms, it can take days to sync 500+ people room
In the end we just use discord, it's free and do the job. Of course they can read our messages and this is a problem for me as I am for privacy and encryption, but it works for everybody, it is friendly and easy to use.
I will keep a close eye on matrix and try to adopt it again, but right now it is not usable for us.
Joining any larger federated room took days or did not complete at all. And then the messages were also delayed for quite a while. That must be the stupidest design I have seen.
If it was not so messed up, Matrix probably would have become huge by now.
I agree with others in comments, devs should have probably improved the federation and written the official server in some compiled language and provided it as a single binary.
In terms of the language: it's not Python's fault at all. Instead, Synapse baked in a bunch of naive assumptions (being originally a prototype) which we have been steadily fixing... while also adding features, which pulls in the opposite direction. Given a choice between a featureful server that's resource-heavy (which we can then make go faster later) versus one which lacks the features to compete with centralised alternatives, we chose the former.
I just think that having a single binary would improve adoption. I like projects that provide a single binary. Without the need for docker, or setting up complex environments.
It still lacks features compared to synapse but I'm happy with the performance. Notifications are sorely missing but I think the PR to add them is finally ready to merge.
Give it a year or two and you'll have a full featured matrix home server that can run on a raspberry pi.
> gazillion TCP connections
We’ve been running a Synapse server with quite a bit of federated rooms, even larger ones like you mention. Never noticed a single issue like this that I recall.
I wonder if the reason is that we’re doing whitelisting, instead of federation with everyone by default. The whitelisting is quite liberal; I’ll add a server as soon as I’m in a conversation thread with someone from that server, or just want to see their avatar. Sometimes when I’m waiting for something else I’ll just scroll through a room and add a handful of servers. Essentially this is my proxy for only federation with “legit” servers and avoid being DDoSed.
> there should be a server wide option to make it "easier but less secure"
Disagree. Sure, UX can be improved, but compromising here defeats the purpose of why we’re doing this in the first place.
Another one of mine:
- E2EE seems complex to implement and very few of the clients implement it fully and properly.
This is the one point I think needs to be resolved before Matrix can be ready for wider adoption. It’s kind of pointless without E2EE, or if Element is the only client that can work for all rooms and messages.
> Disagree. Sure, UX can be improved, but compromising here defeats the purpose of why we’re doing this in the first place.
The thing is, as I ran my own server in my office, SSL (with no E2E) is already a good privacy. And even with that setup, the server users got "harassed" by the UI asking them to sync devices keys.
For many users, a single prompt about security is enough to discourage them.
I think Element is the only client doing this. So if you don’t care about E2EE, there are several options.
My point is, this is something that should be addressed in the client UX, not on the server- or protocol level.
Should have mentioned: I think my approach works fine for single-user or small-closed-community servers. Nobody should do this on servers with public rooms and/or open registration as it will lock out minority servers and cause further centralization.
Use conduit.rs or dendrite. I use conduit and it takes a couple of MB ram even while I’m in large rooms
They work fine, although I’m not sure if discord bridge works
[1] https://matrix.org/docs/guides/introduction#how-can-i-get-my...
Fundamentally they've 'lucked' into a 'nice problem to have' type situation here, and I get how annoying it is - I was freenode staff during a period where the irc<->matrix bridging was unexpectedly popular enough to be a 'nice problem to have' for everybody involved and the author of the blog post we're discussing probably remembers some sharp words on my part during the process - but frankly on balance I'm mostly impressed.
synapse is quite a mess and there are a lot of hidden footguns if you just run a server - this is just the tip of iceberg to keep the postgres database somewhat sane: https://levans.fr/shrink-synapse-database.html - lot's of other issues in the issue-tracker where you can just scratch your head.
bridges are all subtly broken - the xmpp bridge is horrible and broke in so much interesting ways that I'm not going to touch it ever again - telegram works okay most of the time, irc-bridge also have some warts - but it's easy to criticize from my chair and probably unfair to talk so negative about it here but it's often buggy and broken for edge-cases - it works most of the time pretty okay but it's quite a mess to get a mental model for the code and so it's difficult to debug things.
moderation/spam/etc.pp is all hackable but not there out of the box - it looks and feels like mostly quickly hacked up nodejs code that at least for us exploded in all kinds of interesting ways. https://github.com/matrix-org/mjolnir writing 3tb of logs in a few day and eating memory like crazy for instance. You have to babysit it and there is no simple ui for anything.
So it's powerful but requires quite a bit of dedication and patience to get right. It's a full blown distributed system and often state is all over the place and once you make a mistake it's difficult to impossible to get that thing do work correct again without starting over.
But there are so much promising projects that I'm confident that these issues will be resolved and it will only get better but in my experience it will break badly on all kinds of edge-cases - the mentioned xmpp-bridge created usernames that can't be deleted via the http api for instance. someone bridged 1000 channels via our telegram-bridge and there is no code to remove those channels - you have to code something up in python for yourself. irc bridge kicks you after 30 days idle because they can't handle the connections - freenode (before the takeover) said it's not them - maybe single threaded nodejs is not such a good idea for that.
Could I do it any better and delivering? Probably not. But except some adventure and if you want to deploy it for an org carefully test any assumptions you take for granted. It's cool but it's also kind of quick'n'dirty in a lot of ways. Still better than anything else I'd use it over any megacorp messenger anytime but maybe don't switch your family yet.
But for using it you don't have to care - and there are great projects like https://github.com/spantaleev/matrix-docker-ansible-deploy that solve most of the problems out of the box and mobile clients and web clients and E2E crypto also works really well.
In my opinion, the biggest problem with the current Matrix platform, especially if you self-host, is the server implementation. It's not buggy, but it's just awfully slow. The "dpt of ping" [2] section at the bottom of the weekly Matrix updates show that five seconds of latency between servers (measured by basic ping bots) is far from uncommon. The latency is still significant in the non-synapse version of the graph, but it's much better already.
With the speed new MSCs are accepted every month I don't think any alternative server stands a chance at catching up to Synapse any time soon, and I don't know how I feel about that. Yes, Matrix is an open platform with an open source spec, but it's effectively impossible to develop an alternative backend implementation with the way development is currently done, unless you've got a lot of developers and money to throw at the platform.
I'll probably look into Conduit again in a few months, hopefully the project will keep up it's impressive pace so that it's useful for my purposes.
[1]: https://cactus.chat/
[2]: https://matrix.org/blog/2021/12/17/this-week-in-matrix-2021-...
Given, I have a small blog (ie low load on cactus), but I'm in many (15+) medium sized (100-1000 members) federated chats and it's just fine on a day to day basis.
There are some limitations tho:
- I can't join large rooms (2000+ members and whatever number of servers that entails on average)
- Initial sync (first time log in on new client) is crazy slow - syncv3 will hopefully fix this
My definition of not being resource hungry is being able to run on a 1gb, 1 vCPU VM - however you and me might have a difference in that opinion
My instance of Synapse definitely can't join any matrix.org chatrooms, no matter how hard I try. Even rooms with a few hundred participants are awfully slow and leave scrollback state a broken mess. When I tried to join the main Matrix channel, both of the CPU cores I had dedicated to Synapse were pinned for at least half an hour, even after I tried to cancel joining the room!
I hope sync v3 lives up to the hype, but I'm wary. Synapse's Pythonic base seems ill-fitted for an instant messenger backend in my opinion, and no amount of protocol trickery can work around that.
It's not that I think the 1GiB is necessarily that excessive, but Dendrite and Conduit are blowing Synapse out of the park in terms of performance, to the point that I'd almost rather see the Matrix folks dedicate their development efforts more on bringing Dendrite up to speed than to expanding functionality.
The main thing that bogs down Synapse is if users join massive federated rooms and you end up replicating huge amounts of traffic onto your server. So we’re working on that too via Fast Joins and better-than-full-mesh federation for P2P.
I once installed Litecoin Core (on Windows 10) on my desktop PC. I had Comcast (branded as Xfinity) broadband, whatever speeds my region and apartment supported. I had a 1TB monthly limit before throttling. I lived in the east SF bay area. I downloaded Litecoin Core software at the beginning of the month. I got dial-up speeds for the rest of the month. I got a 5x average bill that I had to negotiate down. Nobody on my server even cared about the federated rooms, it was just me trying to give them more flexibility.
Allowing federation with Matrix has the same issue, and I know it's being addressed, but I want to hammer this home. Self hosting means normal, poor people, too, not just EU government entities and Upcoming Big Tech partners. I know you know this, but every advocate needs reminders here and there.
The initial sync issue is only a problem for large accounts which have joined many rooms. Cactus comments creates a guest user for non-logged-in commenters, and an empty guest account is pretty much instant to load in my experience.
You can go to my cactus comments instance (running on aforementioned 1gb machine) to test out the speed https://karmanyaah.malhotra.cc/tech/2021/06/website-things/
Back then I rented matrix hosting with modular.im (now called ems) once I noticed that I did use Riot/Element on a daily basis, and the latency of matrix.org servers was too big for my taste, combined with the lack of online status display.
Now that dendrite is starting to finally mature, I will probably switch to hosting it myself to save some costs.
Main things stopping me are the possible headaches around domain names and federations: How hard is it to move to a different hoster once I have an instance running?
I read in the documentation that things can become problematic if I don't carefully backup parts of my dendrite matrix instance. What would happen if I (or rather my hosting provider) lose my files and have to setup my matrix instance from scratch, will my domain then remain blocked on the federation, even when I still have control over the domain?
It's not that I don't do backups, but I'm curious what would happen in a worst case scenario.
You'll definitely have problems with missing rooms, missing user encryption, the lot of it. I think it should be possible to re-create and re-import signing keys onto a new server, but I wouldn't rely on it. You'll probably end up with broken clients for ages. You'll also definitely run into loads of clients 404ing on your server, trying to download media that no longer exists. Personally, if I'd lose access to my server, I'd just start over with a new domain and get my backups in order that time.
If you have backups you can recover, you can get everything up and running painlessly by pointing DNS to the new server and waiting for DNS propagation to finish.
As for moving between hosters, there's a tool for that: https://ems.element.io/tools/matrix-migration; it's a tool that'll invite your new account into every room your old account was part of, transfer profile pictures and such and optionally kick the old client from rooms after migrations. That'll only help if your new account is on a different server, though.
I'd also like them to name homeserver.yaml in the Synapse Debian package to something akin to homeserver.example.yaml so that updates don't make me have to diff changes to figure out what I need to add (since accepting the package version overwriting my homeserver url is unacceptable). But that's neither here nor there.
I will be trying to migrate to a new server at different host/data center in about a week - hope it's as simple as the base guides I bookmarked make it sound.
Definitely would love an official guide and some kind of check things script maybe - but we'll see, maybe it's not too hard - dunno yet.
Problems with synapse are true, but it was a good compromise to push out an early implementation so things could get rolling quicker with getting people to actually use the clients.
Just when I get used to some issue/missing feature in the clients, they seem to fix it.
And in general I trust and know that decisions with the spec are being made carefully.
I use it for an intimate chat server with never more than 8 users, so I can't speak to some of the problems. I can just say the trajectory itself for Matrix is strong. A good effort to fight slack and discord. Wish all the discord servers people use were matrix instead.
This is one place where I strongly disagree.
Almost all features have "We will add encryption later" which is not a way to build a secure messenger. Stickers, polls, profiles, spaces, relations, reactions... they all start with and unencrypted version with the promise that "encryption will come in a later MSC". They should really take "secure" out of the tagline until they actually add E2EE for all of these features. Basically all clients allow users to use these insecure features in encrypted rooms with no warning at all.
And I constantly see hacky quick fixes get accepted because "we need to do something", "no clear use case for $X" (even when there is) and "N other MSCs depend on this so we'll merge it in and deal with a proper solution later". For example the relations MSC just got merged with support for only 1 relation per message even though it is clear that editing replies would be a problem with this system.
I love Matrix and hope it does in fact unify messaging but I think "carefully" is not the way to describe how the spec is progressing. More like "shoved in whatever direction is needed to implement the newest Element feature". Maybe that is for the better, we do need a featureful protocol to get users after all, but personally it irks me the wrong way. Especially when security and privacy are ignored "for later".
While I’d have loved us to have infite manpower getting all the security features in from day 1, we had to bootstrap and we prioritised usable apps over optimising for a perfect spec. And judging by some of the rest of the thread, we didn’t so well even then; it would have been even worse otherwise.
> While I’d have loved us to have infite manpower getting all the security features in from day 1, we had to bootstrap and we prioritised usable apps over optimising for a perfect spec.
I get the decision. But it leaves a sour taste in my mouth that Matrix advertises itself as "secure" but almost any user is leaking sensitive information without knowing. I suspect that 95% of users would be surprised if you told them about the number of things that leak in E2EE rooms. Probably the better approach would have been to skip pushing E2EE as the default, and disable all insecure features in Element for E2EE rooms (or have an explicit opt-in). The marketing team would hate this approach but it least it is honest and safe.
Sending things unencrypted that users have every reason to think are encrypted is extremely irresponsible. And I assumed, as I think is perfectly reasonable, that everything in an E2EE room is in fact end-to-end encrypted. I'm very disappointed to find out this way that it's not (and won't be for the foreseeable future), and I don't see any justification for failing to making users aware of that.
All that said, I love Matrix, will continue to use it every day, and am hugely rooting for the success of the project. Thanks for all your hard work on it!
Breaking this expectation is a massive red flag, particularly because Matrix is often chosen with this expectation at the core of the decision.
Why couldn't the client simply count the number of reactions?
I can't imagine it'd be a performance issue. After all, if you didn't have reactions you'd have a bunch of in-band "+ 1" comments; if such a thing would bring down the server then I'd assume you'd have much bigger problems.
Edit: clarification
Is this not the case for e.g. Signal reactions?
The text/content of the edit itself is actually encrypted, you can check in Element by viewing the event in de/encrypted form
Some of the most exciting features for me are:
- Dendrite: we really need some efficient replacement for Synapse
- Sync V3 and Fast Joins: it will erase the feeling of slowness when I open the web client
- MLS: It will help to scale E2EE groups [1] (remember the 256 limit in WhatsApp groups ?)
- P2P: We received this year a first glance of what a network of Bluetooth Low-Energy devices can do with the Apple Find My network. A true p2p instant messaging system is not just a dream anymore.
- Metaverse, file storage: I have no idea if they will accomplish any of them, but having a new non-profit working in the field is always welcome.
[1] https://i.blackhat.com/USA-19/Wednesday/us-19-Robert-Messagi...
Congrats to the matrix team, really great work. However, I'm a little confused by the excitement around "social login". Why is it such a great thing? Aren't you just giving someone else power to shut you out of Matrix? We hear so many stories about lives ruined because of arbitrary (often automated) account locking that I just don't understand why anyone would risk using social login at all.
It's more like, "Welcome to our federated decentralized service. You can log in with your Facebook account if you want to." Which doesn't sound like a bad thing to me.
More choices for users is usually a good thing. If users' choices have bad consequences, that's not Matrix's fault. Though maybe Matrix should warn people that if 3rd-party cuts them off they'll be locked out of their Facebook account too—choices need to be informed!
The hex of these types of graphics is that they don't actually say a lot at all beyond "one thing is slightly bigger than the other", or "the thing is a bit faster now".
There are things that are easier to convey with a picture than with words (like the how much bigger the sun is compared to the earth, "really really big" doesn't quite cut it), but this is not such a case.
But I have a great pain with that bug: https://github.com/vector-im/element-web/issues/469
Everytime I’m fighting with myself to not be sarcastic when speaking about this issue, but I want to say I have hard time believing that Element could we a main communication platform for anyone without this being fixed. It drives me mad, and I have just two people to communicate with!
My previous work used a system which queued notifications for delivery. The logic was something along those lines:
- send notifications for urgent items, e.g. incoming calls, immediately to all devices - if there are no active devices, send all notifications immediately - otherwise send to all active devices st internally along with testers.
I ended up tweaking this; it took us a few tries to get it right. I'd be flabbergasted if Facebook messenger, iChat, discord et al haven't done something similar.
1. https://github.com/vector-im/element-android/issues/4147 and also some other
Often when my phone is left behind somewhere in the apartment, my SO will ask me to do something with those notifications, or just will move my phone away from her (and/or my tablet).
BTW it seems that other communication devices did it right. I just tested Google Chat, and I remember that I did not have this issue neither with Slack.
I really would like to move to an e2e encrypted discord alternative. Maybe someday I'll be able to do that with matrix, when ether it gets easier, or I get smarter.
[0]: https://github.com/spantaleev/matrix-docker-ansible-deploy
I would love to finally be able to use my employer's chatrooms with my real name, the animal rights group with my one pseudonym and the techie chat with the other one.
But yes, I think OP is asking to be logged into two accounts and see the rooms/contacts/notification from both in the same application. This is not currently supported in Element and I don't think it is on the near roadmap. But some other clients do have multi-account support.
In practice, though, I'd be impressed if you could even find three different circles that are all on Matrix. I haven't found many people who are willing to switch yet over, anyway.
Kind of funny, but my experience over the last year as been the opposite. Probably because the kinds of communities I am interested in are focused around various FLOSS projects, but it seems like everywhere I look the "Contact Us"/"Get Involved" page for a project linked to their Matrix room(s). Anecdotally, the momentum that Matrix has in the FLOSS space is reaching a critical mass where it will almost be the de-facto communication tool (I am starting to be surprised when I see an IRC link instead...)
Outside of the tech world, I agree with you. Adoption of Matrix is tricky, but honestly it does not seem much worse than trying to get your friends to move the MMS group chat onto any other platform (WhatsApp, GroupMe, etc). Only extra challenge is helping them understand what "homeserver" they should use.
Also, arewep2pyet.org pretty please.
Whatever I step to, I will need to get my extended family onto it, and that is a big enough hassle I don't want to repeat another move..
I saw that, E2E notwithstanding, your home server has plaintext access to all traffic, which makes me leery of using cloud servers.
The other big roadblock is that I don't know of any client that can manage multiple accounts. I would have to use e.g. fluffy for personal, and element for work.
And I found what was labeled a Signal-Matrix gateway, but could not figure out how to set it gu he. If you are not accessible via Signal, your existence fades.
I've been toying with the idea of providing dynamic extensions to Matrix rooms by sending a WASM payload as an event.
That WASM could then provide new UI elements, interactions, interpret events, etc. This could be used to implement polls (I know it's coming, but that's an example) or whiteboards inside existing clients.
It's just an idea, not sure how it would work in practice... But if anyone wants to have a go at it, be my guest!
I was thinking of something that only does I/O with data contained in the room it's in. I think the UI would be the hard part, but it could be doable with a few HTML tags and an extensible UI.
For instance: add an "insert poll" button, or an "add voice message" button. Or an "add sticker"/emoji/gif, etc. Currently every client needs to build this individually. Users could pick the features they want in their clients.
Moreover, not every plugin needs to be third-party. A "trusted store" could very well be maintained.
Thank you so much for making analytics opt in. :)
Last I checked we needed a git account o upvote things or something - but even then I'm not sure we'd get a change in the roadmap / prioritization... I've considered hiring some peeps to make the code and offer it up to the community ti merge it or whatever. This and a few other important to me/my users, but really kind of superficial things in general that needs to attention imho.