Matrix.org homeserver: Severely degraded performance due to database issues | Hacker News Reader

Matrix.org homeserver: Severely degraded performance due to database issues

status.matrix.org

27 pointsjeroenhd20 comments

I find it very surreal that the first performance degradation on the matrix.org homeserver in about 6 months warrants its own HN thread - I guess it's a testimony to how reliable people have grown to expect the server to be these days! Fwiw, this was very much a partial degradation over about 3 hours caused by a pathological database query, and most people didn't notice it. The biggest issue was that to-device messages were delayed, impacting new E2EE sessions.

Eitherway, apologies for the degradation. Obviously anyone on their own server won't have been affected anyway.

> most people didn't notice it

I am on a fair amount of channels on matrix.org and can confirm, if it wasn't for this news article I would have never known.

On a sidenote, thank you for the work you all do!

In the end matrix has the issue that federation seems less widely used than it maybe should be. It was a concern I had when looking at it, and when I spoke to the author of conduit (rust matrix server) i made the point that easy deployment should be a primary goal. The existing python server seems non trivial to deploy.

I think that’s still true, I’d you want a distributed setup like that it has to be as close to trivial to setup as possible. Email has shown that complex setups cause centralized solutions.

The Python server is pretty trivial to deploy.

Set up Postgres, tweak a hand full of settings in the config, that's pretty much it.

Has been running without issues for half a year.

Just a single person deployment though.

> The existing python server seems non trivial to deploy.

This really isn't true (and hasn't been since Matrix exited beta in 2019). `apt-get install matrix-synapse` or `pip install matrix-synapse`, edit the config, and `synctl start` and off you go.

I launched my own matrix server for my company two years ago and I’ve never had to do a thing with it. It gets frequent use by ~10 people, so not huge, but it’s been quite easy to deploy and maintain.

I've installed several. It's fairly simple. The hardest part of setup was reading the quotes for the config files, because there were very many and not all were written clearly back then.

Lack of federating is a real problem, though. I don't federate. It seems too easy for a malicious user to overrequest data and screw over your budget server, and I've heard of it leading to a lot of resource usage nightmares.

I have issues when I join large rooms with my tiny private instance. Does anyone know if there is some setting that can fix this or is this currently a flaw in the way matrix is federated? I don't need the history of the channel and I don't need stuff I missed when I wasn't present.

This is partially a flaw and partially a feature. It should be common sense that joining a massive room on a tiny server may overload the server: it has to talk to all other servers in the room (which is where most of the resources go), and keep track of the current state (ie key/value pairs such as who’s in the room).

It’s also a flaw in that this could be less heavy: it could be faster to join (which we’ve almost solved via Faster Joins - see https://matrix.org/blog/2022/12/25/the-matrix-holiday-update...). We could also speak to a subset of the servers in the room (at the expense of reducing the decentralisation of the room). We can also optimise state resolution further.

For now: if you want to participate in a room whose users are spread over 1000 servers, you’ll need a server beefy enough to talk to 1000 servers every time someone says something.

Anyone can recommend a reliable alternative Matrix server?

Dendrite, written in go. Its super easy to install, is fast and even runs on low specced systems

I loved using dendrite implementation but people should be aware if it’s limitations. Since it’s basically a rewrite of synapse in go, they are currently playing catch up.

All features of synapse are currently not there. For example, most media APIs are not working, like URL preview. Before 2 months ago, I feel like they were making great strides with the project. They were pushing out weekly updates.

Development has definitely slowed down, and it doesn’t help one of the key contributors/maintainers has left the company/project.

Hopefully the project gets a few dedicated maintainers and people to manage and fund it. It really is a great implementation but I ended up switching back to synapse.

If you don’t mind me asking, how easy was the migration to and from synapse? I’m currently spinning up a homeserver with Dendrite and am wondering if I should switch back to synapse, but I wouldn’t want to lose any data.

From dendrite to synapse, I don’t think it’s possible. I just nuked mine completely.

I recommend setting up a separate instance and subdomain.

Why do you recommend setting up the separate instance/subdomain? On my latest attempt, federation has never been enabled, so it feels like I should be able to just reuse the domain.

Think so too

Dev hasn't slowed down that much; there's still 3 other people working in Dendrite's vicinity. We're just focusing more on the new Sliding Sync API first.

An alternative server instance or alternative implementation?

Do you mean an alternative server software or alternative homeserver instance?

On the latter case Mozilla runs one. There are tables of free ones on the Internet, and you can also host your own.

On the former it isn't clear that the matrix.org software was to blame at all.

Thank you, this is what I was looking for.

Reply on news.ycombinator.com