Hacker News Reader

Comment by subway | Hacker News Reader

The matrix.org home server is spectacularly overloaded right now, causing messages to take several seconds to be delivered at a minimum. It isn't terribly surprising considering my own federated home server with under 20 users sits around 4gb of memory utilization with a single core (the server is single process/single threaded by default) pegged.

There's some work that can be done to proxy some of the load off into separate workers, but Synapse is pretty darn awkward to work with.

View on HN

  > 20 users sits around 4gb

dafuq? Are you serious? What could possibly need 4gb?

The problem isn't the number of connected or registered users on the server - the problem is the complexity of the rooms they are participating in. So if those 20 users join hundreds of rooms with tens of thousands of users participating, spread across thousands of servers, with large numbers of joins/parts or messages sent a second... you shouldn't be surprised that it takes a few GB of RAM.

HOWEVER, we have been working incredibly hard at optimisation over the last few months, and we have order-of-magnitude algorithmic performance improvements on the horizon over the coming weeks. https://twitter.com/matrixdotorg/status/979049079568240641 etc. There's also a port to python3 happening which should improve RAM by 2-3x given python3 stores strings in utf8 rather than python2's utf32.

I certainly don't mean to throw any shade on the project -- if anything, the RAM utilization was less of a surprise for me than being bound to single-core performance by default. Sure, it's possible to split Synapse up into workers, but it definitely feels like there's some voodoo involved (my first attempt at setting just a Synchroton worker failed miserably this week)

Is it really necessary to maintain the state of the entire network on all federated nodes? (Since I'm guessing that's what you've described)

It isn't necessary, and we don't do it. What I've described is what happens if you type '/join #matrix:matrix.org' and your server starts participating in a large room which spans tens of thousands of users and hundreds of servers. Rooms are only replicated over the servers which participate in them. So if you don't join massive rooms, your server can be have fine. (And in the near future massive rooms should be usable too).

Have you tried leaving any giant rooms you may be in? Like Riot/Matrix HQ? I use my personal homeserver for my friends/some IRC channels and my load is nonexistent. May not be the answer you're looking for, but those huge (10,000+ users) rooms are a huge strain on homeservers right now. Look for performance updates in the next version of synapse.

Yup -- that's exactly what the end solution was. Unfortunately reaching out to those 20 users and saying "hey, could you be judicious about which federated rooms you join? otherwise you're gonna hose us all" kinda sucks.

I've been considering writing a Synapse-equivalent Matrix server in Elixir, if there was enough interest.

FYI there have been efforts to rewrite an equivalent in Golang: https://github.com/matrix-org/bullettime

Dendrite seems to be a much more active Matrix server written in Go.

https://github.com/matrix-org/dendrite

from the bit I learned at a phoenix/elixer meetup - it seemed that it may be the most well suited for multiple chat streams. I am really interested in this.

I have a single non-federated synapse server with 10 users and 25 client devices, and the python process is using 135MB (RES in top). Load average is 0.00. There's 2 main rooms and probably around 10 chats. The homeserver.db is 26MB.

Reply on news.ycombinator.com