12GB of RAM sounds like something is badly wrong; a typical small server should use a few GB at most. Most likely is that either your users have joined loads of big rooms, or some malformed broken room, and the IO can’t keep up with all the traffic it’s seeing so things are stacking up in RAM. Please come to #synapse:matrix.org so we can help debug it.

Actually it does. XMPP does a pretty good job at keeping up with "mod cons", but people often assume it has stayed the same since 2006.

I'm working on https://snikket.org/ partly to try and fix this perception. All Snikket software (server and apps) uses modern XMPP, are open-source and designed for easy self-hosting.