Matrix.org – Database Incident | Hacker News Reader

Matrix.org – Database Incident

status.matrix.org

17 pointsyabones25 comments

> So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.

The stuff of absolute nightmares...

https://mastodon.matrix.org/@matrix/115136245785561439

10 hours seems like a long time for a db restore of a chat server. Matrix is still just a chat server, right? I have so many questions that maybe I should keep my nose out of.

[Edit] From another comment, 55TB?!? Holy wat-man...

There's more than you might think...

- Probably thousands of large chatrooms, and hundreds of millions of small chatrooms

- Probably hundreds of millions of messages that include a media upload like an image or video, including countless re-posts of random memes

- Overhead from ratchet algorithm cryptography, as well as additional message metadata that is likely in JSON format

- Huge excesses of messages from bridge bots, spam bots, and malfunctioning utility bots. To give a sense of scale... the entirety of Libera.chat (formerly Freenode IRC) used to be bridged to matrix.org, meaning almost every single message from Libera would be copied to matrix.org automatically.

- Everything from other homeservers that federate with matrix.org and have been joined by at least one matrix.org user, including homeservers that no longer exist

There's around 50M rooms, and the dataset is back to 2014.

However, much of the room is taken up by the Synapse DB schema being particularly denormalised (prioritising performance over disk footprint) - especially caching snapshots of historical key/value state for rooms, which currently takes up ~65x more space than then actual underlying dataset. Ironically, we're looking into that currently, but not fast enough to speed up this DB rebuild.

Thanks for that explanation. It makes a lot more sense now. Back in the day I managed a Jabber + bandersnatch corporate server and it had many years of Jabber + AIM + MSN + ICQ + Yahoo chat but the database was tiny which is why I was very surprised to see a chat server take up so much space.

Good luck on getting the schema overhead out of the way. I'm sure nowadays you are probably also using faster underlying storage SSD's behind the raid controllers. Dell/HP keep them overpriced of course but I found them to be very much worth it for databases as did the DBA's.

I hope your on-call teams get to take a week off after that incident.

So that means it's possible to reduce the DB size under 1TB, your recovery time would take less than 1 hour, rather >17hours.

for the sake of completeness, we got the DB recovered after about 23h of downtime. postmortem will be forthcoming.