Hacker News Reader

Comment by dzaima | Hacker News Reader

The issue of broken rooms was reportedly the database having gotten corrupted indexes, which affected a bunch of rooms (2/55 of ones I was in): https://github.com/element-hq/synapse/issues/18606#issuecomm....

Not much a protocol/implementation can do about that. It did take over 2 weeks for it to be resolved though, which is a rather long period of time.

Also on the status page - https://status.matrix.org/incidents/8gljb3gtlv11 (shows start at Jul 2, but I noticed it on Jun 28)

After that was resolved, messages that were sent and received by/on other homeservers during that time never ended up on matrix.org, so much for federation :/

View on HN

The corrupt room bug which pushed the OP over the edge was due to a corrupted DB index on matrix.org which caused millions of rows to be incorrectly deleted and was an absolute nightmare to debug and mitigate. You can see some of the story at https://marc.info/?t=175161814600006&r=1&w=2.

We need to write up the incident report; it’s not clear whether the root cause was a bug in postgres (potentially years ago, and only caused problems when we started using the corrupt piece of the index by coincidence) or due to potential corruption from a HW failure years ago.

The rooms affected should now work again after reconstructing the lost data. No data should have been lost over federation though; the room history should have synced up as normal. Also, only room state (not msgs) was effected. Do you have a bug report anywhere for the missing msgs?

Thanks for the link to the postgres discussion! An incident report would definitely be interesting.

I haven't reported the missing messages thing anywhere as I assumed that was just expected (IIRC (and this is a big IIRC) it was mentioned a long time ago (i.e. entirely unrelated to this) by TravisR on Discord bridge problems as a possible cause for missing messages).

Here is one event in an affected room, made by a t2bot.io account on July 6, to which t2bot.io accounts can reply to, but which gets an "Event not found." error when requested by me, @dzaima:matrix.org: (and for reference t2bot.io functioned properly on other rooms during that time, so overall t2bot.io↔matrix.org federation should've been fine; and for reference there's a third homeserver that federated with t2bot.io fine in that room just fine too; and to be clear the room works fine now)

    https://matrix.org/_matrix/client/v3/rooms/!WpdazzauuDxyGNAiCr:matrix.org/context/$BM9s8e3vUhk_iHfdg7D_s7CsQ5COFSPdlJdygz7TKvU

Huge thanks for giving details we can investigate; i've taken a look:

* That event is in the matrix.org DB...

* ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent

* This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index

So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday.

Sorry you got hit by this, and thanks again for the actionable report.

FWIW, my specific room didn't lose much due to the missed syncing, I manually copied over the lost messages (..and we're used to doing so due to the flaky discord bridge). I just noted it as it seemed like a generally relevant thing (and of course perhaps more important for other affected rooms). Thanks for looking into it though, good to know that's at least not how it's supposed to function!

Wait, did anyone investigate on why messages were not stored by other homeservers? That's rather odd, given the selling point of matrix is federation

I've heard that messages not getting through federation in some cases is just a thing that can happen. How that's compatible with calling matrix as having good federation, I don't know.

I’m not aware of any scenarios, including this one, where messages don’t federate (unless the destination server has been unavailable and marked down, obviously).

No, you are making things up. That cannot happen unless the homeservers are not configured properly. If you don't have anything to back that story up, I suggest you slow down on making matrix look bad

Unfortunately I cannot find where I had read that, might've just misinterpreted discussion about delays as about dropping messages / interpreting messages as answers to questions I didn't actually post... Sorry for the unnecessary negativity!

in dzaima’s defence they did hit an instance here where messages were effectively dropped - but only because the matrix.org db was so corrupted it incorrectly quarantined them (and even then, they can be flushed through). corruot db is effectively an extreme form of misconfiguration, i guess.

Reply on news.ycombinator.com