Last time I've tried Matrix (this spring) with a group of peers, my E2E rooms were full of random "failed to decrypt" and lots of out-of-band communications "hey, are my messages working for you today?"

Yes, we had one Synapse server running on a resource-constrained machine that sometimes "fell behind" the rest of the network. I believe that is what had caused such issues. Still, the fact things easily break with server load or network issues means there is something faulty about the protocol. Resilience and reliability are no less important than security.