Mainly messages getting delayed or undelivered (often on the IRC bridge), everything getting randomly very slow, encryption randomly breaking... sadly what I consider the usual for Matrix.
I recently had a particularly vexing issue in a small encrypted room of 3 people, new messages were shown as undecrytable and then only readable a few hours later, we never figured out why.
I re test it every few months/years because I hear it's getting better, the server is properly configured now, encryption system was remade, there is a new client, etc...
Losing my whole chat history each time made the endeavor even more frustrating. I know not everyone keep logs but I can search my Mails and Irc logs from 20 years ago and I find it really useful.
Sorry for the rant but since you asked. Like I said I fully support your work on Matrix, I just find it so ... frustrating and it make me sad because I really like the idea.