Are you using matrix.org? Maybe you could try a server closer to you?