Federation can feel like "just a feature" but the E2E encryption (also in group chats) is a reason for Matrix to exist and a big reason why it's so slow.
The sync between large groups used to be slow because of amount of data, but Element X and "sliding windows" were rolled out to help with it.
AFAIK, the public Matrix server used to be slow because of a heavy load (I think), but on my self-hosted instance that's not a problem at all.
And event the slower Element seems far better than Discord that I'm forced to use, where I can't even scroll history without the whole thing stuttering.