That definitely used to be the case with Matrix. Sync has been improving gradually over the last year to alleviate that issue, though, assuming your server regularly gets updated and you keep your clients up to date.

Sync still takes a second in large rooms, but you no longer need to download a thousand messages just to see the three latest ones.