Try talking with friends / groups on a few different servers (it's decentralised, remember) and you will soon see what scalability issues are. As you add n servers to grow Signal you need n^x for Matrix.
As a longish time Matrix user I've never experienced scalability issues due to federation. The federation actually works pretty well. Scalability from Matrix comes/came due to the fact that some servers were too popular. Though that is actively being worked on: https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-sca...