It is a simple fact that the Earth and solar system are small relative to the universe, because everyone agrees on how to measure dimensions and volumes. It is not as simple a fact that something more large is more significant -- whatever that means (if we knew what it meant, this claim would be settled).
Physical size is such a universal metaphor for relevance/significance (eg, "it's a big deal", "it's a small matter") that they're implicitly treated as synonymous.
So the precise claim is that to describe any interplanetary scale (or up) process you can completely neglect humanity in its entirely, because its contribution to the process is negligible.
You are right that there's an unspoken step from "too small to impact the sun in any important way" to "insignificant to the universe", but it is not actually a very large step: being unable to affect one's surroundings is a classic sign of powerlessness.
So if we can reach the edges of the Milky Way, other galaxies should be within reach. But doing that before the Heat Death/Big Rip/etc. might be a challenge.