Of course it makes no sense if you are answering the wrong question.

The question being asked is not: "Why do we not see grabby aliens?"

The question being asked is: "Under what circumstances can humanity become grabby aliens that colonize the galaxy in reasonable time given that we do not see grabby aliens?"

The simplest circumstances that would allow humanity to become grabby aliens in the future is that becoming grabby aliens is easy for a modern human-like society. However, grabby aliens that colonize the universe in reasonable time preclude humanity from existing if we intersect with their existence.

That means there must either be something else special about our existence, or that we are unlikely to ever become grabby aliens. That is the entire point of the thought exercise, figuring out if there is a hope that humanity will become grabby aliens (Star Trek).