For example, we could have begun building a base on the Moon by now if we wanted to, but we have decided not to.
It could actually serve well as a strategic and logistic launchpad for subsequent missions.
But still, you highlighted a key point yourself: That it's not always going to be possible (or desirable) to continue expanding through the galaxy because we may run into a "dead end" with no stepping stones (i.e. habitable planets) within our range of travel.
The other thing is that the paradox isn't just that they aren't physically here with us. It's that there's no evidence of intelligent life whatsoever. We're at the point where we can take direct optical images of exoplanets. At some point we have to conclude that if they are out there, we would have seen some indication of it.
Why? How? How soon would they be able to start? Why haven't we begun doing so ourselves? Why haven't we even established a presence on our neighboring moons and planets, even though we are technically capable of already doing so?
At the risk of sounding like a broken record, the "we should have met someone by now" argument relies upon a bunch of thin assumptions that have been brought up elsewhere in response to other instances of it: