The other way of looking at that is that it would only take a few hundred thousand years for a technologically advanced civilization to permeate the galaxy. But the galaxy is billions of years old and that has not happened.

The other thing is that the paradox isn't just that they aren't physically here with us. It's that there's no evidence of intelligent life whatsoever. We're at the point where we can take direct optical images of exoplanets. At some point we have to conclude that if they are out there, we would have seen some indication of it.