I think your first point ignores one of the fundamental insights of the Fermi Paradox, which is that while space on an astronomical scale is massive, so is time. The milky way galaxy is ~100,000 light years across. So traveling at 1% of the speed of light, you could go end to end in 10,000,000 years. That sounds like a long time, but it's really not.
The earth is ~4.5 billion years old. So we can say with certainty that it's possible (not guaranteed) to become a space faring race (although not a galaxy crossing one) in 4.54 billion years. If we tack the amount of time it takes to cross the whole galaxy end to end we get 4.55 billion years. The milky way galaxy is 13.61 billion years old. That means the galaxy had 9.06 billion years worth of chances to churn out a another planet that could have expanded across the galaxy from edge to edge and been here waiting when humans arrived on the scene. Planets with Earth like conditions are certainly rare but my, admittedly limited, understanding is that they're not "1 in 9.06 billion years" rare.
There are obviously other constraints here. I think your second point about wealthy technological races potentially stabilizing at zero population growth is totally reasonable. But I think the fact that space's apparent scale doesn't actually matter is the whole reason the Fermi Paradox was such an "ah-ha!" moment for many people. The reason it's a paradox is that, given all of our assumptions for variable values in the Drake Equation at the time, it really seemed like we should have met some aliens. The point was that if distance wasn't the hurdle we thought it was in the equation as currently defined, then we were either missing some relevant variables, or we had made some bad assumptions for the ones we had.