True, but that a form of reasoning inevitably gives a wrong conclusion under rare circumstances is only a weak argument against it.

-Based on our experience, it takes ~4bn years for life to evolve in a planetary system.

I don't think we have a strong reason to believe that there's something special about that length of time. Not much change happened on Earth for most of that time, suggesting dependence either on stochastic evolution of a few things like photosynthesis (so 4bn probably isn't much more likely than 3bn) or on something like increasing solar output crossing a threshold (it's doubtful such a threshold would be crossed at ≥ 4bn years at every location). That aside, even a few tens of millions of years is plenty of time for life to cover a galaxy.