But that's the beauty of this calculation it just takes one splinter group of one splinter group and they could easily populate the whole galaxy in under 10-100 million years - which, being the conclusion that Fermi found, is nothing at the time scales of when our galaxy was formed.