100k years is tiny compared to the age of the universe. Even if their ships only travel a tenth of the speed of light, they could have colonized the whole galaxy in a million years. That's a third of the time it took humans to evolve from chimpanzees, and only a tiny fraction of Earth's total history.