Space is mind boggling large and sparse. Millions of years is not a long time to be traveling through it.