So, either there's a filter that's blocked interstellar colonization... or it has occurred already, and we just aren't aware of it (some sort of dark forest thing / we are being secretly watched by an extraterrestrial civilization).
(This is a haunted galaxy either way!)
There's no reason to believe a civilization is capable of million year projects.
Saying "I have a bad model but there's some unknown Great Factor which would make it a good model" isn't very convincing.
There are myriad factors: the biological challenges of space travel constitute a long list in itself; the economics and politics involved (ROI on any useful timeframe for interstellar travel is zero); the engineering challenges (the vast majority of interstellar travel attempts will result in the death of all crew somewhere in interstellar space); the fact that over million-year timescales, species evolve and change - humans are only around 30% of a million years old.