The first intelligent life to evolve will be able to colonize it's entire galaxy in a relatively short amount of time. The fact that Earth hasn't been colonized is pretty strong evidence to me that we are the first. To me at least.

Even if aliens chose to leave Earth alone for some strange reason, we still haven't observed any evidence of aliens elsewhere. No dyson spheres, no signals from space, no stars arranged in perfect geometrical patterns. Let alone visits from aliens, either to help us or hurt us.