All it takes is some egotistical civilization that builds von Neumann replicators to build KILROY WAS HERE monuments made of diamond on every moon in the galaxy (it's a big galaxy, but there's millions of years to play with). It's at least mildly surprising that there's no evidence of any extrasolar visitors in our solar system's past.