How do we even know those holes are 500m years old? You cannot carbon date a rock (that's when the rock formed, not when the hole was made). Perhaps we're missing evidence of earlier civilizations in plain sight
Also, while I was using hundreds of millions because that was the timeframe of the original question, the reality is that we actually have fossils that are more than 3 billion years old.