Enormous piles of very skillfully worked massive stone blocks. Artifacts made of common, long-lived matter so precisely crafted that we could barely duplicate them today.
Just because something doesn't visibly keep disappearing in front of your eyes doesn't mean its some ultra stable form to last for a billion years.
Also, one proper earthquake and your fine stone structure is a big pile of rubble. Now do it 100x.
During the last 2.6M years - and man's been around -at least- that long - there've been a dozen or two ice-ages. Between them were interglacials, some of which lasted 2 or 3 times longer than our 10,000-year holocene. More than enough time and resources for an earlier (even wiser or saner) humanity. Or three.
"Our historians say it is the genitalia of a species who lived in ancient times."
;)