You mean like pyramids? Heh look at what few thousands of years caused to them, and its not even very challenging environment (very low humidity, mostly sand abrasion). In fact, go to mountains and actually try to look around a bit from geological perspective - they are constantly being weathered, all types of rock, by mostly rain and elements. Elements cut through stone like through butter.
Just because something doesn't visibly keep disappearing in front of your eyes doesn't mean its some ultra stable form to last for a billion years.
Also, one proper earthquake and your fine stone structure is a big pile of rubble. Now do it 100x.