Take two stones of similar roughness and rub them together for many, many hours. Eventually you'll have to flat surfaces that fit almost perfectly together. Beyond that, it's not hard to see how some skilled craftsmen with some knowledge of geometry (and lots of laborers and a royal mandate) could construct what the Inca did.

What they built is incredibly impressive, but you don't need to invoke magic woo to explain it.