We know the answer to this, and it was... the Egyptians. The mistake the earlier Egyptologists made is exactly the same one you're making here, which is that history is some kind of inevitable march towards progress where knowledge and skills are never lost, and we never go backwards.

There's firm archaeological evidence around quarry sites, etc.

The same (pretty nonsense) questions could have been asked about Roman concrete or how tunnels were bored through the Apennines for aqueducts or whatever else you feel like during the late Middle Ages, despite the fact that the demographics of Italy hadn't really changed. The gap in time between Khufu and Ramses is basically the same as the gap between the reign of Augustus and the Hundred Years' War.