Well I don't claim to be an expert, but I've read elsewhere that there are a lot of different qualities of sand, and that desert sand is not (as) useful as building material due to its shape and structure.
https://www.quora.com/Why-is-desert-sand-not-used-for-constr...