Why not just build a house with very thick stone outer walls? Seems like all the advantages and none of the disadvantages.
Somebody built that house because it was their dream project, and then sold it because it turned into a nightmare.
If you want a thick stone house, you will need to acquire and move the stone. The usual construction in the US is wood stick framing. It's so ubiquitous that even finding a contractor who will work with steel framing instead of wood is difficult. The material cost isn't awfully different, but the techniques are.
Finding a contractor to handle stone would require a much, much larger budget.