In high cost-of-living areas it's the land price.
By "not entirely", I mean that while land value often dominates, lowering it doesn't solve the problem, because even if you somehow acquired a city lot for $50k in the Bay Area, you wouldn't be able to build a house on that lot for $100k. So, solving one problem doesn't make it affordable.