I agree with your statement that "cultural heritage is worth preserving, it needs non-market-driven measures to preserve it in some form". However, I think that given situations in which the choice between access to the market and the preservation of culture are fundamentally at odds with one another, access to the market must dominate. I think the downstream effects of the preservation choice are devastating and counterproductive. Let's take an extreme example, if 50% of a given, highly productive, landmass is marked as culture heritage, that would mean that suddenly the available supply to meet housing/commercial demands is now artificially constrained to the remaining half. If the supply does not adequately increase in density to meet this demand, prices shall no doubt increase and people will be priced out of that area. I think this is what is happening across a whole slew of cities across the US and is contributing to homelessness, increase in crime, loss of access to high quality education, etc.
I don't think that this is a zero-sum situation, nor that the example I just gave is the canonical experience. I'm just saying that if I was forced to make a tradeoff, I think the preservation of cultural heritage must be placed as a secondary (albeit very important) value.