In all the historic places you visit and that, in some parts of the world, add up to a significant amount GDP thanks to tourism or related activities. There are some intangible values as well as they help shape the history and evolve a community around specific places (think West Village, Manhattan for example).
I know it can be frustrating and that sometimes is poorly implemented but historic preservation of places eventually pays off. Maybe not all of the landmarked buildings will pay off, but sure some of them will(not unlike startups in that regard).
And what is not culturally important today might be a century form now, or viceversa. You are applying survivor bias. Think of all the historic places no longer with us because they were not deemed culturally relevant or properly protected in first place.