The problem imho is that a lot of people, not only "the very rich" but also some very "middle class" people/families, have access and can make use of houses as investment vehicles, but houses (aka shelter) is also a primary need for humans. If you hold share of a publicly traded company, you can hold it forever until you think somebody can accept to buy it from you at a price that you like, hoping to make a profit if such price is high enough, and nobody will suffer from this process. But with houses, there is always somebody in absolute need for it, which means that either they will accept to rent it to a price that covers extra taxes applied by the state to you (as a landlord), or they will try to squat if they can not. It's really hard to enforce the right set of disincentives that are wide enough to convince people not to "hold" but at the same time does not apply to too many people but mostly the ones that are using houses as investments.
The whole thing is complicated by being geographically unequal: for example I think even Spain is full of affordable houses, only they are not in Madrid or Barcelona or Valencia, which is where people really want to live. So if you have a second house in an unpopular town you actually have not much - thus you are not rich - and you have often an empty house (nobody wants to rent/buy it) which is an easy target for squatters, and therefore you will become "one of the poor people ruined by squatters", while othen you are somebody who accepted the narrative that using houses as investment was a good idea, both an investment house cheaply in a town that never attracted enough people, thus "lost the game" and is now also losing the house to squatters...