>Is it socially just to stop all these classes of property buyer?

You don't have to stop them completely, they should just be deprioritized while there's a housing shortage.