You have to change the structure of the market so they no longer see these investments as profitable.
One way local areas do this is by "homestead tax exemption" which reduces property taxes if you live in your own home, but this is binary and punishes small landlords equally as big ones.
My town is experimenting with allowing this exemption if anybody claims the address as their primary residence, whether it is the owner or not. The main purpose is to cut down on people keeping vacation homes, but it should also make things more expensive for flippers and speculators.
Small landlords are no better than big ones if they’re keeping properties empty.
Once an area reaches a certain % of landlords (no idea the actual %, but it's low), it leads to a general decline of the neighborhood. These landlords are buying starter homes (apartments and houses) whether they are PE companies or individual landlords, driving up the cost of "cheap" housing.
Now you might say those bans aren't 100% effective, but why do they need to be 100% to be justifiable?
Housing is not a free market by any stretch of the imagination, so if you just move one lever you don't always get the response you would in a true free market.