The argument is often as you say, that Land Lords add value by giving housing where people couldn't otherwise afford to live. And this is undercut by two points:
1. The landlord isn't the essential part there. If the point is to provide housing without having to buy, that's easily done with land trusts and co-ops. The only downside is that there isn't a Landlord to make a profit, but it's better for society as a whole.
2. If landlords didn't buy up properties for the purposes of turning around and renting them, the property values would be lower because there wouldn't be as much demand on the market. Then a decent portion of the people you're talking about not being able to afford to live there ... would be able to afford to live there.