That's the gist of it. If land is so worthless even rich people don't want anything to do with it, why would it make someone poor better off to have it?
There is a difference between something being intrinsically worthless and the taxes driving the market price to $0.