land value tax is not assessed on the basis of who is presumed able to afford it. it's assessed against the use of a public good --- the ability of more people to live and work in the city that perhaps makes the land more valuable than when the owner bought it.

also observe that the purpose of the tax is not strictly to raise revenue, but to address what would otherwise be an externalized cost and make the housing market more efficient.