> It is a proxy for other people, so you are essentially saying "everyone else owes me a house and I am willing to use governmental violence to get it."

This is pretty much the principle behind distributive and legal justice, expressed crudely. I have no prima facie problem with the idea of a program which is constructed to recognize that everyone is owed shelter, and if it takes declaring that as a "human right" (whatever that might mean) then so be it.