If you believe that dealing with homelessness is critical, then you should be willing to take some sacrifices yourself in order to personally house homeless people. If you start making exceptions, then you are saying that dealing with homelessness itself is not inherently good, but only good in _some_ places when it impacts _some_ people.
You can apply this to literally anything. This argument is silly.