I think the burden of proof is on your position. When activists in my area pick a fight with local government, it's never over that invisible group. Indeed, as far as I can tell, they don't give a rip about the veterans living in the forest over by the VA hospital. I've never seen them over there. They're too busy picking fights defending the ones pooping at the downtown parks (and not in the public toilet) or trying to get city funding and approval for safe drug use, etc.

Now, you're right that there are people that do care and put tremendous effort into that more invisible group. Churches and church-backed organizations do a lot of that. Difference is, except for fund-raising from like-minded folks (and a lot of them hate even asking for money), they go about their work quietly, without seeking attention or reward.

There'd be the possibility of joining forces, but not with the DSA-types that jump in the limelight and scream the loudest and advocate for free needles. The people doing the bulk of the hard work don't want to make a statement on Palestine, they just want volunteers for the hard-to-staff weekday shower-cleaning shift. They need men willing to be mentors, etc. They care about the actual people they're serving, not trying to bring about glorious revolution. The two groups aren't compatible at the moment.