Homeless people and drug addicts may well be victims of circumstance or victims of society depending on your point view - or even victims of their own choices or actions - but there’s a difference between saying “this race of people is problematic” and “this class of people is problematic”.
I think DHH was trying to refer to junkies and homeless people most of whom happen to be immigrants. The immigration status is the particular attribute that invites usage of the term "gypsies" to signal that the problem with those particular junkies and homeless could be solved via immigration enforcement. At least in his eyes, I presume.
Again, not defending his choice of words, just providing a somewhat plausible explanation as to how one might arrive to this specific terminology.