In Portland, OR, which has a similar homeless problem to SF's, I learned to always include "sorry" when I turned them down, as when I did so, they almost never were rude to me in response. When I didn't say "sorry," I could never be sure what to expect.