Saying privilege doesn't factor in when someone trots out the old "poor people would be just fine if they weren't lazy/ignorant/poor/etc" is just as much an attempt to silence a viewpoint you clearly don't agree with.
Or something else, entirely? Really, I'm curious... As someone whose parents had no privilege and were poor.