If you "feel bad for him", you are not looking down on him.

If you "feel bad for poor him", you are referring to him in a way that implies his position is lesser.

"Poor $noun" is nearly always derogatory towards $noun. Off the top of my head, I can't think of a neutral or positive use for "poor $noun".

Verbal communication is imprecise enough that people will overlook "poor $noun". In text, you have the opportunity to proofread and revise your statements, so "poor $noun" drastically changes the meaning of the phrase.