I don't understand what the difficulty is. You either act, or you don't actually perceive immorality.

People are not acting immorally at all times when they're not attempting to remedy the most immoral thing, as long as their moral framework indicates not acting (or continuing an ongoing action) is more moral. Should I donate all my money to charity? No, so long as, for example, I am concerting my efforts to use that money toward a greater moral goal.