"Dumb" actually has the same problem as "retarded". We are just not as concerned about the politically incorrect implications to the group of people which are unable to speak.
Context that matters the most. Groups can either try to reclaim the word or start using new ones, which I believe many are doing.