Dumb isn't used to denote what its original meaning is (a person unable to speak - we would use the word "mute" today). Dumb just means stupid, as if one is culpable for their own stupidity.

Retard in its noun form literally means a mentally handicapped person. So by using it to offend a non-mentally handicapped person (albeit, a stupid one), you are concurrently levelling an offence against the mentally handicapped in general and since the mentally handicapped are not deemed culpable for their situation, this is deemed socially unacceptable.