I've never thought of retarded as a putdown, it was just the sound used to describe a certain group of people. Be careful assuming you are in possession of the one true way.
You may be one of the few, in that case. Good! Other people might not understand that about you if you used that word around them.