I know people who grew up with disabled relatives and I can assure you that insult is among the worse things you could use. Maybe those who use it don't think of themselves as bad persons, and "thats just the way language evolved", but if you use this word it shows that you never reflected on the other persons position. It tells me, you are either not the brightest bulb in the shed, unempathic or just mean.
So yes, depending on the context using retard as an insult is worse than many other things. It might not be to you, but so might have been the n-word a few generations before. Just because a lot of people don't get it, doesn't mean it is okay.
Edit: and now that you know that this insult might cut two ways (because you never know who has a disabled relative), you might also deem it less useful.