You're probably right that this betrays a bias of mine. And you're also probably right that this isn't a great guiding principle for decision-making in the future.
Still, if I'm interested in increasing the total number of people contributing to open source projects, I would give an issue like the use of the word "retard" a fighter's chance.
Maybe it's not worth bending on free speech in some specific cases, but there is a point where it is worth it, ie, if there are enough people in the offended audience that the chance of losing a potential contributor is nontrivial.