If what you're excluding is political speech of a sort, then someone is going to be able to accuse the system of censorship, no matter where the line is drawn. It's inherent in the system's goals.
Personally, I think that's a point where any system is going to be rightfully challenged and examined critically, but I think it's still worth doing on a greater good basis. Spam filtering works, it makes my inbox a more tolerable place. It's a good idea to extend it to other types of crappy speech that makes other peoples' lives worse.