Free speech is only free if we allow all of it.
Americans have thick skin. It's in our culture. We've accomplished so much in spite of internal disagreement and oppression: voting rights, women's rights, gay marriage (which still isn't enshrined in the constitution -- work to be done!)
Imagine if we froze back in the 50's because we didn't have free speech. Pick any point in the timeline and freeze the American experiment. It will lead to a worse outcome.
I can tolerate some idiots with racist hate flags, Westboro funeral protests, etc. Because the freedom we enjoy is worth so much more than their minor nuisance.
edit: Yep, predictably downvoted. Go move to another country that isn't America so you don't have to listen to people you disagree with.