"Freedom of speech" has nothing to do with social opprobrium. Freedom of speech means the government is not going to step in and silence you for failing to use "PC terminology". It does not mean you should be immune from other people deciding you said something distasteful enough that they no longer want to employ you or associate with you.