Anti-semitism exists and is reprehensible. That having been said, using it hand-wave away what could be legitimate examination and discussion of a person whose behavior may be controversial is dangerous- and could encourage true anti-semitism. Over-use of accusations of anti-semitism in such a way that discourages discussion
encourages bigots to say "All <insert a people> are <insert a pejorative>", particulary if the discussion topic is a figure whose actions may be controversial at times, such as Soros. I'm focused on Soros here because he came up in discussion; this is but one example of over-use of claims of racism, anti-semitism, etc. being used to stifle legitimate discussion.
I can remember 4 controversies surrounding Soros off the top of my head. There is (or there should be) room to discuss the statements below without redirecting off-topic by raising a curtain around the subject through claims of anti-semitism:
* In the late eighties Soros was convicted of insider trading by some French regulatory authority. If memory serves, his appeals of these convictions failed. Some people claim the conviction and fine was politically motivated, some claim the conviction was true as written.
* In 2009 a Hungarian agency fined Soros Fund Management LLC for manipulating values of Hungarian OTP bank stocks. The same arguments were made for/against the justice of the conviction.
* There are claims that Soros took advantage of the British screwing up when they pegged the Pound against the Deutschmark under the Exchange Rate Mechanism, and that he gamed a lot of value from the British economy as a result. I probably don't have the details right, I've heard people say that he sucker-punched the British, and others say that it actually helped the Brits in the long run.
* There are claims that in the 90's Soros manipulated the Maylaysion economy, society and currency for his profit, and that many suffered as a result. Some believe Soros had nothing to do with the Malaysion troubles and call this a conspiracy theory- it's still worth discussing.
My point is that the use of labels that identify real problems to distract from or avoid altogether a potentially controversial discussion has the potential to make [real] racism, [real] anti-semitism, etc. much worse, and should be avoided.
EDIT: spelling / grammar