"Boycott Israel because of their treatment of Palestinians." A perfectly acceptable political statement.
"Boycott Israel because they are a bunch of greedy Jews." An obviously anti-Semitic statement.
The problem is that people who believe the latter will often say the former. This causes opponents of the latter to doubt the authenticity of people who say the former. Some people who oppose the former might also accuse people of the latter to discredit them. It becomes can quickly become confusing, but it should be clear that a boycott can clearly have both appropriate and inappropriate motivations.