Generally from the start the word meant hating Jews (not Semitic peoples as in those that speak Semitic languages - Jews specifically). If it didn't originally, it certainly came to mean that over time.
https://en.wikipedia.org/wiki/Antisemitism#Etymology
Maybe people that speak Semitic languages are Semitic (I don't know - feels like pseudoscience, though there could be a lot of commonality in different ways). That being said, "Antisemitism" means hating Jews. It's not a special prize or anything. More than anything, the terminology is an unhelpful distraction from the more important things around when it comes up. I wish they stuck with "Judenhass".