The issue with sugar is bone char—animal bones are often used to bleach white sugar. That’s similar to the problems with some beers. Guinness, for instance, uses isinglass (fish gelatin) for purification.

While I believe that some vegans might have some concern about harvesting. I’ve never met anyone who wouldn’t eat conventionally-harvested vegetables.

NB: not a vegan but a kosher vegetarian and so have some similar restrictions.