I've found that the biggest problem with using an exercise ball is that everyone asks you why you are sitting on an exercise ball. That and it is fun to kick.
Yes, it's just not worth it to be of "those" people. Whatever benefits it gets, its just hard to take someone seriously when their sitting on a giant inflatable fucking ball.