Of course there are plenty of downsides. Constantly glancing at your watch is just as rude if not worse than looking at your smartphone in a meeting, conversation, etc. As you mentioned, there is the high cost for a fairly dumb device. Charging it. And depending on the person, every notification is wedged that much further in to your daily life which is probably not for everybody.
I'm not sure there is a lot of value in a super interactive smart watch that allows you to write texts, make calls and take pictures. The interface seems to be small enough that it's kind of prohibitive among other things. But as a read-only satellite device that displays information from a smartphone/tablet/etc, a smart watch can definitely be a thing.