The technology is also just a bit behind.
People see it as geeky and not really solving any problems they have.
Personally I am a fan of the new hybrid watches that are coming out.
They look great (they look exactly like classic watches, because they are), the battery issue is gone (modern hybrids last up to 6 months or longer) and do what people really want a smart watch to do:
see if that notification is something worth taking your phone out for. (most hybrids have different symbols or vibrations or lights that show what it is and who its from).
Throw in the activity tracking and controlling your music from your wrist and you have the perfect combo.
If the marketing is done right I can see hybrids really taking off.