It's too awkward to hold your arm out for the amount of time needed to meaningfully interact with it. They're big and clunky on small wrists, and the screen is way too small for people with big fingers. For the price you pay for one, you can get a really nice designer watch, which isn't limited by the lifetime of lithium-ion batteries.
It's just a silly gimmick, but so many people have buyer's remorse about it, that they forced themselves into the habit of using it.