Why do people pay for The North Face, when you can have knock-off brands for 1/5th the price?
Part of it is paying for better quality assurance, more predictable and coherent UX, and so on. But a big part is that yeah, this day and age, watches are fashion / status accessories. They do nothing your phone can't. That doesn't mean they have no value to the buyers: the continued popularity of pricey mechanical watches is proof that there's more to this story.
The problem is that smartwatches never really positioned themselves as particularly... interesting. They don't signal social status, they aren't pretty, they don't do anything mind-blowing. They're just somewhat cheap-looking and oversized black slabs that display messages from your phone, clumsily.
So far, the closest to a "killer" application for the tech is trying to gamify everyday activities by tracking steps, sleep patterns, or heart rate. But how many people stay fixated on this and really keep it up in the long haul? As for the fashion part... there's only so much mileage you can get out of the ability to change wallpapers or bands.