You are exactly right, which is why this is so worrying from an investment perspective.
No matter if you like it or not, the wealthy drive fashion and taste. The reason why we think of BMW, Porsche, Ferrari's as "luxury" is primarily because the rich buy them.
No stock trader is going to trade his Philippe Patek for a $350 watch that his pool-boy wears.
This is something that is not getting talked about. No one buys watches to tell time. They are almost exclusively used as status symbols.
This is the miscalculation in Apple's plan. No one needs a watch anymore. Trying to make one relevant for a purpose other than status symbol is going to be tough.
They should've made a "smart band" that connects to any watch face. That would've been a clever strategy.