I have made a similar experience as a semi-competitive chess player. As a child, I never perceived chess as an outlandish hobby. Chess clubs are full of people from all walks of life. But nowadays especially (maybe after enough movies used chess players as a trite metaphor for 'genius mastermind') people seem more often associate chess with that kind of stuff.
It strips something from the activity itself if it becomes associated with certain exclusive groups or classes. It's also happening to healthy nutrition or sports which are increasingly seen simply as signifiers of elite status and optimised lifestyles, rather than communal, shared activities.