Many people who go to stamp dealers are collecting because of a personal interest in history. It's a hobby, but it's a serious hobby for them with non-entertainment goals. It's those people who start stamp stores, visit them, and generally keep the entire industry alive.

Finding someone who simply does it for 'joy' is against the grain.

It's like having a Porsche and not knowing anything about its internals or engineering history and simply driving it for fun on the track. If you join a car club, they'll think you are weird for not caring or knowing the displacement of your engine or your tire size.