Ah yes, embracing your inner elitist.
What's wrong with hanging around with folks who get your jokes (and vice versa)? They don't have to form the totality of your identity.
Somehow it seems uncontroversial when someone finds a group of, say, fans of their favorite team (or rock band) and says, "now I am with my people!" Is there some peculiar difference in regards to books or maths jokes?