I know a couple of people outside of my field who are taking stimulants and talking about it. I know a couple people within my field (and outside) who take drugs solely recreationally, not as a performance booster or a way to get to sleep or anything, and certainly not on a regular let alone daily basis.
And I definitely know enough people within my field closely to know that they're not taking stimulants.
I think you just inhabit a different world, yes. (And I worry whether you inhabit a world of people who think they're productive; a sizable number of the the really productive and skilled people I know tend to be so clean that they don't even drink.)