This is the kind of thing I always read, but have never seen. I don't doubt you, or Joe Rogan. I've just personally never seen an adult who takes just a couple puffs every once in awhile. In my experience, it is a lifestyle of playing video games and sitting around smoking, unless they are someone who also uses other drugs and/or drinks a lot.
It may be anecdotal, but that is the reason it concerns me. It doesn't appear to be a positive for the numerous people I've known.