Anytime the topic is brought up I bring up the term "stoner" and leave it at that. Something about too much use of the stuff is definitely wrong and not researched enough.
In any case, the question isn't so much how it affects you while it's active, but how it affects you long-term. Transitory effects of any nature are not a concern, unless we're talking about rage fits or other behavioral changes that are actively and directly harmful to others.
It will take awhile for accurate research to produce hard truths