I think we are mostly in agreement - that any substance entering the body will have some effect.
The question is what effect, how much of an effect, and wether it is permanent/long lasting. Questions which require a study to answer.
Unless we're able to find twins that live largely the same lives absent the one factor, we are really only capable of sussing out the especially harmful things with long term lifestyle studies. If cannabis is as bad as sugar, then we have 40 years of back and forth to look forward to, and the answer will ultimately be: A little bit is bad, but not bad enough to matter more than how many steps you took today, or how sad or angry you were, or how late you ate dinner.