The answer is it’s all about tolerance and what drugs are social acceptable. The guy who smokes all day just gets back to baseline with his first joint in the day, he’s not actually high as you would be if you did that - same as with the caffeine addicts.
I think in a free society everyone should be able to put in their body what they believe is good for them - it’s a bit more complicated in real life but still…