For most of history people didn't have the free time to be regularly using drugs the way some people today imply. If your life is farming from sunrise to sunset you're gonna need a lot of sleep and there isn't much time left to be tripping balls. You'll probably only can justify using your available free time to do whatever your local drug is at special occasions, whatever handful of religious holidays your culture has and maybe a handful of other times every year. Yes, there's downtime in the winter and summer but depending on the size of the family you have to support there's still an infinite list of things to do. There's always a leaking butter churn or cracked axle handle that needs fixing or a cart wheel that needs greasing. The time to just sit around and toke it up does not exist excerpt but for the few who prioritize that sort of recreation over improving their material situation. Responsible adults drink a few beers (or toke it up) on Friday night (with the definition of "few" depending on how many responsibilities you have). The vast majority of people weren't sitting around getting baked or drunk every other evening.