In that situation, no matter how fulfilled you might be most of the time, you're at a super vulnerable low point. You feel damaged, like your resiliency is compromised.
Then you take drugs, prescribed or not, while at that low point. Many substances can pattern compulsive, addictive behavior after just weeks of use.
Now you're regularly under the influence of substances that actively oppose your returning to a fulfilled, resilient lifestyle. Not only do you have to recover from what happened to you, you have to do so while also ceasing use of chemicals that tend to pattern "just do this instead of expending effort on other things" behavior.
True, many people do bounce back. And there are many cases where drugs' side effects are worth it.
But the compounding effect is real, and handwaving it away with "happy, healthy people don't abuse drugs" is missing the point. What if someone takes those drugs during a moment when they're not happy or healthy?
Relatedly, there's a feedback effect here. You know what makes it really, really likely that you'll have an unfulfilled, hard life and feel like you're on the outside of society? Being around people who feel that way, which, as you point out, can often be heavy drug users. Now you have questionable behavior modelling (often starting young) as well as an emotionally unhealthy environment that makes it likely that you won't be resilient to the addictiveness of anything you try.
This applies equally to transgenerational drug abuse and to people who just live in/around communities with lots of drug abuse. If you spend a lot of time in a household/group/community like that, you're more likely to abuse drugs as well* .
This is part of the reason that radical decriminalization programs are great ideas: they focus on improving quality of life and everyday function of drug users with the goal of both reducing immediate likelihood of overdose etc., and of reducing the ingress rate--the number of new users of a given drug.
* This should not be taken as an assertion that drug abuse is the only, or even the primary, reason that people in many communities/groups/households are marginalized and feel unfulfilled, just that it is a single factor with a negative feedback cycle that can be broken.
Edits: grammar, clarified the idea of transient unhappiness.