The harm we are reducing is decreasing the amount of drug users, and the attendant crime and degradation that comes with them, within the environment that non-drug users have to live in.
We'd be reducing the harm to them and in a subjective way, to the overall community. That some people's "harm" will increase is obvious but this is a price many are willing to pay to achieve a better society for themselves.
In summary, "we will hang you" isn't "harm" but justice, and the solution to the "harm" that needs to be decreased.
> it doesn't seem to stop them
Sure it does, it decreases overall drug use, where it is harsh enough.
>that didn't stop most people from trying it.
Did it decrease overall drug use vs when these laws were relaxed? Yes. Mostly though the penalty for drug dealers needs to be harsher. Additionally, there is a hierarchy of drugs, and in US the biggest problem is hard drugs / opiates. We should focus enforcement and punishment efforts at those, while also not enabling the addiction of users.