What is the non-violent use of a rifle that is important? I sold pot in college in the 90s, and there was no violence. Perhaps it is the ilegalization of various plant materials that causes violence in the distribution process?
Unless you can personally claim ownership of those people, among other things you're claiming ownership* of, there is no aggression (initiation of violence) taken upon your person or things when someone is destroying their life.
If you can personally claim ownership of those people, and society as a whole, then you are correct and the selling of the drugs would be an act of aggression towards yourself.
* The act of withholding something from others.