How does my use of drugs infringe upon someone else's rights?
What if reading hacker news at work results in you losing your job and becoming dependent on welfare, should we outlaw news sites?
You have a right to drug use in private or where everyone present consents to your using drugs, like in group psychedelic rituals.
What I mean is, if you think about extreme situations, drug use will stop people from being able to provide you with legal rights. But forget about drug use, if everybody quit their job - which is totally legal - you wouldn't have any legal rights either.
And outside of the realm of people doing paid work, a child has the legal right to be raised by a non-abusive parent or guardian. If the parent is fucked up all the time, legally and at home, that child is having its rights infringed upon. Whether the government intervenes is another question.