It's probably only "not smart" because you're assuming some hypothetical people are going to be hypothetically upset about the weed use when they have no real reason to be. In effect, you'd be that reactionary person you're worried about.