I too am fairly drug-averse. And after 40 years of avoiding all drugs but caffeine and alcohol, I dipped my toe into psilocybin and found not only was it profoundly therapeutic in terms of self-acceptence, self-love, and anxiety-reduction, I found after 3 or 4 uses, I had no further interest in it whatsoever.
Is my experience universal? Of course not, but an anecdote to consider alongside yours.