It is a decision, but, I wouldn't say it's a "big" one. It is possible to come off TRT, though it's a long ramp-down period. The TRT itself isn't really a hassle, I just poke myself in the butt twice a week. I've worked it into my morning routine: shave, shower, teeth, poke.
I was like you before I started TRT, super hesitant, and worried I would just be poisoning myself. A good friend of mine kept pushing me to try, and I can't say I have any regrets.
For me personally, I wouldn't want to come off TRT, I feel so much better consistently. IMHO, there are no downsides, at least none yet that I've experienced.