Brother, it sounds like your life is made of horrendous side effects. I am just a random internet commenter, not a doctor, but I am a fellow man. And if there were ever a time to ignore your fear and roll the dice, it's now.
I am naturally a bit of a worrier about "big" medication regimes like TRT and I over research and over think everything which is obviously not a great thing when it comes to medicine as everything has side effects.
I am slowly starting to adjust my mindset with regards to trying TRT but I need to see what my results are to help me make a decision.
I was like you before I started TRT, super hesitant, and worried I would just be poisoning myself. A good friend of mine kept pushing me to try, and I can't say I have any regrets.
For me personally, I wouldn't want to come off TRT, I feel so much better consistently. IMHO, there are no downsides, at least none yet that I've experienced.