Some people have more receptors and/or are more sensitive to certain things.
Higher testosterone leads to more risk taking and other adverse behaviors, as well as increased blood pressure and, anecdotally, "increased aging." I believe it has something to do with collagen synthesis.
You absolutely made the right decision to come off TRT.