I have a feeling you're probably right about stress and HT.
I strongly believe everyone needs to lift weights. For me it helps with stress. It's a great way to funnel all the days work BS out of your body.
I feel super energized and at the top of the world, even after a quick 20 minute session.
As people get older and they lose muscle/bone density they have a habit of falling down and breaking things, so it's never to late to start. That's my main long term concern.