As the sibling said, a good physiotherapist should be able to point you in the right direction in terms of which areas to focus on and what muscles you need to work out (i.e. which ones are too weak). In my case, I had done boxing for many years and I had the typical boxer's posture (shoulders tilted forward, round back etc.), and it also surely didn't help that I was sitting the rest of the day. Now, while boxing is great for your body overall, it does train the muscles on the front of your body a bit more than on your back, and to make matters worse the stance you take in boxing is asymmetrical, so that's where my issues where coming from: My bad posture was basically putting excessive pressure & stress on the nerves responsible for my arms & hands, which led to the latter constantly cramping up, being irritated & hurting, tingling, and so on.
So they had me focus on my traps, rear deltoids, neck musculature, and all the smaller back muscles in order to get me straight again (straighten my back, pull back my shoulders and head, et cetera).
At the same time, they also noticed a rather strong anterior pelvic tilt (which goes hand in hand with rounded shoulders), so I also worked on that through a number of core exercises (abs, lower back, glutes, etc.)
As you say, we can all try to focus on walking straight but we'll forget in no time. The solution to this is to train your muscles to remember for you. Now, to be clear, there's no magical memory in your muscles, it's simply about force balance: If your posture is skewed, some of your muscles are likely too weak, while others are too strong (and/or gravity is too strong), so your bones and joints get pulled in the wrong direction.
You say standing straight feels weird to you now. It did to me, too, because I had to overcome my weak muscles to stay straight, until my muscles became stronger and my new posture became second nature. Nowadays, I feel weird when my posture deteriorates again, so that's a clear signal to me then that I have neglected my exercises and need to put in more work again.
Finally, a word on the topic of ergonomic equipment, because I tried a lot over the years: Of course a decent desk setup (keyboard, mouse, chair, screen, …) is valuable and might help alleviate the pain but let's not kid ourselves: You are (or become) what you do most of the day, which among us techies likely is sitting. So passive instruments (sitting differently) rarely get you fixed 100% (likely not even 20%); we actually have to put in active work ourselves!