The bigger problem is trousers - generally it doesn't matter how fit I am, the size of my thighs (muscle again, played rugby from a young age) will stop any trousers from fitting me properly. Anything that fits around the thighs is guaranteed to be several inches too big around the waist (no, a belt doesn't solve this all the time: ruins cut and comfort).
The problem is as much with the attitudes & economics of high street fashion as it is with you. I don't get as frustrated and depressed shopping when I'm in better shape physically but it's still a very unpleasant experience - and that's without even thinking about style.
Most people say "go tailored" with all the obvious expense and silliness of asking for tailored jeans etc. Buying and adjusting is more realistic if you or someone you know can sew or wants to learn (or maybe affordable professionally). But again, sadly, many high street clothes don't have sufficient build quality or extra material (at least one tailor refused me on that basis).
Always seemed to me that there is a business opportunity for providing web-ordered clothes. If the fittings were better than a single number, and were consistent, it would be a great service from the consumer point of view. Perhaps difficult to get started from the business one though, before the volume is large enough/process easy enough?