I have the same problem - unusually skinny, baggy clothes make it look worse. Levis fortunately fit me, and European brands like H&M are fairly reliable.
Protip #1: try clothes in store, buy online via Gilt or one of those discount shopping websites. When they have something you want, the discounts are often amazing, though you have to make quick decisions because small sizes often sell out fast. Most of these sites are very good on reporting the sizes correctly or helping you convert US-EU.
Protip #2: you can't wear jeans and a nice-fitting black top your whole life unless you're Steve Jobs. And designer menswear can get very pricey...but a good tailor can make an something with an average fit look absolutely perfect. Buy sizes that fit comfortably where you're largest (eg my neck wants a 38 shirt even though my shoulders and chest aren't) and then pay a few $ extra to have the garment altered. This might add $30 or $40 to the price of a shirt, but suddenly you'll have access to a world of things that don't exist in your size, and it will actually be worth looking at what's on sale.
the discounts will easily pay for the alterations, and after a while you have a bunch of stuff in your closet that fits well, whether casual or business. Dressing actually becomes enjoyable instead of a chore.