If you think you can pull it off, I've become a strong proponent of blazers and sport coats. I call them "wearable purses for men".
For me, this is basically an evolution and refinement of what I guess I've always been trying to do. As a teen, I'd wear un- or partially-buttoned button-up shirts over my t-shirts. Later, zip hoodies as much of the year as I could get away with. I'd also be sad during this whole period when it'd get too warm and I couldn't get away with wearing a coat everywhere (and it got too warm for a hoodie, once I adopted those) and I'd have to go back to putting my stuff in my trouser pockets. That just sucks.
Now, I've gone to blazers and sport coats, which to me feel like a very similar idea but are wildly more practical[1] for most situations I find myself in, and most folks perceive them as looking a lot better, too.
I've found the heavier winter fabrics to be the easiest to pull off. Simple tweeds, or I've got this one really heavy but soft wool twill in grey-black that works great. Corduroy (there are wool corduroys, too!) covers fall really well, and can do for the cooler parts of spring, too.
Summer is trickier. Unlined Summer-weight wools can add effectively zero heat (and keep some of the sun off you!) but may look too formal, especially when nobody else is wearing a jacket of any kind. Really schlubby unstructured cotton, or a simple linen jacket can work without looking "too fancy", but they're a bit high-maintenance if you don't want them to look wrinkled to hell all the time. I haven't quite found what I'd call a perfect solution for me, here.
[1] Why are they more practical? A blazer or sport coat can usually dress pretty far up, and pretty far down, so has better situational range than a hoodie. I also find the pocket layout is just a ton better. My standard way to wear them is keys in left-hip, phone in right-hip, wallet in left-inner-breast. Generally, these leave tons of room in all three pockets concerned for cramming change or receipts or whatever, none of them are crowded. When traveling, I'll give a whole pocket to my passport, right-inner-breast if that one's big enough, or I might give that to my left hip if I don't need my keys (like when flying or taking a train). Sunglasses can go in the outer breast pocket. Hip pockets are usually big enough to temporarily cram leather or knit-wool gloves in, for the winter, when I need to take those off but don't have anywhere to put them, without overcrowding the pockets enough to cause a problem. I can double up my keys and a big ID/keycard in the left hip, when I need to go into the office, still not crammed tight or anything. The hip pockets are also great for those "pocket size" paperback books, and you can even fit stuff like Modern Library hardcovers in there as long as they're not super-thick volumes, especially the older editions that had smaller dimensions.