Also if I'm at any sort of business or social event and someone is comes up to me and says something like: "Hey bro, I've been sitting on this great idea for a startup that I came with when I got my MBA at Oswego College, and I need a cofounder. Here, sign this NDA and I can tell you about it. Okay fine, don't sign the NDA, bro. It's called 'Fratastic.com,' it's like frat-oriented humor videos. It's gonna totally be the next youtube, it's ridiculous how much money it's gonna make, bro. I have the domain name and everything, I just need someone to actually do all the coding and I'll do the marketing, bro. How does 4% equity sound to you?"
... well, that person is almost always wearing a shirt, coat, and tie.
So why would I want to dress up beyond a t-shirt and jeans? It makes me uncomfortable, it has nothing to do with my production, technical or otherwise, and I'm less likely to be associated with the expensively educated clueless douchebag I described above.
Also, I know a lot of hackers who do go to the gym or otherwise keep active and look just fine in a t-shirt.