Maybe I'm the weird one, but who would wear a t-shirt from your previous company to work at your current company? That just seems to send all the wrong signals.
My hot take: I've always interpreted it as "street cred" thing. Getting a job at Google is difficult, for example, and wearing a t-shirt from your Google days could show intellectual pride.
Pride here isn't a good thing.