So you just turn up in jeans/t-shirt to listen to some Rachmaninoff?
Wearing a suit feels a bit pompous, but it would be interesting to hear what the musicians and conductors think about the audiance's attire. I assume most don't care.
That said, I'd encourage anyone to go to a concert wearing jeans vs not going to a concert. The music is what's important.