You can be proud of the term as per your understanding of it, but if you use it as a description of yourself then you're relying on people sharing that understanding. From my experience that's often not the case.
pg's "Keep your identity small" comes to mind. But when it's a description of your mental status, it seems less like a label you choose and more like a label that chooses you.
In particular, it's different from wanting a smaller belly or bigger muscles or more money. All of those things are mutable. But being an introvert seems like a fundamental fact of your existence that's unlikely to change. In that context, isn't it better to wear it on your sleeve?
It's hard, certainly. But that's where a sense of community can help.
As a closing point, it wasn't until minorities started being proud of their labels that they made significant progress toward equality and justice. There's no reason to tolerate society's perceptions when they're unfair.
"Label pride" played little role in the success of the vast majority of success of minority groups over time in the US. At one time Chinese, Japanese, Germans, Irish, Catholics, Italians, Jews, and countless other groups faced hardships and issues in our, sometimes not so melty, melting pot. They invariably overcame by actions, not words. Their degree of success can be shown by considering the fact that when you refer to "minorities" you're mostly not referring to these groups even though they are even smaller groups than those you do implicitly refer to with the word. Again with the meanings of the words!