It's not that simple.
All of perception comes from noticing patterns. Labeling a thing, eg an apple, initially happens because you've seen multiple apples, or you've had it explained to you, which is using previous patterns (turned into words) you've already learned to manufacture a new pattern. Without pattern matching we wouldn't be able to identify or perceive things.
The mind pattern matches things. It's how we learn. We build abstractions, even when we don't know we're doing it.
The sense of self is an abstraction. Without abstractions we wouldn't be able to perceive anything.
(If you're talking about the sense of self as a negative, I'm going to assume here you're coming from a Theravada Buddhist perspective, but if not, I can adjust teaching/explanation style and terminology accordingly. Just let me know.)
The first fetter https://en.wikipedia.org/wiki/Fetter_(Buddhism)#Lists_of_fet... ie identity view / non-self / anatta, is being able to fully identify all of the details that build the abstraction of self, which can only be done during high levels of mindfulness from deep meditation. You'll find many of the things people attribute to self, are things they have, not things they are. eg, the skills you've learned isn't you. It's the skills you have. Though, can an abstraction really -have- things? (It's a failing of English, as this doesn't translate into English properly.)
So, you go through first path and identify all the things you think you are: body, mind, psychology, cultural beliefs, and so on, examining each thing, recursively breaking those things down, and identifying how they are not you, but how the idea of self is built off of it. Eventually you get to see how your identity isn't you, and the freedom of not identifying with things, as well as the suffering that is caused from identifying with things.
Anyways, you'll not see an arhat or zen master wondering around unable to interpret things. They just have fully mapped out what self is and isn't, as well as the other causes of suffering. They're not brain dead.
So is the sense of self a psychological disorder? Self is a delusion, not an illusion, so no. Sense of self (identity view, that is) just causes unnecessary stress.