Were you ever as competitive (that is, skilled and practiced) in any particular physical sport as you were in chess? I suspect that if you'd put as much of yourself into, say, tennis or soccer, that you'd have felt similarly.
EDIT:
I've been pondering this a bit more, and I want to share my thoughts.
I was thinking about how some of my friends react to losses in mental sports/games versus physical sports/games and their particular histories, as well as my own.
Playing a bit to the idea of dualism of mind and body, some people focus more of their energy on the mind and some on the body. This affects their notion of self. For you, your focus was on chess and the mind, consequently a loss in that game or evidence of your mind not being as honed as your opponents strikes your ego more than a loss on the tennis court might. I have teammates on a rec soccer league who take it very seriously. They spent their youths practicing soccer or another sport at least as much as you spent on chess. A loss on the field, or a near loss, hits them hard because their sense of self is so tied to their physical capabilities. Anything demonstrating that they aren't as good as an opponent hits them harder than it would me. I'm a tad bit more like you (though never seriously competitive in mental games/sports), my particular focus in those years was on acquisition of knowledge and understanding. This is not a directly competitive thing (outside of trivia games like Jeopardy!), but meeting someone who understands a field, that I've studied, far better than I do is both inspiring and a blow to the ego. "I've spent X hours, days, months studying this, and still I'm nowhere near this person's level of understanding and capability."
All this is not meant to say that your view is wrong, but that what each person experiences as a hit to their ego is going to be different, and largely determined by what they put their energy into. Outrun a former sprinter and they'll feel the effects of their age and it'll weigh on their mind. Beat an old and unpracticed chess champion and it'll be the same. But beat the sprinter at chess and they won't care, and the chess champion might even laugh at the suggestion of a footrace assuming they'd lost before it began.