People of integrity with utilitarian leaning are often labeled amoral or unemotional.
People with a strong drive not attached to financial gain are often termed unambitious.
Your perceived worth really depends on values of society, so if you remove yourself from the equation, you only have no worth to others, not to yourself. Sequestered from others the sense of self has no meaning, so naturally you have no self-worth. Not because you are worthless, but because you have no self and place no importance on your perceived worth.
Or more strongly phrased, what's the point of self-worth if you can do everything you put your mind to?
I'd you can't but think you can, then that's a harmful psychological schism. Since no one person can achieve everything, self-worth is only meaningful in areas for which you lack competency
For example - and this deals not with self-worth but with stress - I'm pretty inflexible in my goals. When I can't meet my goals I tend to shut down. I'm often asked if I'm stressed, to which I can only respond, "Stress - what's that?". And yet clearly I've suffered a harmful schism between my perception of self and reality, as indicated by my lack of stress response.