What does this have to do with his tenure on the FSF?
RMS may not have much in terms of money, but he has a lot of influence due to his stature in the libre software movement, and with influence comes responsibility and accountability to those in the movement he started.
Why is his goal the only one that matters? Is he a prophet, or is he just a guy who created a compelling idea? Should the movement continue to be centered on him as a personality? The FSF movement is by definition far bigger than him, and it can reasonably contain other related goals. Perhaps those with other goals can make peace with keeping him around, perhaps they can't.
But I would understand if people that are part of the FSF would not like to associate with him.