This is why it is so important to have the courage to get up and leave. If one has been successfully sidelined then, yes, they will become an actual pariah because they feel like they have to explain everything that they do and will continue to do so because they feel like no one is acknowledging them. This is part of the gaslighters trap.
It should not be a surprise that there are some who might feel threatened by people who want to challenge the status quo in search of the truth or constantly push their mental and physical limits.