Consider that most of your indirect colleagues are not well-versed enough to know whether you're doing your job well or not in the first place.
Why are so many clowns in positions with high rank, and so many better-qualified people are further down the totem pole? Because people don't respond naturally to qualifications. They respond to getting their personal interests and desires satisfied.
If you present a credible professional front and you are well-liked, that's all that matters, and you will be well-connected and rewarded well outside of any rational proportion.