As an individual ("I need to get a visa and it's hard and there's nothing I can do about it") I can understand, but as a society/country? How are you powerless to change the situation?
Armenia has great desire to change its standing in the global passport rankings. However, it is under mounting military threat from neighbouring Azerbaijan. This forces Armenia to maintain very close ties to Russia, to have the assurance of their military protection. Being friends with Russia makes it difficult to also be friends with Western countries and improve one's passport rankings. Trying to sign an economic treaty with the EU can easily result in Russia withdrawing stationed troops.
Armenia requires protection, because it's a tiny landlocked mountainous country with no notable natural resources, surrounded by unfriendly and/or conflict-torn neighbours, i.e. with no reliable transport corridors. Meanwhile, Azerbaijan has oil and natural gas reserves, which it's used to build a formidable offensive military force and regularly declares its intent to attack.
Would you not feel powerless in that situation?
I used Armenia as my example, because I'm familiar with the situation there. But I'm pretty sure every other disadvantaged country has its own set of near-insurmountable obstacles. I say "near-insurmountable", because each country/society works hard to constantly chip away at those obstacles, and eventually, sometimes generations later, they succeed. But in the short-term, it's hard not to feel powerless.
At best, you're powerless regarding controlling the rate of change (and the changes required could take many, many decades). At worst there's people fighting against that change, preventing it from happening in the first place.
Who exactly? People from the country of the passport holder? People from other countries?
I try to get to the root issue here. Yes, it will take time. Yes, it can take many, many years and maybe won't be relevant for you anymore (but maybe for your children), but that is not "powerless" - that is "it takes time" and "it is stressful". So, is powerless a shorthand for those things or is there something else?
For you it's easy to say that it's not "powerless", but how many things have you personally sacrificed knowing that the outcome will be a few decades from when you started? And if you did sacrifice a ton, keep in mind and most people aren't like that and you're an outlier. So most people _will_ feel powerless when they try to change something and that thing doesn't budge for decades even.
That something can be political, economical, military, or social. But whatever it is most times it either takes too long to achieve, or is unreasonable. And in the end there's still no guarantee since the decision is with the "destination". Just look at all the agreements the US signed and how a single president chose to simply ignore.
There's the other option that you're part of a certain alliance that precludes you from being to nice to people in "the other" alliance. Enemies.
You might not realize it but most countries get a privilege like this because they have some negotiating power. They have something to offer. If you have nothing to offer you're SOL.
All this means that for all intents and purposes some countries and peoples are powerless as it goes.
If by "you" you mean a single person: To really affect the society (at the scale you can feel any echo of your deeds at least in a decade or so), you need to be an activist/to be heavily into politics. This is not for everyone. Most restrain even from engaging into discussions with the nearest circle to avoid being waived off as annoying.
If by "you" you mean the actual society: Society has the power when it has a realization of a problem and determination to resolve it. If it can't see the problem or doesn't understand the roots of it, it loses its power.
All those countries listed above, none had problems before foreign interests intervened. And most of this is true for any 3rd World country.
While you and I have the opportunity to make a change in our countries the same is not true for any country, you can't work by extrapolation. Especially when all you can do is struggle to survive.
This is not true at all.