Here is one about striking workers masacred: https://en.m.wikipedia.org/wiki/Banana_Massacre
Or there is this Guatemalan coup d’état which was lobbied for by United Fruit Company: https://en.m.wikipedia.org/wiki/1954_Guatemalan_coup_d%27éta...
So, yes. I think it might just happen that buying banana from somewhere keeps them poor. These are so well known cases that I somewhat suspect you asked the question as a softball.
What would happen if there was no market for those bananas?
Maybe they find other ways to thrive, maybe not. Banan is not the only thing they could grow, and growing things is not the only way to make an economy work.
This is not an abstract economy problem from a econ 101 textbook. History cannot be rolled like that back and “what-if” is not a question we can answer.
What we know is that violence has been commited with the direct goal of securing said banana supply.
To be fair, it's not clear whether it would have had any impact by the late 20th century.