Poverty is the absence of a lot of things. If you're born in a small isolated tribe in the middle of the Amazon, you're poor. Nobody took anything from you, there is almost no inequality, but you're still poor.
But I think I know where you're trying to get: "They're only poor because X took whatever from them". Probably where X is capitalism, Western countries, or whoever is the fashionable enemy.
Nope.
People can choose what to believe and all, but it's pretty hard to look at things like the African slave trade or Diamond mining in Zimbabwe or Sierra Leone and find anything but capitalism and imperialism run amok.
This doesn't absolve the misdeeds of exploiters - indeed colonies end in independence usually specifically because of mercantilist mismanagement squanders the true potential of the country by seeing it only as a well of land resources instead of an extension to nurture for mutual good but it is important to recognize that getting out "the Imperialists" won't make things better automatically and the wrong replacement can ironically be even worse. As bad as the British were in Rwanda they never decided to genocide the Tutsis even though their manipulations lead to it indirectly. Evil comes from within and without "the tribe" and it is important to recognize that.
Another response is "deprivation". Unfortunately, this response is effectively an appeal to the definition of poverty.
Since you asked what I believe, I don't believe there is a well-defined single cause of poverty.
If anything, Capitalism has lifted 90% of the world's population out of poverty. It's a solution to absolute poverty, not a cause. It may, however, be a cause of relative poverty. But to paraphrase Winston Churchill, it's the worst system for eliminating poverty, after all the other ones we've ever tried.