Except it's not really a "zero sum game with an ever increasing pie".
Even if the pie remains exactly the same, it's not a zero sum game: simultaneous value for two people can be created out of nothing simply by the voluntary exchange of goods and services between them while everything else remains the same, see my other comment here for examples:
https://news.ycombinator.com/item?id=33719823
As another example more relevant for this context, poor people can exchange their time for money, which can then be used by them to acquire other things they value more than their time (food, housing, etc).
So yes, poor people can get themselves out of poverty even if the pie doesn't change. There are many examples of this happening (i.e. poor people lifting themselves out of property much more than the pie grows).
But of course, they can do that even better if the pie increases, i.e. if we get better at producing food and housing even more efficiently.
But even if the pie remains the same, value can be created out of nothing by the simple voluntary exchange of goods and services and poor people can also greatly take advantage of that.
It doesn't mean that some people have to lose something when poor people gain something. Both poor and rich people can simultaneously benefit from exchanging goods and services between them, while everything else remains the same.
If nothing else, because rich and poor people value things differently (and even among each group, at any point in time each person also values things differently to a great degree).