In places we didn't invade such as Mongolia and Japan, they're actually doing quite well.
Before you look down on the rest of the planet, consider that Europe is one of the best places to survive on earth. It's temperate, it's fertile, it rains a lot, it's geologically stable. Is it surprising that life is easy in Europe?
Compare that to North Africa, which is one of the least hospitable places on the planet: a huge desert. Installing western culture in North Africa would not fix anything.
Many people do in fact enjoy being "poor", as you put it, as long as they have everything they need in life. The western alternative to "poor" for these people is to work crazy hours in a factory. What kind of prosperity is that?