It's the opposite. Outsourcing gives an opportunity for people in poor countries to earn money, bringing their wealth up.
Just look at Mexico and the US. In the US, auto manufacturing labor was paid $80k+. I don't know the exact numbers for Mexico, but it seems reasonable to assume people in Mexico typically work jobs that pay far less than $80k. Automakers go to Mexico and say we'll give you $40k to do the same work we are paying $80k for in the US.
For people in Mexico, $40k is a much better option than anything else available, so the people in Mexico jump at the opportunity to earn $40k, and are wealthier because previously they were earning less than $40k (by definition, otherwise why would they jump at the opportunity to earn $40k). People in the US are now earning $0.
Quality of life has increased for many, many millions around the world due to outsourcing, and one could make the claim quality of life (or at least security of income) has decreased for millions in the developed world. But this is arbitrage at work. Of course, labor arbitrage isn't the only type, there's also environmental arbitrage (which is bad for the world).