And, believe it or not, those people working in foreign sweatshops want you to buy their products. They work there because that factory job provides higher pay and a generally safer work environment than the rice-paddy (or wherever) they otherwise would have been working. I think they would find it a bit patronizing and naïve if you were to tell them they should leave those jobs behind. To people in poor countries, a job at a factory is part of the path to a better life. For the country itself, it is part of the path to industrialization and prosperity.
Also remember that just because you could make something better, it doesn't necessarily follow that you must to act morally. Much of a rational notion of morality involves avoiding acts that cause harm to others or that violate the person or the free will of other individuals. All acts are not only moral or immoral. Some actions are supererogatory -- morally praiseworthy but not morally necessary.