That sounds to me like "if we don't let our suppliers use child labor, those children's families will starve" argumentation.
We in the west are complicit in the use of child and slave labor when we buy products produced in this manner, but because it happens on the other side of the globe and is cheaper that way, we turn a blind eye.
The solution is not to ignore the issue. The solution is to draw a line in the sand and say "this is unacceptable, clean up your act or no deal", and for that to be backed by international trade agreements. It is up to us, as the lucrative market they wish to sell to, to make these demands.
Full transparency and accountability in all supply lines is a must. It should not be possible to hide behind "a supplier did this, we're not responsible".
I hope we'll see a move to more local production, instead of shipping stuff around the globe like crazy. It makes no sense for me in Europe to have my clothes produced in Asia, when we have world-class textile and garment production right here, with better environmental control and worker's rights.