I think that's contradictory. If you are setting out to save the world, but you can't improve factory conditions for your own workers, how have you shown the capability to save the world?
I'm reminded of having a strong AI agent that's given the condition "save humanity". That condition can be satisfied by having no humans on the planet to save.
It's irrelevant. Nothing you do for worker satisfaction in one factory demonstrates whether you can or cannot save the world. There is absolutely no contradiction in treating them separately. This doesn't excuse making workers unhappy, but it's really a separate issue.
A similar argument could be made about the Cold War and McCarthyism. It is simply nonsensical to mistake the goal of "saving the world" with the goal of "don't ever treat anyone slightly questionably ever again".
It's just not relevant. It's a non sequitur. It's "A is bad, B is good, you're doing A, therefore you can't do B."