I think it comes down to what you expect from people who are being treated poorly. Do you only accept "pure" moral acts as valid? Is small unfairness to a few justified to prevent large unfairness to many? It seems like we have different answers to these questions, but I don't think either can be dismissed as inherently against worker freedom or agency.
I care very very much about practical power for people who need it right now. I care a lot less about abstract freedoms that material conditions prevent people from being able to act on.