Well yes of course, we should curtail the freedoms of bosses to be abusive, steal wages, and generally behave horribly to his or her employees in the same we curtail the freedoms of a physically large man to physically assault others.

And even if that isn't okay for you, there are plenty of ways to do this without even curtailing anyone's "freedoms" to be abusive: we can provide government jobs to all workers if they want them and require managers at government run institutions behave decently, for example, as we did in the New Deal.