Protectionism isn't going to help workers in the long term. We live in a global market; accepting that and using legislation to deal with negative outcomes is a totally valid approach. The fact that it hasn't been done is because of the large amount of money standing in the way of it.
There can absolutely be a middle ground. We can tax rent-seeking. We can quit the austerity focused trickle-down policies that have been shown to fail, and put money into the hands of people who will actually spend it. We can actively redistribute wealth.
Why would you rather give up protections that cost our ancestors their lives to establish?
You can send your factory bucks overseas a lot easier than a poor person can move to a country with a higher standard of living.
We need to start asking the question of whether our legal systems should be friendly to capital at every opportunity.