I'll be the first to admit that the disadvantages outweigh the benefits for specific workers. NAFTA sucked for autoworkers. H1B visas suck for IT and software workers.
I think other types of workers benefit more than the portrait you paint, and not just the capital owners.
Unless you are a utilitarian (I'm not), I agree there is a valid debate on how much policy should disadvantage a small group for "the greater good".