We have no scientific evidence of the contrary. There are some studies pointing to mass migration being (slightly) detrimental to the wages of low-income natives, but that's pretty much it. You will find a lot of opinions though.
Why is "mass migration beats down the working class" not bad enough to warrant restricting it?
I think you would probably not, because you know it would be a war zone within weeks, with the poorest and most desperate people from all around the world swarming to your small part of the world, bringing with them huge social issues.
People who have the wherewithal to move across the globe are not "the poorest and most desperate people from all around the world", they are the ones with the means to pay thousands of dollars to mafias because we have no better system to bring them in.