Isn't lower class work interchangeable to some extent ?
Less illegals to do construction means a larger portion of construction it will be done by legal employees, which reduces the available pool for manufacturing.
In other words: you can fill part of your staff with interchangeable workers, but not all of it.