What they are saying is a particular group should not be scapegoated, particularly when that group is taken advantage of to subsidize the quality of life of those that are demonizing them.
It comes down to having a nuanced understanding of the issue and rejecting overly simplistic (and wrong) mental models.
Why isn't the same argument being made about food? More people equals more demand, right? Shouldn't there be a commensurate increase in food costs? But people are able to mentally understand that immigrants are disproportionately providing the food supply. Not only are they more willing to do that work than native born Americans, they are also more productive in terms of both output and cost. If we were to employ only native born Americans in agriculture and meat-packing, costs are likely to go up, not down.
The same relationship holds for construction, especially in the south. Over 60% of housing costs are construction. Labor is a big part of that. Immigrants provide a disproportionate amount of that labor and they generally do so at lower cost. So by reducing immigration, you are going to increase the largest input to building homes in the hopes that you can bring down overall costs through reduction in demand.
Where that all balances in overall costs, I'm not sure. But I do know that overly simplistic models like "less immigration = lower housing costs" tend to miss those competing dynamics.
Of course costs will go up. So will pay. You're right in that in the case of food it's hard to say how that would all equalize. However, are we to rely on what's essentially an underclass forever? That seems pretty dystopian, and unsustainable.
Home prices I'd argue wouldn't be affected as much. The "dumb" labor in building houses is pretty cheap. Framing, roofing, etc. isn't much of the labor cost and they aren't on the jobsite for that long. Skilled work such as plumbers, electricians, and the like cost much more. There are kind of inbetween jobs such as drywall taping and tiling and I don't know how much of that is done with immigrant labor. In my experience it's basically none but I could be off on that.
Regardless about how much we export, prices are tightly coupled to supply and demand. If your theory holds, we should see a commensurate increase in cost. It doesn't matter if net exports go down unless there are effects from steep tariffs.
I agree that a large part of that many Americans are unwilling to do that work for that pay. And yet they are beneficiaries of that low-pay work in the form of low food costs. I find it particularly unsettling when those who directly benefit simultaneously demonize the immigrants who they derive some of that benefit. However, there have been studies showing that Americans underproduce migrant labor in industries like agriculture. So it's not just the pay, it the ability to produce. Similar work quotes farmers as saying they literally can't hire Americans to do the work. Unless I suppose you want to pay them salaries that risk farms being unsustainable financially. The extension of these points is if you were to hire Americans at an American wage, you get both less production and a higher cost while simultaneously risking the solvency of the farms.
I also agree that the use of an immigrant underclass is dystopian. Especially when you factor in the abuse that occurs in workplaces where employees are fearful of recourse due to their immigration status. And yet it happens and one of the often used excuses is that it's necessary to essentially subsidize a certain quality of life for Americans.
FWIW, it's usually termed "general" labor rather than "dumb" labor. You're point here seems to contradict your previous point that immigration also brings down the wages of skilled jobs. They way they do both is if they are represented in both skilled and unskilled employees.
As far as construction, this is largely region-dependent. (That's why I put the south in the original response. It's especially reliant on immigrant labor.) A rough rule of thumb is that labor is 20-40% of construction cost. Regardless, in my region, immigrants are the bulk of skilled labor as well. Again, anecdotal, but once you peel back the thin veneer, you can see how much of the American lifestyle relies on immigrant labor. And simply banning them in the hope that it increases housing availability will have many second order effects.