What's the empirical evidence that proves this? If the economy gets worse as a result of lack of immigration, its going to reduce opportunities for low income and working class people.
You’re making an assumption that immigration lead leads to a better economy, ignoring the possibility of more competition for jobs, lead leading to less power and wages for workers.
You're focusing on one side of the supply-demand graph. Immigrants increase both the demand & supply side, they require more jobs but also the increase in population increases demand for more products and services. There is no fixed amount of jobs in a country.
I do agree if a country were to let in 100 million immigrants over night or some insane number that it would mess up the economy, there is an upper bound to this.
I would also introduce the concept of culture – if the economic indicators seem fine, but the culture is becoming balkanized and sectarian, then I would argue you still have a problem driven by quantity and or pace of immigration.
How do you measure culture empirically and how can you empirically say that culture is being balkanized specifically by immigrants and not other factors? You can't just take anecdotal evidence to say okay no more immigration because you have big feelings about it. There are places in the world still where having a non-zero amount of people of a different race moving into the neighbourhood would trigger some people to make the claim that the culture is becoming balkanized and sectarian.