For example, let's say immigration reduces the cost of housekeeping services from $20 per hour to $15 per hour. That's great for an upper-middle class person who saves a bit of their weekly cleaning bill, but a lower-middle class person saves nothing, because they clean their own house. And the people who were displaced from that semi-skilled industry are now likely looking for jobs in other semi-skilled industries, depressing wages across those classes of jobs.
And of course immigration also contributes to rent inflation and increases in the cost of real estate, which primarily benefits those with significant capital in real estate (again, disproportionately the very wealthy) while driving down disposable income for everyone else.