I'm not even convinced that illegal immigration necessarily harms low-skilled domestic workers. At least in California's Central Valley (with which I'm at least somewhat familiar, having grown up there and having family and friends there), there's a significant shortage of agricultural labor because illegal immigrants are facing crackdowns while legal residents seem to have no desire to fill the positions even at $20+/hr wages. The only option farmers have left is automation, which would actively reduce the number of jobs available and be just as "harmful".

Taxpayers might be harmed, but I suspect they'd be harmed much more severely by food prices skyrocketing because farms don't have sufficient labor to keep up with the demand from a growing national population (let alone global; California is the leading - and in some cases the only significant - exporter of a staggeringly-large number of fruits and vegetables).

Meanwhile, cracking down on illegal immigrants' ability to work and live in the US gives employers of said immigrants more leverage ("work these long hours for chump change or we'll report you and your whole family to the authorities"). Amnesty programs and other means to convert illegal immigrants into legal immigrants shifts at least some leverage back to the illegal worker, which then allows better negotiating power for higher wages and thus addresses the supposed problem that illegal workers are undercutting legal workers.