Given that ICE has deported US citizens before, pretty sure it's a reasonable concern for immigrants to be concerned about ICE agents.

My opinionated and probably-biased answer to your question:

1. Due to their being no real central database, it's hard for physical persons in the US to prove their identity and consequently their citizenship status.

2. There are a lot of illegal immigrants in the US.

3. There are a lot of illegal immigrants in the US that have lived here for a long time so the "legality" of their status is muddied. This then makes the whole debate around this more about "what is right" and the emotions involved, instead of what the laws say.

Allegedly, in the timespan 2003 to 2010, the US detained (in some cases for months) or even deported over 20k US citizen:

https://jacquelinestevens.org/US-Unlawfully-Detaining.Steven...

Not in the US but the UK. [0] a large group of people who moved here legally between 50 and 70 years ago have been deported, detained and refused legal rights. This does happen.

[0] https://www.jcwi.org.uk/windrush-scandal-explained

If someone is honestly paying their taxes, why would they fear having their finances uprooted by being audited by the IRS?

There's no "papers" in the US. You're not required to have a birth certificate or a Social Security card or a drivers license or a passport. There's no list of who is and isn't a citizen.