The CRUCIAL thing to note is that ICE gets stuff wrong. Their info is often stale or flat wrong - so even though they say "this is only for illegal immigrants, don't worry about it ;)", it can ABSOLUTELY affect citizens.
Note also that, since it's ICE and immigration officials (again: all executive branch) making these determinations, the executive is also deciding whether there's probable cause to think that an illegal immigrant is in a particular house. This damage to due process is ostensibly only aimed at immigrants, but it affects all of us.