More precisely it is taxation by citizenship, not residence.
For example, the Netherlands will happily tax everyone on their assets worldwide, but you're not going to be paying Dutch income taxes if you're not living and working in the Netherlands.
Both. Non-citizens (inc. illegal immigrants) are required to pay US taxes just for residing in the United States. The US will actually give non-citizens a unique tax ID (TIN) for this express purpose.
So the US will tax both based on residential status AND citizenship status.
That's... Slightly insane?
They don't really deport people that way, though. It could certainly happen in the future, but currently they're focusing on criminals and suspected gang members for deportation.