More specifically, I think we're worried about AI taking our incomes, not our jobs. I would love it if an AI could do my entire job for me, and I just sat there collecting the income while the AI did all the "job" part, but we know from history (robotics) that this is not what happens. The owners of the robots (soon, AI) keep all the income and the job goes away.
An enlightened Humanity could solve this by separating the income from the job, but we live in a Malthusian Darwinian world where growth is paramount, "enough" does not exist, and we all have to justify and earn our living.