Some might fall into the category you've described, but I'd hazard a guess that a lot more are afraid of the rise of AI due to its owners. The cost to develop and operate these machines is high, and you can be sure whoever is using them to replace work done by people today will capture and hold every possible penny.

People are afraid that AI will not serve the common good, and will instead serve a very rich few. Why? Because that's how it's always been with new advances, and more than ever how it is today. The vast gaps in wealth inequality will grow much larger with AI - it needs to be addressed first.