If most of the population doesn't participate in the exchange, money is worthless. You can make numbers, but those are meaningless without the underlying exchange.
It's effectively a disruption of a modern society.
There are many millions of people in the USA who don’t work but own things. Some don’t work and own a lot of things
Say, you have 98 people who have literally nothing and no established way to obtain basic needs, and 2 people who has something based on whatever.
Say, these 2 people can close their needs by asking a magic machine to do something for them, or even exchange some goods between each other, but there is no economic sense to hire anybody.
Those 98 left aren't going to sit and die, watching.
Also what if it’s not 2 vs 98, but something more like 10 vs 90, or 30 vs 70? Or 50 vs 50? And this kind of thing doesn’t happen overnight. It happens over a generation, or at least a few decades.
Anyway, the point isn’t this will happen, the point is that really bad outcomes that are almost impossible to walk back from are completely within the realm of possibility. And we have no real way to predict where we’re headed. So people are rightly very afraid.