That's a very valid reason to destroy technology, and the very logic behind e.g. nuclear disarmament agreements.
Technological products are not gods or inevitable forces -- it's just stuff we make, and that we should be totally free to break or stop making if we don't like the effect they have on our lives.
It should be us and our desires and needs that shape our societies -- and technology should just be a secondary concern that provides the tools to help us do what we want, not an end goal in itself to force us to accept consequences or ways of living we don't appreciate.
Besides, the Luddites weren't merely "afraid of what it might bring". They were actively disenfranchised and left to starve but what it brought to their communities. It's easy for privileged $100K/year SV devs to talk about people re-inventing themselves and investing in new skills, and so on.
But whether its hand-to-mouth paid workers with few other options in the 18th century, or the people in dying ex-factory towns and communities, it's usually others that benefit from any "new development", and they just get the short end of the stick: unemployment, starvation, fall from middle/working class (and things that come with those: broken marriages, alcoholism, etc).
>calling the 1800s Luddites on the wrong side of history.
It's easy to gloat and be "on the right side of history" when you're benefitting from a change. Less so when you're destroyed by it.