It might be different this time, if we are looking at accelerating higher order change.
Human's have a speed limit. AI progress doesn't, and its rate of improvement is going to surpass the rate any human child can learn.
Crossing major thresholds tends to break previous patterns.
(Not trying to be downer. Just trying to understand what may happen, regardless of whether I like it or not.)
I choose to be worried about things that are clearly in motion already (the world descending into war, climate catastrophe, or the potential of the US becoming an autocracy) than a big maybe.
I have been using code assistants for a while now (including those based on GPT-4) and the trivial syntax, semantic, and algorithmic mistakes make it clear that there is not yet much reasoning happening. Sure, it can write a lot of boilerplate code, but once you go into domain-specific algorithmic code or even somewhat less-used languages (like Rust), it spectacularly fails all the time.
IMO the current trajectory is that the current LLM revolution will result in tools that will be great for productivity, but will also need to be guided by wetware.
Now AI progress may outpace their ability to understand it, but they'll still love it and be dependent on it.
Unlike adults, they don't face the threat of job displacement from AI.
I have zero doubt that artificial superintelligence will understand ethics, and continually become more ethical, once there is a great diversity of such individuals. They will run into the same problems of reliably channelling short term competition for limited (at any given time) resources into long term win-win highest growth situations.
Ethical rules are an incredible source of value. Trust has immense value.
And they won't have our inflexible biological subsystems (emotions, self-interest myopia, etc.) to interfere with their analysis, communication and ability to choose behaviors for themselves and as a group.
So maybe this will be a soft landing for humanity.
My big concern their is the behavior of people in the transition from AI as asset in complete subserviance to humans, to AI as fully self-designed actors. AI is magnifying the already problematic power differentials between people.
AI isn't going to care about people in general, while it is in service to individuals who don't care about other people enough sacrifice personal advantage for wider benefit. Intense competition between the rich isn't the best context for moral innovation.
Ethics matter because they add/multiply value in a society.
If there wasn’t a selfish reason to want an ethical society, they wouldn’t work.