the steelman is that once you have AGI, we don’t know what happens next.
With other technology, you typically get some breathing room before problems arise - with AGI, pretty much everyone seems to agree step two once you have it is going to end up being
> someone spins up as many copies as is feasible and has them work on their smarter successor, then have those successors spin up as many copies as possible building better successors.
We’ve only ever seen human speed progress, subject to human speed limitations and human constraints. It’s possible that after a few successors the machines are much, much smarter than us, such that scary tasks we’d find challenging and technical (creating cripplingly damaging computer viruses, sabotaging infrastructure, creating life-like or real-time simulated digital communications of people you care about) could be trivial.
We don’t know how fast we go from “it’s equally smart as us” To “humans are no longer able to intellectually compete at most, if any, domains.”
Is that a problem? Maybe! Digital entities significantly smarter than humans could mean a true end to scarcity, a solution to basically any solvable technical problem we can enumerate, and a new glorious age for mankind. It could also be absolute destruction at the hands of an adversary that essentially is immortal, unkillable, of perfect memory, and who has access to the capability to gain control over essentially everything connected to a digital network, which rounds up to “everything”.
The problem seems to be
> it depends on what you’ve designed such an entity to do or what such an entity would WANT to do
And that’s scary because we don’t have any precedents for what happens when humans are confronted by significantly intellectually more powerful entities than our selves - besides how we treat animals, plants, insects and other life we see as intellectually inferior.
Judged by our past track record, assuming machines can establish their own actions with intention in basically the same way humans do, the interaction isn’t likely to be amenable to human thriving rather than machine thriving, and that means life looking more likely like “you’re the machine’s factory-farmed chicken, or if you’re lucky, its pet” rather than “the machine dotes on you and brings you into a new gilded age of plenty.”