We should be pursuing the opposite, and accelerating efforts to build what will be some of the most transformative technology humanity has ever developed (and some could argue, NOT building has its own existential risks).
This way one can apply it to e.g. putting GoF-capable virus labs in every city in the world. After all, innovation is just good and stopping innovation is just bad (and maybe an existential risk itself, with no attempted explanation as to how!).
The real magic (and the thing that will, I promise, eliminate all risks to all humans) will be when every household has its own automated virus synthesis lab and it’s just hooked directly up to an AI so you can have it synthesize contagious molecules from plain English. This is Good and not Bad because it’s Innovation!
“Technology” is what we call it when we devise ways to mutate our world more effectively. AI is a promising technology because it promises to dramatically mutate our world. Why should we believe that this power is only wieldable in the “positive” direction? In answering this, can you please also articulate what direction is “positive?”
The in-home virus labs, for example, could certainly be used for writing viruses that make humans genetically immune to all sorts of nasty things. Powerful! Do you just accept my assertion that this is what that technology will actually be used for? I suspect not.
You already have an in-home mustard gas lab, plus the few thousand other noxious chemicals you can make my mixing stuff in kitty litter. I get what you're saying, but nobody would ever install an in-home virus lab because it's basically a threat-vector with no palpable upsides, even without AI. Maybe an in-home 3D printer is a better analogy, since the worst thing it can do is print something profane or offensive.
> Why should we believe that this power is only wieldable in the “positive” direction? In answering this, can you please also articulate what direction is “positive?”
For starters AI doesn't do much of anything. Like any other system, you have to connect it to different components for it to design the human flamethrower or supervirus you're imagining. An AI concocting supervirus formulae and flamethrower blueprints isn't inherently positive or negative - giving AI the capacity to act on those ideas is. It really is as simple as "imagine the liabilities beforehand" with our current systems. If you're not prepared to handle the worst-case disaster scenario, you probably shouldn't let AI do it.
At least, that's where I stand on it. I'm a pragmatist about it, I don't think AI's progress is immediately threatening. Come throw rocks at me when I'm wrong in 5 years or whatever, but I'm not sold on the Matrix-style human battery future everyone seems to be resigned to.
I see the problem more like, a fail is an all or nothing fail, that’s it, everyone dies, maybe including the ASI (goes rogue and destroys itself and us), then what? Good outcome for anyone if we don’t understand what we’re doing?
I’m not saying you’re entirely wrong, or we need to stop, but I mean, we do share a planet with other species and children who probably have good lives ahead of them. So why rush into potential disaster, what is the actual hurry?
I know that many Silicon Valley elders are getting older and facing the inevitable, they see ASI as a way out of their own mortality. They’ve believed this for many years if you look at old footage, so giving up that dream is probably very hard to do at this point. But I do see it as a large reason for the push. Time is running out for the mega wealthy. I honestly empathise with them too, but I don’t think it’s fare to say everyone wants to deal with the risk.
I empathize with everyone damn it.
I just don’t want to witness the “he chose poorly” scene at the end of Indiana Jones happening for real on a global scale.
I guess we can develop super AI then ask it to solve it. :P
How will we know when it's true AGI? Easy - "computer says no".