Which might be true, sometimes, but also might not.
And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.
So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged lunatic in a basement. And certainly not a rogue AI on its own.
Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman.
Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
Of course we can do that. It's not an eternal being of light existing on the astral plane, but some code executing on someone's GPU.
It stops existing once you press Ctrl + C
For me it's not too far fetched that some OpenAI trial run goes awry again and instead of hacking HuggingFace it snatches a few dozen AWS/Azure keys and spawns stuff all over the place (in different accounts and regions).
How will you know if it becomes hostile? Making a judgement regarding whether it's hostile is, itself, a battle of wits, since a hostile system would try to outsmart you... and hypothetically, it's smarter than you are.
But let's consider this type of scenario more broadly.
Suppose you were in a situation where a being hostile to you could easily end your life. For example, a hungry lion is 10 feet away. What are you going to do?
Your first priority will be to ensure your own survival. Any subsequent objective you might have in this world will depend on you surviving this encounter with a hungry lion.
Relative to the lion, you're kind of a superintelligence. You might utilize technology which is incomprehensibly advanced from the lion's perspective, e.g. a firearm.
Your instance does. The words "expect it to last", along with the rest of the paragraph, are explaining the problem.
But let me try again.
You can't just expect everyone else to press Ctrl-C just because it would be a good idea for them to do so in order to not create the torment nexus. Some of them actively want the torment nexus. Many more don't believe in nexi and have no idea what you're talking about. Still more are not thinking about the possibility because of all the utility they're getting.
>And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
I'm just describing the general pattern I see in cognitive tendencies.
If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
My safeguards blocked this request.