Are people really this afraid of what is essentially a fancy autocomplete? I read the letter but concretely what are the legitimate concerns?
The main problem I see is this being a next gen spambot.
The main problem I see is this being a next gen spambot.
It's starting to look like the achievement of AGI might be closer than we think. And that's an absolutely terrifying possibility, as the game theory is almost identical to nuclear weapons development, except with significantly more existential risk for those who don't achieve it first.
And never forget, you are just generalized fancy autocomplete with a semi-malleable goals framework.
Not a complete answer, but gives a good deal of context around the discussion, I think.