Take this blog post for example, which between the lines reads: we don't expect to be able to align these systems ourselves, so instead we're hoping these systems are able to align each other.
Consider me not-very-soothed.
FWIW, there are plenty of AI experts who have been raising alarms as well. Hinton and Christiano, for example.
Nuclear weapons proliferated explicitly because they proved their scariness.
Anyway this is a good example of the completely blind-faith reasoning that backs AI optimism: we’ll figure it out “because it’s what we do.”
FWIW we have still not figured out how to dramatically reduce nuclear risk. We’re here just living with it every single day still, and with AI we’re likely stepping onto another tightrope that we and all future generations have to walk flawlessly.
We have figured out stuff in the past, but we also came shockingly close to nuclear armageddon more than once.
I'm not sure I want to roll the dice again.
And what does charisma of AI alignment folks have to do with anything?