AGI is a break-capitalism lottery ticket I'm willing to entertain. If the marginal cost of labor collapses to zero then either we'll get robot police or finally be free to pursue lives of leisure.
Not saying there aren't good points. Only pointing out that many, many people can all view the consequences of this dynamic differently.
I am replying to the parent comment, but I too get that this petition is being signed onto by people at openAI specifically because they already made it with GPT-4. They definitely would not sign on to it if their (arbitrary) threshold for what to stop doing AI research with was a tool at a GPT-3 or GPT-3.5 level or higher or it included GPT-4 in their threshold, and not merely beyond it.
[0]: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
As it stands, it says more about the author's egoistic view of humanity than of AI. Imagine for a second taking Feuerbach's The Essence of Religion and applying it to Descartes' rationality and Plato's nous. We'd get a critique of the rational human's construction of his own ideal - the essence of intellect.
AI threatens this ideal and ego-threats get sublimated into existential threats by those unable to understand themselves well enough to express themselves directly.
People write page after page about how it might kill us all in a flash, without ever offering a good explanation of why it would want to. My experience with people whose immediate reaction to something they're scared of is to destroy it is that they're panicky screechers who are an annoying distraction to the person actually handling the situation, whether that's wild animals, fire, or interpersonal violence.
I'm not saying 'just let me handle it bro, I totally got this.' There's a lot of potential risks, I don't think anyone is qualified to say they can mitigate all those, or even most of them. But I trust a machine intelligence - even one that's vast, cool, and unsympathetic - far more than the would-be Butlerians.
The point of much of the alignment debate is that people like Yudlowsky are pointing out that it doesn't need to want to, it just needs to not not want to enough.
You're hoping for an outcome ranging from "When The Yoghurt Took Over" to "The Metamorphosis of Prime Intellect", but many other people are expecting an outcome more similar to gwern's "Clippy".
Two, I would generally not set aside an hour of my time for a video without at least some indication of what it's about. I'd rather spend that time reading than in the 'hot' medium of video.
Three, I find this video deeply suspect. It purports to document a 'private' gathering, yet it's clearly a well produced event that was intended to be documented and shared on video. People who actually want to keep a thing private are generally well able to keep it private. So while the participants have a valid point of view with many legitimate arguments, the facts of its existence suggest to me that it was absolutely intended to become public, and the 'private gathering!!' is essentially a marketing hook.