But even so I think we can't afford NOT to have this kind of development if we want to survive as a species. If we want our lives to improve, our loved ones to live longer healthier lives. The amount of good that can come from this technology is immeasurable. The only thing we as humanity need to do is reign in our ego just like chess and go players had to do when their crafts were rendered supposedly irrelevant in the face of an insurmountable intelligence.
And I think the risks from AI are overblown, is AI really more dangerous than the invention of gunpowder, electricity, and of course the nuclear bomb? I don't think so.
What's scary is that we live in the transition period and it could be messy. What comes after the "transition period" of having a world without AI to having a world with AI? I'm not going to try to predict that but I think there are as many if not many more positive outcomes than there are negative outcomes.