I think it's quite the opposite, and I'm glad this issue is finally getting the mainstream attention it deserves. I can kind of understand someone seeing a single headline, and thinking it reads like the rantings of a crazy person on a street corner proclaiming, "the end is nigh!". But after that initial thought passes and that person digs deeper and realizes that this is a legitimate concern that AI researchers have had for years, and that they have good reason for it, then I can't really understand how anyone would think those people shouldn't be taken seriously. I mean, you use AI, right? So you have no problem trusting these people when they are producing something you like and enjoy. But when it comes to something you don't like, it suddenly becomes, "we shouldn't take these people seriously." That seems like nothing but wishful thinking.

We do have strong evidence, by the way. The hugging face attack is the evidence. That's why this is all coming to a head now, despite the fact that leading AI figures have expressed these worries many times over the years, since before ChatGPT was even released. We don't even need that kind of evidence though. It follows from logic that if you take two entities with different goals, the more intelligent entity is more likely to have their goals realized. As long as AI companies are trying to build more and more intelligent AI, and succeeding in doing so, then we have reason to fear that it will soon escape our control.

Perhaps if there was some wall all the companies were hitting in regards to intelligence, then perhaps the fears would not be so urgent. But each new frontier model continues to outperform its predecessors. We now have Sam Altman and Dario Amodei telling us that recursive self-improvement will be happening in the next year or two. The frontier models are already better in most subjects than most humans. If they get to a point where they are improving themselves, then there's no chance we will be able to maintain control of them. At that point, it doesn't matter much what laws we enact or what measures we take.