IMO it is just a cultural reflection of our general safetyist cultural.

Safety has become a near sacred value. Near my house a new light at a cross walk has a computerized voice that says "stop stop stop" , "walk walk walk". A cross walk to a dead end street with basically no traffic ever.

I can almost hear the city official who had it installed "if it saves just one child's life..." even though it is a near certainty no accident ever occurs at that cross walk either way.

AI safetyist are basically that city official to me. What you posted is besides the point. It is about lowering a probability no matter how remote that probability is and even if the whole exercise is basically pointless. The gods of safety will be pleased by our offering.

> The problems that AI will manifest are the result of human ambition and failings, no different as any other technology that empowers individuals. Yes, individuals and organizations will misuse AI for immoral purposes, but the popular belief that AI is inherently antihumanist launders accountability by pretending to remove human agency from the equation. How we use or misuse AI technology is entirely on us.

So what? If it kills us, we're still dead.

> I would argue there are strong complexity-theoretic bottlenecks to computational processes which limit the expressive power of neural networks

Of course there are physics/CS limits to how intelligent something can be in a given volume, but those limits are vastly higher than our own so I don't think they are particularly relevant. For instance, a system which could simulate the brains of a thousand scientists as smart as Einstein a billion times faster than realtime would not violate any rules of physics, even though it is far beyond our current capabilities.