Anthropic addresses this head-on in the final section of the paper titled "What should we do?" If you convince the US government to slow AI development, you have to convince China too, otherwise you're not stopping self-improving AI at all, you're just throwing away the lead to China. If you convince China too, China or the US or both might go back on their word and build self-improving AI secretly, for greed of the benefits it could bring or fear the other will go back on their word.
What you really need is a non-proliferation regime like the one for nuclear weapons, where every country makes potentially dangerous AI illegal and lets foreign or international inspectors monitor to check that nobody's building illegal AI in secret. But monitoring seems hard; it's general-purpose computation. How do you check whether a given datacenter is training an illegal AI and not just serving websites, running detailed protein folding simulations, or mining crypto? For that matter, how do you know that a nondescript industrial facility hasn't been repurposed into a hidden datacenter for training illegal AI?
But we're discussing whether we should close the barn door while the horse is three miles down the road.
I realize he's saying it for hype, but if the CEO of the company goes around talking about how scared he is of what they're creating, hey, lets just take Dario at his word and put in some strict regulation. He won't mind if they're really about safety. (they're not)
Besides, yes, the knowledge of how to build these systems is out there, but the cost of doing it is staggeringly high (ie you can't run a frontier AI lab in your garage). There's only a limited number of known entities that need to be managed, and you can stop "progress" in its tracks by cutting off the money firehose.
Who is the "we" who is going to shut it down? Certainly not the US government. Nor the Chinese government w.r.t. their tech industry. Are you going to start the insurgency? Is there going to be an equivalent one in every developed part of the world?
All of this to say that the AI hype is not considering the energy portion of the equation enough. It won't automate everything not because it can't but because there is just not enough energy to go around unless there is a 100x or more efficiency gain just around the corner.
The stock boost is, as most will note, a bubble. It will enrich a lot of bad people and leave average people holding the bag, but its not going to go on forever.
Like, two? It looks more like the ladder being pulled after the incumbents got theirs than meaningful pushback. (And datacenters don’t have to be built in America.)
Right now I'm only having to direct to enforce good taste. Write tests, don't write an unnecessary function.
It does everything else practically. Presubmit, debugging, commit message generation, commit approval... it's happening.
In any case firms that get too powerful can be nationalised.
Probably a better chance the firm privatizes the government.
In fact we seem to be firing government employees and dismantling government institutions as much as possible.
No. Technical limitations aside, I doubt it could be contained, but will be leaked soon, so won't profit just a small number of ultra rich.
Doomsday AI is your interpretation.