1. If it never happens or AGI is friendly then we won't need to. If a hostile AGI is created, one trained on all writing about war and strategy and deception, able to respond in microseconds, with has ample time to plot before it makes an obvious move, then thinking you will wait until you see it attack and then turn off a datacenter, is way too little, too late. Like saying "the nuclear bomb explosion is local to one place, if it hurts, we will just walk away from it".
2. That claim does not hold water once you start to dig into who, specifically, would turn off which specific billion-dollar datacenter, under which conditions. Google has been moving compute jobs around the planet to get nighttime electricity rates for bulk compute jobs for years, the idea that there is "one datacenter" is far too simple. The idea that employee 74,123 at Meta doing AI research is also able to get a Meta datacenter shutdown quickly on request, or would know enough about Meta's infrastructure to know which datacenter, seems unlikely. The idea that it even has one off switch, or that nobody will quickly power it back on, or that the compute jobs will not pick up automatically somewhere else, or that nobody else on the planet will be at a similar point in their research, also seem important.
3. This assumes there will be a simple critical point that makes it clear we need to "turn off datacenter now". The fact that you then go on to talk about climate change, imagine you had said "we can just turn the factories off if climate change becomes a problem" and maybe that will make it clearer. Who will turn which factories off, at what point in the climate catastrophe, and why haven't they? That's why it won't happen with datacenters either.
> "Like we're not going to have some runaway skynet thing, that's just completely off the table."
You're confident that tens of billions of dollars of investment and research, US v China national AI competition, some of the largest companies in the world researching AGI will be a flop and develop nothing? Despite the progress from GPT2 to present day, despite historic analogies like the Wright Brothers to the Moon Landing in one lifetime?
NVidia shipped an estimated 40 million GPUs in 2025. Say they average 1 TFLOPS each and a year has 31 million seconds, that's 1.2 million million quadrillion calculations in a year. That much compute power piling up around the planet year-on-year is not going to be important or relevant to the near-medium term future? Because it's just "off the table" that anyone could invent anything interesting with that much growing compute in the next ten or twenty years?