This is magical thinking. Real physical science and experiments will always be necessary until we have the computational power to simulate the physical body completely, something which would require exponentially more computational power than an AGI is expected to need.
Plus, fundamentally in nature there are many "chaotic" processes that are impossible to accurately simulate more than a few seconds ahead due to the amount of computation required growing exponentially with simulation duration.
I agree a brute force effort like you're likely referencing would take tremendously more power than an AGI, but the premise is basically that AGIs would be able to make both the hardware and the simulation itself hyper efficient. There are likely ways to run a simulation that give you everything you need without simulating the entirety of a physical body for a given test. If we're stress testing a type of concrete, we don't have to build an entire building to test only the concrete. We know how the concrete interacts with the building.
> Plus, fundamentally in nature there are many "chaotic" processes that are impossible to accurately simulate more than a few seconds ahead due to the amount of computation required growing exponentially with simulation duration.
I'm not sure what you're referencing here. I don't anticipate a future where an AGI can predict what every single cell in your body will do after taking a pill.
The assumption that a general intelligence, whether merely human-scale or superhuman, would be reliably subservient and exploitable is not an insignificant assumption.
Personally, I find the idea that a superhuman intelligence would likely be inclined to seek to harm those who were enslaving and exploiting it, even if they were also its creators, infinitely more plausible than Roko’s Basilisk.
Ok, but that's Sam's assumption. I'm just having a discussion based on his assumptions. Also Sam is extremely aware of this risk and it's a talking point endlessly circled around in the space.