That reminds me of the argument about sentience lately. Why can't a neural network be sentient if all our actions have been trained by interacting with the world since birth?
There was a recent HN comment about this that I think illustrates the point well [0]:
experimentation is an act on the world to set its state and then measure it. That's what learning involves. These machines do not act on the world, they just capture correlations.