> Fundamentally, if humanity is given data set A, it will eventually generate dataset A' of 'higher quality' than A.

This same phenomenon happens with evolution, which is obviously not ‘intelligent’.

Each human can basically be randomly exploring a problem space near things they have seen and understand, but they have a way of knowing when they hit on something new and interesting.

Even a dumb AI could randomly try a bunch of things… as long as it doesn’t try the same thing twice and had a lot of time, who knows what it might come up with. Validating the solution is usually easier than finding it.