At the end of the day, even if they are some insane oracle (pun intended), they're still bounded by training data and how it relates to the real world. Even if they're a near perfect tool, we are still the interface between them and our lived experience. If that stops being the case then why do we care about the output?
This assumes it doesn't graduate to just killing all of us and doing it's own thing, but within this paradigm it doesn't really have goals.
Until they are embedded into automatons that can interact with the world.
(To be clear, I'm both on board with this, and I think it's the natural progression. Currently they are bounded by their training set, but interaction with the world is the imperative for theory -> test -> analysis -> update that is essential for growth and creativity.)
Yes and no. They can extrapolate and build upon the training data, as was the case with the last dozens of math proofs
To put it differently, if you have some idealized model in front of you that can do anything a team of humans can do, what do you say to it? It's still just a model that takes an input and provides an output.
Someone will still need to ask for something (or maybe it will just run autonomously making things it thinks we'll like) but it won't require any expertise in the field.
For things like sickness, old age and death, generally the lacks are not knowledge but the consistent will of the group. Like global warming, there is no real profound knowledge gap except how to get everyone to agree to mitigating the carbon burn. Perhaps it is likely LLMs would be able to persuade the group to stop burning the carbon before it’s [more] uncomfortable, but the small group of people committed to accelerating the carbon burn can also pay for LLM persuasiveness.
LLMs giving you something novel would be like if you let a model play chess against itself and become the best player in the world this way. Totally impossible.
/sacrasm