How does the model know when a human has to take over?
I think most extrapolations of current "AI" capabilities into future capabilities are fun and useful in some ways, but also doomed to fail. It's very easy to miss a tiny detail which may in practice be a fundamental problem.
> Use the actual usage data as training input.
Given that those bigger state-of-the-art models train on terabytes of data, how would you know how much training data to generate to sufficiently change the output?
My understanding of "AI" is that it's mostly about some very complex models which are capable of solving previously unsolvable problems. However, those problems are always extremely specific. Going the other way of thinking of problems or future possibilities first and then applying "AI" to it is likely to fail.