>>We just don't know what the maximum capability of AI is

For all theory purposes there is no limit. Thats what the latest loop engineering trend is about, you are asking AI to find solutions to a problem going by listing steps, and if solution not found in those steps, to treat each step as a separate problem and repeat the process until the master solution to the master problem is found.

Once a solution is found, or new data/insights are generated through this process, the LLM can be trained on this. So in theory you can just keep going like this forever.

Secondly. This is as close to agency you can build inside a machine.

Practically speaking, hardware is a limit. But that can scale up with time.

So we are already looking at some kind of runaway intelligence even if not sentient.