That sounds entirely possible then. I'd wager the limits we'd hit would be memories - it's not known if our brains, in their current form, would be able to cope with hundreds of years of memories.
>> self-replicating robots are quite literally what all of biology is
> By the same argument AGI is just same as human brain.
Yes, but the main difference is that biology is merely too big for us to keep track of - it just has too many moving pieces that have been, from our perspective, heavily overfitted. Still, we've reached the level where we can reprogram some of the nanobots. In contrast, the topic of intelligence still has a lot of big mysteries.
Another way of looking at it is, we already have a well-mapped way of building nanotechnology: reprogramming the one that is around us, and is us. Plenty of little self-replicating programmable bots to pick from. But we're not at similar stage with poking in brains just yet.
> Many people think ChatGPT is so ompressive, that we are on the cust of AGI. I am convinced otherwise - chat gpt makes limitations of current AI approached very clear.
I agree. Though to me, it also revealed limitations of human cognition (or rather, it was already quite apparent at GPT-2 level; ChatGPT is only rubbing it in everyones' faces).
Have you ever felt that your own thinking, in many situations, is mostly cache lookups? That your speech and inner monologue (if you have one) both resemble a Markov chain, and your "self" mostly just observes and censors the output? I certainly did, quite a lot over the decades. I know others did, many obviously rejecting it as silly association. But what the recent LLMs show us is that maybe, just maybe, it's not silly at all. Which invites the question, if a lot of our cognition works this way, then just how much more complicated are the bits that don't?
(Another thing ChatGPT and the like are making apparent, is that AI risk isn't tied to reaching AGI. I think people kind of assumed it would take AGI taking off to end human civilization, but at this point I believe it's pretty clear that LLMs as a class of models are up to the task already - all they need is to have more memory, access to APIs that let them affect the real world and observe the results, and being run continuously; Reddit corpus already supplies the token association patterns that would let them end the world if given the means...)