We aren't close to creating a rapidly self-improving AI

The only reason we have things like ChatGPT today is because of the surprise development of the transformer. Nobody understands how intelligence fundamentally works, so we literally have no ability to predict when the next groundbreaking architectural development will occur.

And self-improvement concerns are mostly orthogonal to the main risk of AI: the development of AGI. Even a mediocre AGI that is only "decent" at tasks, can be weaponized as a collective superintelligence.

But when people say we're close, I think they mean 1 accidental breakthrough away.

Like the "just add more layers" meme, the solution to the own-dataset creation problem could be stupidly simple, just waiting for some bored student with access to a decent machine to accidentally find it. For example, maybe just giving the AI the right novelty seeking behaviors (like children have) will be enough to get the process kicked off.

My objection to the singulatarian concept of a self improving AI is simply that I haven't really heard a reason to think that there exist intelligence algorithms that are dramatically superior to ours. It is entirely possible that our thinking algorithms are optimal-enough that rapid self improvement leads straight to something not much different from human intelligence, just faster, and in silicon.

The AI system would be bottlenecked by data in the sense that it will have to run experiments, but it's not clear that it needs a new paradigm to resolve this bottleneck. It just has to propose an experiment and interpret its results. So it writes code, and the experimental outcome gets fed back into the model as any other normal input. As an AI researcher, I'd like to believe that this is not going to happen anytime soon, but I'm not sure we're far.

The truth is we don’t fully understand what it means to be intelligent, and how intelligence scales. All indications are that intelligence comes with diminishing returns, in that to achieve small increases in intelligence requires large increases in connectivity. Even if we can design an intelligence that is generally smarter than us, and capable of building even smarter ai on its own, it’s intelligence might increase exponentially with each step, but each step could easily take increasingly longer amounts of time to achieve, leading to linear or even sub linear increases in capabilities over time. Resources are limited, hardware is limited, energy is limited. There are all sorts of constraints that are likely to play a limiting factor in runaway intelligence.

This doesn’t mean that these intelligences won’t be dangerous. Even an average human can create a lot of trouble in this world. It does however mean that a singular AI is unlikely to just run away with things.

Active learning would be useful at creating a superhuman AGI, but I don't see it as a requirement. As soon as AIs can iterate on future models and produce increasingly useful training data for future models, then models would likely continue to grow more capable at an exponential rate.

But I'm not even convinced active learning is that difficult once you have fairly capable AI programmers...

My AI professor once said, "all problems are search problems". And this suck with me because the world looks different when you start seeing all problems as just iterations of different optimisation problems. So long as you have a good heuristic and enough compute, eventually a suitable solution for any problem will be found. This is what makes generalised learning algorithms so powerful.

The author states that, "active learning with neural networks are uniformly terrible", but this just reeks to me of a search problem in need of optimisation. So as long as we can tell GPT5, "I want you to iteratively write code which improve on existing state-of-the-art active learning algorithms", then eventually you'll get there... GPT5 like humans might not get very far in the beginning, but assuming it can iterate faster than humans and future iterations faster still, then breakthroughs will eventually be made and progress should accelerate exponentially.

Of course, this also assumes that humans don't get there first. And honestly given the amount of investment in AI and rate of progress in AI right now I just wouldn't bet against a significant advancement in active learning coming soon.

The whole thing doesn't quite work together too well. But it's more of an integration problem now. This is still in its early phases (only a few months, or less), so I wouldn't say it can't happen.

AGI is a flying car mythology.

Narrow AI is improving and the rate is accelerating. The current speed (1st derivative) is unimportant, but there is a differential equation-like feedback function to it and it will accelerate (2nd derivative): speed proportional to the amount present.

People and technology form a coupled system that is somewhat a super-organism. That cycle is amplifying in a way that is not necessarily obvious (it doesn't happen in public) or continuous (or discrete for that matter). For the foreseeable future, technology does not yet constitute a self-assembling or self-modifying closed system. It is still entirely dependent on human effort. What is happening is that people are relying on and exploiting the efforts of AI that had not happened previously. This is one change, more visible than most, that is leading to more efficiencies, more changes, and more technological advancement. Fully automated factories, software development, systems engineering, and chip design may not happen anytime soon but they are both possible and probable due to the forcing function and Tragedy of the Commons of human greed.

Nitpicking but an intelligence explosion due to an AI able to enhance itself is called the "technological singularity":

https://en.wikipedia.org/wiki/Technological_singularity

The reason these don’t work now is that the gradients are way too sparse to learn from. But at a certain level of capability this will stop being true. The big Q is whether the existing gradient of “all written content” gets the loss low enough for another information source to smoothly take over.

More pithily, there are plenty of high-quality datasets if you are smart enough to generate OK predictions. The smarter you get the more you can learn from.

There are plenty of obvious synthetic datasets that GPT-5 could be trained on, eg generated problems for all of the physics/maths it currently gets wrong. (I’d be surprised if this isn’t an angle they are working on.)

From an external perspective it's as if the AI is self-improving by leveraging human agents to assist the process. Not to say there is any 'intent' behind it, but rather to view it as an emergent phenomenon.

wouldn't fine tuning an existing LLM model like GPT4 could be enough to filter our automatic dataset from "garbage"?

if so we could basically do the same iterative process, but on different level use ai to improve dataset, retrain ai, use new ai to even better improve dataset

Recursive self-improvement is not a necessary condition for a fast takeoff. A strongly superhuman AI could emerge at training time. AlphaGo Zero went from subhuman to superhuman performance within a few hours. The same could happen with a general AI. A training run of a few weeks or even months would still be considered fast.

the hypothesis is a pattern recognition exercise in itself (i.e., looking at a number of documented occurences around algorithmic development and extrapolating), which I think neatly highlights the fundamental difference between human pattern recognition and machine pattern recognition

3.5: As an AI language model, I cannot guess, but I can provide some general guidelines based on current research and best practices.

If we want to improve the results of Large Language Models (LLMs), one aspect of the architecture that we could focus on is increasing the model's capacity to learn and retain more information. This could be achieved by increasing the number of parameters in the model or using more sophisticated architectures such as transformer-based models that use self-attention mechanisms to capture long-range dependencies in the input sequence.

Another important aspect to focus on is improving the model's ability to handle rare and out-of-vocabulary (OOV) words. This can be achieved by using subword-level tokenization, which breaks down words into smaller units and enables the model to generalize better to new or unseen words.

We could also focus on improving the training process by using larger and more diverse training datasets, regularization techniques to prevent overfitting, and optimizing hyperparameters such as learning rate, batch size, and number of training epochs.

Finally, we could also focus on incorporating external knowledge sources such as structured data, knowledge graphs, or ontologies into the model architecture to enhance its ability to reason and make more accurate predictions.

Overall, there are many aspects of the LLM architecture that can be improved to enhance its performance, and the choice of which to focus on will depend on the specific task and the available resources.

Wrong. Full self driving is still not here despite access to a huge amount of high quality data.

2023: The best AIs score in the top 10% of law, medical, engineering exams, and outperform licensed professionals on real-world tasks.

Please stop speculating. You don't know what we are (or aren't) "close" to.