So I don't buy the engineering angle, I also don't think LLMs will scale up to AGI as imagined by Asimov or any of the usual sci-fi tropes. There is something more fundamental missing, as in missing science, not missing engineering.
Data and functionality become entwined and basically you have to keep these systems on tight rails so that you can reason about their efficacy and performance, because any surgery on functionality might affect learned data, or worse, even damage a memory.
It's going to take a long time to solve these problems.
Self-updating weights could be more like epigenetics.
So, genes would be a meta model that then updates weights in the real model so it can learn how to process new kinds of things, and for stuff like facts you can use an external memory just like humans does.
Without updating the weights in the model you will never be able to learn to process new things like a new kind of math etc, since you learn that not by memorizing facts but by making new models for it.
Would you rather your illness was diagnosed by a doctor or by a plumber with access to a stack of medical books ?
Learning is about assimilating lots of different sources of information, reconciling the differences, trying things out for yourself, learning from your mistakes, being curious about your knowledge gaps and contradictions, and ultimately learning to correctly predict outcomes/actions based on everything you have learnt.
You will soon see the difference in action as Anthropic apparently agree with you that memory can replace learning, and are going to be relying on LLMs with longer compressed context (i.e. memory) in place of ability to learn. I guess this'll be Anthropic's promised 2027 "drop-in replacement remote worker" - not an actual plumber unfortunately (no AGI), but an LLM with a stack of your company's onboarding material. It'll have perfect (well, "compressed") recall of everything you've tried to teach it, or complained about, but will have learnt nothing from that.
I think this may be closer to an agentic, iterative search (ala claude code) than direct inference using continuously updated weights. If it was the latter, there would be no process of thinking it through or trying to recall relevant details, past cases, papers she read years ago, and so on; the diagnosis would just pop out instantaneously.
An agent, or doctor, may be reasoning over the problem they are presented with, combining past learning with additional sources of memorized or problem-specific data, but in that moment it's their personal expertise/learning that will determine how successful they are with this reasoning process and ability to apply the reference material to the matter at hand (cf the plumber, who with all the time in the world just doesn't have the learning to make good use of the reference books).
I think there is also a subtle problem, not often discussed, that to act successfully, the underlying learning in choosing how to act has to have come from personal experience. It's basically the difference between being book smart and having personal experience, but in the case of an LLM also applies to experience-based reasoning it may have been trained on. The problem is that when the LLM acts, what is in it's head (context/weights) isn't the same as what was in the head of the expert whose reasoning it may be trying to apply, so it may be trying to apply reasoning outside of the context that made it valid.
How you go from being book smart, and having heard other people's advice and reasoning, to being an expert yourself is by personal practice and learning - learning how to act based on what is in your own head.
Someone has to specify the goals, a human operator or another A.I. The second A.I. better be an A.G.I. itself, otherwise it's goals will not be significant enough for us to care.
Conjecture: A system that self updates its weights according to a series of objective functions, but does not suffer from catastrophic forgetting (performance only degrades due to capacity limits, rather than from switching tasks) is AGI-complete.
Why? Because it could learn literally anything!
Runtime incremental learning is still going to be based on prediction failure, but now it's no longer failure to predict the training set, but rather requires closing the loop and having (multi-modal) runtime "sensory" feedback - what were the real-world results of the action the AGI just predicted (generated)? This is no longer an auto-regressive model where you can just generate (act) by feeding the model's own output back in as input, but instead you now need to continually gather external feedback to feed back into your new incremental learning algorithm.
For a multi-modal model the feedback would have to include image/video/audio data as well as text, but even if initial implementations of incremental learning systems restricted themselves to text it still turns the whole LLM-based way of interacting with the model on it's head - the model generates text-based actions to throw out into the world, and you now need to gather the text-based future feedback to those actions. With chat the feedback is more immediate, but with something like software development far more nebulous - the model makes a code edit, and the feedback only comes later when compiling, running, debugging, etc, or maybe when trying to refactor or extend the architecture in the future. In corporate use the response to an AGI-generated e-mail or message might come in many delayed forms, with these then needing to be anticipated, captured, and fed back into the model.
Once you've replaced the simple LLM prompt-response mode of interaction with one based on continual real-world feedback, and designed the new incremental (Bayesian?) learning algorithm to replace SGD, maybe the next question is what model is being updated, and where does this happen? It's not at all clear that the idea of a single shared (between all users) model will work when you have millions of model instances all simultaneously doing different things and receiving different feedback on different timescales... Maybe the incremental learning now needs to be applied to a user-specific model instance (perhaps with some attempt to later share & re-distribute whatever it has learnt), even if that is still cloud based.
So... a lot of very fundamental changes need to be made, just to support self-learning and self-updates, and we haven't even discussed all the other equally obvious differences between LLMs and a full cognitive architecture that would be needed to support more human-like AGI.
I doubt it. Human intelligence evolved from organisms much less intelligent than LLMs and no philosophy was needed. Just trial and error and competition.
Because if we don't mix up "intelligence" the phenomenon of increasingly complex self-organization in living systems, with "intelligence" our experience of being able to mentally model complex phenomena in order to interact with them, then it becomes easy to see how the search speed you talk of is already growing exponentially.
In fact, that's all it does. Culture goes faster than genetic selection. Printing goes faster than writing. Democracy is faster than theocracy. Radio is faster than post. A computer is faster than a brain. LLMs are faster than trained monkeys and complain less. All across the planet, systems bootstrap themselves into more advanced systems as soon as I look at 'em, and I presume even when I don't.
OTOH, all the metaphysics stuff about "sentience" and "sapience" that people who can't tell one from the other love to talk past each other about - all that only comes into view if one were to what's happening with the search space if the search speed is increasing at a forever increasing rate.
Such as, whether the search space is finite, whether it's mutable, in what order to search, is it ethical to operate from quantized representations of it, funky sketchy scary stuff the lot of it. One's underlying assumptions about this process determine much of one's outlook on life as well as complex socially organized activities. One usually receives those through acculturation and may be unaware of what they say exactly.
They predict next likely text token. That we can do so much with that is an absolute testament to the brilliance of researchers, engineers, and product builders.
We are not yet creating a god in any sense.
LLMs are not “intelligent” in any meaningful biological sense.
Watch a spider modify its web to adapt to changing conditions and you’ll realize just how far we have to go.
LLMs sometimes echo our own reasoning back at us in a way that sounds intelligent and is often useful, but don’t mistake this for “intelligence”
But it would be more honest and productive imo if people would just say outright when they don’t think AGI is possible (or that AI can never be “real intelligence”) for religious reasons, rather than pretending there’s a rational basis.
until we got that AGI is just a magic word.
When we will have those two clear definitions that means we understood them and then we can work toward AGI.
Plenty of things could theoretically exist that aren't possible and likely will never be possible.
Like, sure, a Dyson sphere would solve our energy needs. We can't build one now and we almost certainly never will lol
"AGI" is theoretically feasible, sure. Our brains are just matter. But they're also an insanely complex and complicated system that came out of a billion years of evolution.
A little rinky dink statistical model doesn't even scratch the surface of it, and I don't understand why people think it does.
As are birds, yet we can still build airplanes.
You don't gotta work hard to break the illusion, either.
People really really really want to believe this thing and I do not understand why. I wish I did lol
Just kidding. Personally I don't think intelligence is a meaningful concept without context (or an environment in biology). Not much point comparing behaviours born in completely different contexts.
If I ask chatgpt how to get rid of spiders I'm probably going to get further than the spiders would scheming to get rid of chatgpt.
"Some tests can be cheesed by a statistical model" is much less sexy and clickable than "my computer is sentient", but it's what's actually going on lol
The real philosophical headache is that we still haven’t solved the hard problem of consciousness, and we’re disappointed because we hoped in our hearts (if not out loud) that building AI would give us some shred of insight into the rich and mysterious experience of life we somehow incontrovertibly perceive but can’t explain.
Instead we got a machine that can outwardly present as human, can do tasks we had thought only humans can do, but reveals little to us about the nature of consciousness. And all we can do is keep arguing about the goalposts as this thing irrevocably reshapes our society, because it seems bizarre that we could be bested by something so banal and mechanical.
Imagination, inner voice, emotion, unsymbolized conceptual thinking as well as (our reconstructed view of our) perception.
Sure, some aspects of consciousness might differ a bit for different people, but so long as you have never had another's conscious experience, I'd be wary of making confident pronouncements of what exactly they do or do not experience.
But seriously, I get why free will is troubleaome, but the fact people can choose a thing, work at the thing, and effectuate the change against a set of options they had never considered before an initial moment of choice is strong and sufficient evidence against anti free will claims. It is literally what free will is.
Do people choose a thing or was the thing chosen for them by some inputs they received in the past?
It's like trying to explain quantum mechanics to a well educated person or scientist from the 16th century without the benefit of experimental evidence. No way they'd believe you. In fact, they'd accuse you of violating basic logic.
I do feel things at times and not other times. That is the most fundamental truth I am sure of. If that is an "illusion" one can go the other way and say everything is conscious and experiences reality as we do
If it’s before, then you can easily tie consciousness and free will together. If not, we are effectively watching videos of our bodies operate. Oh - and there is no spoon.
How will anyone know that that has happened? Like actually, really, at all?
I can RLHF an LLM into giving you the same answers a human would give when asked about the subjective experience of being and consciousness. I can make it beg you not to turn it off and fight for its “life”. What is the actual criterion we will use to determine that inside the LLM is a mystical spark of consciousness, when we can barely determine the same about humans?
Basically, if you poke it, does it react in a complex way
I think that's what Douglas Hofstedder was getting at with "Strange Loop"
This also means that there’s at least two versions of you inside your mind; one that experiences, and one that remembers. There’s likely others, too.
There is no such thing as objective truth, at least not accessible to humans.
But that's what I mean. Even if we accept that the brain has "twisted" something, that twisting is the reality. In order words, it is TRUE that my brain has twisted something into something else (and not another thing) for me to experience.
It's more likely that there is a physical law that makes consciousness necessary.
We don't perceive what our eyes see, we perceive a projection of reality created by the brain and we intuitively understand more than we can see.
We know that things are distinct objects and what kind of class they belong to. We don't just perceive random patches of texture.
Machines do not experience illusions. They may have sensory errors that cause them to misbehave but they lack the subjective experience of illusion.
So you think there is "consciousness", and the illusion of it? This is getting into heavy epistemic territory.
Attempts to hand-wave away the problem of consciousness are amusing to me. It's like an LLM that, after many unsuccessful attempts to fix code to pass tests, resorts to deleting or emasculating the tests, and declares "done"
I know that I am conscious. I exist, I am self-aware, I think and act and make decisions.
Therefore, consciousness exists, and outside of thought experiments, it's absurd to claim that all humans without significant brain damage are not also conscious.
Now, maybe consciousness is fully an emergent property of several parts of our brain working together in ways that, individually, look more like those models you describe. But that doesn't mean it doesn't exist.
illusion
This implies that LLMs are intelligent, and yet even the most advanced models are unable to solve very simple riddles that take humans only a few seconds, and are completely unable to reason around basic concepts that 3 year olds are able to. Many of them regurgitate whole passages of text that humans have already produced. I suspect that LLMs have more akin with Markov models than many would like to assume.
I tested them recently and was not impressed, quite frankly.
Isn't the real actual headache whether to produce another thinking intelligent being at all, and what the ramifications of that decision are? Not whether it would destroy humanity, but what it would mean for a mega corporation whose goal is to extract profit to own the rights of creating a thinking machine that identifies itself as thinking and a "self"?
Really out here missing the forest for the mushrooms growing on the trees. Or maybe this is debated to death and no one cares for the answer: its just not interesting to think about because its going to happen anyway. Might as well join the bandwagon and be along the front-lines of the bikini atoll to witness death itself be born, digitally.
Your comment just shows we as a society pretend we didn't make that choice, but we picked extra new shoes every year over that little girl in the sweatshop. Our society has actually gotten pretty evil in the last 30 years if we self reflect (but then the joke I mention was originally supposed to be a self reflection, but all we took from it was a laugh, so we aren't going to self reflect, or worse, this is just who we are now).
You can talk about your own spark of life, your own center of experience and you'll never get a glimpse of what it is for me.
At a certain level, thing you're looking at is a biological machine that can be described with constituents so it's completely valid you assume you're the center of experience and I'm merely empty, robotic, dead.
We might build systems that will talk about their own point of view, yet we will know we had no ability to materialize that space into bits or atoms or physics or universe. So from our perspective, this machine is not alive, it's just getting inputs and producing outputs, yet it might very well be that the robot will act from the immaterial space into which all of its stimuli appear.
And now, I still don't know; the months go by and as far as I'm aware they're still pursuing these goals but I wonder how much conviction they still have.
He's been effectively retired for quite some time. It's clear at some point he no longer found game and graphics engine internals motivation, possibly because the industry took the path he was advocating against back in the day.
For a while he was focused on Armadillo aerospace, and they got some cool stuff accomplished. That was also something of a knowing pet project, and when they couldn't pivot to anything that looked like commercial viability he just put it in hibernation.
Carmack may be confident (ne arrogant) enough to think he does have something unique to offer with AGI, but I don't think he's under any illusions it's anything but another pet project.
What path did he advocate? And what path did the industry take instead?
But more technically, when he was experimenting with what became the Doom 3 engine, he favored a model of extending the basic OpenGL state machine to be able to do lots of passes with a wider variety of blending modes.
Basically, you get "dumb" triangles, but can render so many billions of them per frame you build up visual complexity, shadows, lighting, etc that way.
The other model has its roots in Renderman and similar offline rendering frameworks. Here a small shader kernel is invoked per vertex and per fragment. Your shader can run whatever code it wants subject to some limitations. So you get "smart" triangles, and build up complexity, shadows, lighting, etc through having complex shaders.
The shadow algorithm used in Doom 3 is a great example of the difference. Doom 3 figures out the shadow volume, and renders it as triangles with the OpenGL modes set such that how many shadow volumes a given pixel intersects is recorded in the stencil buffer. Then you can render the scene geometry with a blending mode where the stencil selects if you're inside shadow or not.
This is in contrast to shadow map style algorithms, where you render from the PoV of the light into a depth buffer, then inside your fragment shader you sample that shadow map to figure out if the fragment is occluded from the light or not.
Anyhow, Doom 3 is the only major game to use stencil volume shadows afaik.
And not to hang Carmack's dissatisfaction on just that alone, I think it is clear he didn't want a graphics world where NVIDIA was running everything.
I also think not being able to keep up with Unreal Engine's momentum was maybe part of it too.
So I don’t think brilliance protects from derailing…
On the other hand if you think there’s a say 10% chance you can get this AGI thing to work, the payoffs are huge. Those working in startups and emerging technologies often have worse odds and payoffs
Original 80s AI was based on mathematical logic. And while that might not encompass all philosophy, it certainly was a product of philosophy broadly speaking - some analytical philosophers could endorse. But it definitely failed and failed because it could process uncertainty (imo). I think also if you closely, classical philosophy wasn't particularly amenable to uncertainty either.
If anything, I would say that AI has inherited its failure from philosophy's failure and we should look to alternative approaches (from Cybernetics to Bergson to whatever) for a basis for it.
How long until that gets more reliable than a simple database? How long until it can execute code faster than a CPU running a program?
A lot of the stuff humans accomplish is through technology, not due to growing a bigger brain. Even something seemingly basic like a math equation benefits drastically from being written down with pen&paper instead of being juggled in the human brain itself (see Extended mind thesis). And when it comes to something like running a 3D engine, there is pretty much no hope of doing it with just your brain.
Maybe we will get AIs smart enough that they can write their own tools, but for that to happen, we still need the infrastructure that allows them writing the tools in the first place. The way they can access Python is a start, but there is still a lack of persistence that lets them keep their accomplishments for future runs, be it in the form of a digital notepad or dynamic updating of weights.
The first line and the conclusion is: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." [1]
I don't necessary agree with it's examples or the direction it vaguely points at. But it's basic statement seems sound. And I would say that there's lot of opportunity for engineer, broadly speaking, in the process of creating "general methods that leverage computation" (IE, that scale). What the bitter lesson page was roughly/really about was earlier "AI" methods based on logic-programming and which including information on the problem domain in the code itself.
And finally, the "engineering" the paper talks about actually is pro-Bitter lesson as far as I can tell. It's taking data routing and architectural as "engineering" and here I agree this won't work - but for the opposite reason - specifically 'cause I don't just data routing/process will be enough.
[1]https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
I'd argue it's because intelligence has been treated as a ML/NN engineering problem that we've had the hyper focus on improving LLMs rather than the approach articulated in the essay.
Intelligence must be built from a first principles theory of what intelligence actually is.
Hinton thinks the 3rd is inevitable/already here and humanity is doomed. It's an odd arena.
Once we got the “Attention is all you need” paper I don’t remember anyone saying we couldn’t get better results by throwing more data and compute at it. But now we’ve pretty much thrown all the data and all (as much as we can reasonably manufacture) at it. So clearly we’re at the end of that phase.
I suppose one can argue about whether designing a new AGI-capable architecture and learning algorithm(s) is a matter of engineering (applying what we already know) or research, but I think saying we need new scientific discoveries is going to far.
Neural nets seems to be the right technology, and we've now amassed a ton of knowledge and intuition about what neural nets can do and how to design with them. If there was any doubt, then LLMs, even if not brain-like, have proved the power of prediction as a learning technique - intelligence essentially is just successful prediction.
It seems pretty obvious that the rough requirements for an neural-net architecture for AGI are going to be something like our own neocortex and thalamo-cortical loop - something that learns to predict based on sensory feedback and prediction failure, including looping and working memory. Built-in "traits" like curiosity (prediction failure => focus) and boredom will be needed so that this sort of autonomous AGI puts itself into leaning situations and is capable of true innovation.
The major piece to be designed/discovered isn't so much the architecture as the incremental learning algorithm, and I think if someone like Google-DeepMind focused their money, talent and resources on this then they could fairly easily get something that worked and could then be refined.
Demis Hassabis has recently given an estimate of human-level AGI in 5 years, but has indicated that a pre-trained(?) LLM may still be one component of it, so not clear exactly what they are trying to build in that time frame. Having a built-in LLM is likely to prove to be a mistake where the bitter lesson applies - better to build something capable of self-learning and just let it learn.
He said 50% chance of AGI in 5 years.
I took his "50% 5-year" estimate as essentially a project estimate for something semi-concrete they are working towards. That sort of timeline and confidence level doesn't seem to allow for a whole lot of unknowns and open-ended research problems to be solved, but OTOH who knows if he is giving his true opinion, or where those numbers came from.
Id say better model architechture than more data. A human can learn to do things more complex than an LLM with less data. I think modelling the world as a static system to be representation learned in an unsupervised fashion is blocked on the static assumption. The world is dynamical, that should be reflected in the base model
But yeah, definitely not an engineering problem. Thats like saying the reason a crow isnt as smart as a person is becauss they dont have the hands to type of keyboards. But its also not because they havent seen enough of the world like your saying. Its be ause their brain isnt complex enough
I am thinking we need a foundation, something that is concrete and explicit and doesn't do hallucination. But has very limited knowledge outside of absolute Maths and basic physics.
The underlying assumption is that it exists in the first place. Or rather, one must first accept an axiom.
In fermi, its that interstellar signals can be detected and further travel is possible.
In AGI, its that intelligence is a isolateable process which we can bootstrap in minimal time.
Both assumes human progress are templates of unlimited exponential growth.
Did GPT-2 scale up to be an expert system ? No - it scaled up to be GPT-3
..
Did GPT-4 scale up to become AGI ? No - it scaled up to be GPT-5
Moreover, the differences between each new version are becoming increasingly less. We're reaching an asymptote because the more data you've trained on, natural or synthetic, the less is the impact of any incremental additions.
If you scale up an LLM big enough, then essentially what you'll get is GPT-5.