Andrej Karpathy – It will take a decade to work through the issues with agents
dwarkesh.com
dwarkesh.com
I think this is an important way of understanding AI progress. Capability improvements often look exponential on a particular fixed benchmark, but the difficulty of the next step up is also often exponential, and so you get net linear improvement with a wider perspective.
The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.
A marathon consists of two halves: the first 20 miles, and then the last 10k (6.2mi) when you're more sore and tired than you've ever been in your life.
It's the major point of contention between him and the host (who thinks growth rate will increase).
But this method of AI is still pretty new, and we don't know it's upper limits. It may be that there are no more 9s to add, or that any more 9s cost prohibitively more. We might be effectively stuck at 91.25626726...% forever.
Not to be a doomer, but I DO think that anyone who is significantly invested in AI really has to have a plan in case that ends up being true. We can't just keep on saying "they'll get there some day" and acting as if it's true. (I mean you can, just not without consequences.)
20% of your effort gets you 80% of the way. But most of your time is spent getting that last 20%. People often don't realize that this is fractal like in nature, as it draws from the power distribution. So of that 20% you still have left, the same holds true. 20% of your time (20% * 80% = 16% -> 36%) to get 80% (80% * 20% => 96%) again and again. The 80/20 numbers aren't actually realistic (or constant) but it's a decent guide.
It's also something tech has been struggling with lately. Move fast and break things is a great way to get most of the way there. But you also left a wake of destruction and tabled a million little things along the way. Someone needs to go back and clean things up. Someone needs to revisit those tabled things. While each thing might be little, we solve big problems by breaking them down into little ones. So each big problem is a sum of many little ones, meaning they shouldn't be quickly dismissed. And like the 9's analogy, 99.9% of the time is still 9hrs of downtime a year. It is still 1e6 cases out of 1e9. A million cases is not a small problem. Scale is great and has made our field amazing, but it is a double edged sword.
I think it's also something people struggle with. It's very easy to become above average, or even well above average at something. Just trying will often get you above average. It can make you feel like you know way more but the trap is that while in some domains above average is not far from mastery in other domains above average is closer to no skill than it is to mastery. Like how having $100m puts your wealth closer to a homeless person than a billionaire. At $100m you feel way closer to the billionaire because you're much further up than the person with nothing but the curve is exponential.
He was also pointing out that the same high cost of failure consideration applies to many software systems (depending on what they are doing/controlling). We may already be at the level where AI coding agents are adequate for some less critical applications, but yet far away from them being a general developer replacement. I see software development as something that uses closer to 100% of your brain than 10% - we may well not see AI coding agents approach human reliability levels until we have human level AGI.
The AI snake oil salesmen/CEOs like to throw out competitive coding or math olympiad benchmarks as if they are somehow indicative of the readiness of AI for other tasks, but reliability matters. Nobody dies or loses millions of dollars if you get a math problem wrong.
I can imagine other task on a human/rules-based "frontier" would have a similar quality. But I think there are others that are going to be inaccessible entirely "until AGI" (or something). Humanoid robots moving freely in human society would an example I think.
Academia has rediscovered itself
Signal attenuation, a byproduct of entropy, due to generational churn means there's little guarantee.
Occam's Razor; Karpathy knows the future or he is self selecting biology trying to avoid manual labor?
His statements have more in common with Nostradamus. It's the toxic positivity form of "the end is nigh". It's "Heaven exists you just have to do this work to get there."
Physics always wins and statistics is not physics. Gamblers fallacy; improvement of statistical odds does not improve probability. Probability remains the same this is all promises of some people who have no idea or interest in doing anything else with their lives; so stay the course.
First time being hearing it be called "march of nines", did Tesla make the term, I thought it was an Amazon thing
No magical thinking here. No empty blather about how AI is going to make us obsolete with the details all handwaved away. Karpathy sees that, for now, better humans are the only way forward.
Also, speculation as to why AI coders are "mortally terrified of exceptions": it's the same thing OpenAI recently wrote about, trying to get an answer at all costs to boost some accuracy metric. An exception is a signal of uncertainty indicating that you need to learn more about your problem. But that doesn't get you points. Only a "correct answer" gets you points.
Frontier AI research seems to have yet to operationalize a concept of progress without a final correct answer or victory condition. That's why AI is still so bad at Pokemon. To complete open-ended long-running tasks like Pokemon, you need to be motivated to get interesting things to happen, have some minimal sense of what kind of thing is interesting, and have the ability to adjust your sense of what is interesting as you learn more.
Right now the median actor in the space loudly proclaims AGI is right around the corner, while rolling out pornbots/ads/in-chat-shopping, which generally seems at odds with a real belief that AGI is close (TAM of AGI must be exponentially larger than the former).
No, that would be a warning. Ab exceprion is a signal something failed and it was impossible to continue
Now I see why Karpathy was talking of RL up-weights as if they were a destructive straw-drawn line of a drug for an LLM's training.
When you’re talking about an agent, or what the labs have in mind and maybe what I have in mind as well, you should think of it almost like an employee or an intern that you would hire to work with you. For example, you work with some employees here. When would you prefer to have an agent like Claude or Codex do that work?
Currently, of course they can’t. What would it take for them to be able to do that? Why don’t you do it today? The reason you don’t do it today is because they just don’t work. They don’t have enough intelligence, they’re not multimodal enough, they can’t do computer use and all this stuff.
They don’t do a lot of the things you’ve alluded to earlier. They don’t have continual learning. You can’t just tell them something and they’ll remember it. They’re cognitively lacking and it’s just not working. It will take about a decade to work through all of those issues.
>Overall, the models are not there. I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it’s not. It’s slop. They’re not coming to terms with it, and maybe they’re trying to fundraise or something like that. I’m not sure what’s going on, but we’re at this intermediate stage. The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot. But sometimes, for some types of code, I will go to an LLM agent.
>They kept trying to mess up the style. They’re way too over-defensive. They make all these try-catch statements. They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it’s okay. I don’t need all this extra stuff in there. So I feel like they’re bloating the code base, bloating the complexity, they keep misunderstanding, they’re using deprecated APIs a bunch of times. It’s a total mess. It’s just not net useful. I can go in, I can clean it up, but it’s not net useful.
He's a smart man with well-reasoned arguments, but I think he's also a bit poisoned by working at such a huge org, with all the constraints that comes with. Like, this:
You can’t just tell them something and they’ll remember it.
LLMs are a necessary part of AGI(/"agents") due to their ability to avoid the Frame Problem[1], but they're far from the only needed thing. We're pretty dang good at "remembering things" with computers already, and connecting that with LLM ensembles isn't going to take anywhere close to 10 years. Arguably, we're already doing it pretty darn well in unified systems[2]...
If anyone's unfamiliar and finds my comment interesting, I highly recommend Minsky's work on the Society of Mind, which handled this topic definitively over 20 years ago. Namely;
A short summary of "Connectionism and Society of Mind" for laypeople at DARPA: https://apps.dtic.mil/sti/tr/pdf/ADA200313.pdf
A description of the book itself, available via Amazon in 48h or via PDF: https://en.wikipedia.org/wiki/Society_of_Mind
By far my favorite paper on the topic of connectionist+symbolist syncreticism, though a tad long: https://www.mit.edu/~dxh/marvin/web.media.mit.edu/~minsky/pa...
[1] https://plato.stanford.edu/entries/frame-problem/
[2] https://github.com/modelcontextprotocol/servers/tree/main/sr...
He has the ability to explain concepts and thoughts with analogies and generalizations and interesting sayings that allow you to keep interest in what he is talking about for literally hours - in a subject that I don't know that much about. Clearly he is very smart, as is the interviewer, but he is also a fantastic communicator and does not come across as arrogant or pretentious, but really just helpful and friendly. Its quite a remarkable and amazing skillset. I'm in awe.
With all these issues already being hard to manage, I just don't believe businesses are going to delegate processes to autonomous agents in a widespread manner. Literally anything that matters is going to get implemented in a crontrolled workflow that strips out all the autonomy with human checkpoint at every step. They may call them agents just to sound cool but it will be completely controlled.
Software people are all fooled by what is really a special case around software development : outcomes are highly verifiable and mistakes (in development) are almost free. This is just not the case out there in the real world.
Yea, there aren't a ton of problems (that I can see) in my current domain that could be solved by having unattended agents generating something.
I work in healthcare and there are a billion use cases right now, but none that don't require strict supervision. For instance, having an LLM processing history and physicals from potential referrals looking for patient problems/extracting historical information is cool, but it's nowhere near reliable enough to do anything but present that info back to the clinician to have them verify it.
Karpathy’s definition of “agent” here is really AGI (probably somewhere between expert and virtuoso AGI https://arxiv.org/html/2311.02462v2). In my taxonomy you can have non-AGI short-task-timeframe agents. Eg in the METR evals, I think it’s meaningful to talk about agent tasks if you set the thing loose for 4-8h human-time tasks.
If anyone can suggest a more accurate and representative title, we can change it again.
Edit: I thought of using "For now, autocomplete is my sweet spot", which has the advantage of being an exact quote; but it's probably not clear enough.
Edit 2: I changed it to "It will take a decade to work through the issues with agents" because that's closer to the transcript.
Anybody have a better idea? Help the cause of accuracy out here!
>They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff. They don't do a lot of the things you've alluded to earlier. They don't have continual learning. You can't just tell them something and they'll remember it. They're cognitively lacking and it's just not working.
>It will take about a decade to work through all of those issues. (2:20)
Also they discuss the nanochat repo in the interview, which has become more famous for his tweet about him NOT vibe-coding it: https://www.dwarkesh.com/i/176425744/llm-cognitive-deficits
Things are more nuanced than what people have assumed, which seems to be "LLMs cannot handle novel code". The best I can summarize it as is that he was doing rather non-standard things that confused the LLMs which have been trained on vast amounts on very standard code and hence kept defaulting to those assumptions. Maybe a rough analogy is that he was trying to "code golf" this repo whereas LLMs kept trying to write "enterprise" code because that is overwhelmingly what they have been trained on.
I think this is where the chat-oriented / pair-programming or spec-driven model shines. Over multiple conversations (or from the spec), they can understand the context of what you're trying to do and generate what you really want. It seems Karpathy has not tried this approach (given his comments about "autocomplete being his sweet spot".)
For instance, I'm working on some straightforward computer vision stuff, but it's complicated by the fact that I'm dealing with small, low-resolution images, which does not seem well-represented in the literature. Without that context, the suggestions any AI gives me are sub-optimal.
However, after mentioning it a few times, ChatGPT now "remembers" this in its context, and any suggestion it gives me during chat is automatically tailored for my use-case, which produces much better results.
Put another way (not an AI expert so I may be using the terms wrong), LLMs will default to mining the data distribution they've been trained on, but with sufficient context, they should be able to adapt their output to what you really want.
(I did listen to a sizable portion of this podcast while making risotto (stir stir stir), and the thought occurred to me: “am I becoming more stupid by listening to these pundits?” More generally, I feel like our internet content (and meta content (and meta meta content)) is getting absolutely too voluminous without the appropriate quality controls. Maybe we need more internet death.)
I don't follow. If we, in some distant future, find a way to make humans functionally immortal, does that magically remove our agency? Or do we not have agency to begin with?
If your position on the "free will" question is that it doesn't exist, then sure I get it. But that seems incompatible with the death prerequisite you have put forward for it, because if it doesn't exist then surely it's a moot point to talk prerequisites anyway.
Every AI lab brags how "more agentic" their latest model is compared to the previous one and the competition, and everybody switches to the new model.
In a nutshell we are mimicking neural activity in a certain region based on certain abstracted data which is quite removed from how we as humans process reality.
And why witnesses are preferably interviewed very shortly after they witnessed a crime. Before their brains start to 'fill in the blanks'
He is singlehandedly enabling millions of people to understand what is going on, what + and * do, actually demystifying the "wires".
I just wish he start thinking of himself as more than 'collapsing weights', regardless if it turns out to be true.
> The models have so many cognitive deficits. One example, they kept misunderstanding the code because they have too much memory from all the typical ways of doing things on the Internet that I just wasn’t adopting.
> I also feel like it’s annoying to have to type out what I want in English because it’s too much typing. If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code.
> They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it’s okay. I don’t need all this extra stuff in there. So I feel like they’re bloating the code base, bloating the complexity, they keep misunderstanding, they’re using deprecated APIs a bunch of times. It’s a total mess. It’s just not net useful. I can go in, I can clean it up, but it’s not net useful.
Why? Because humans—including the smartest of us—are continuously prone to cognitive errors, and reasoning about the non-linear behavior of complex systems is a domain we are predictably and durably terrible at, even when we try to compensate.
Personally I consider the case of self-driving cars illustrative and a go-to reminder for me of my own very human failure in this case. I was quite sure that we could not have autonomous vehicles in dynamic messy urban areas without true AGI; and that FSD would in the fashion of the failed Tesla offering, emerge first in the much more constrained space of the highway system. Which would also benefit from federal regulation and coordination.
No Waymos have eaten SF, and their driving is increasingly nuanced; and last night a friend and very early adopter relayed a series of anecdotes about some of the strikingly nuanced interactions he'd been party to recently, including being in a car that was attacked late at night, and, how one did exactly the right thing when approached head-on in a narrow neighborhood street that required backing out. Etc.
That's just one example, and IMO we are only beginning to experience the benefits of "network effects" so popular in tails of singularity take-off.
Ten years is a very, very, very long time under current conditions. I have done neural networks since the mid-90s (academically: published, presented, etc.) and I have proven terrible in anticipating how quickly "things" will improve. I have now multiple times witnessed my predictions that X or Y would take "5-8" or "8-10" years or "too far out to tell," instead arrive within 3 years.
Karpathy is smart of course but he's no smarter in this domain than any of the rest of us.
Are scaled tuned transformers with tack-ons going to give us AGI in 18 months? "No" is a safe bet. Is no approach going to give us AGI inside of 5 years? That is absolutely a bet I would never make. Not even close.
Today we have an extraordinary invention—comparable to the wheel in its time. That invention is: predictive inference over all human knowledge. Period. I don't like calling it "Artificial Intelligence" because it's not intelligence; it's a prediction system that can project responses by illuminating patterns across all human knowledge encapsulated in text, audio, and video. What companies like OpenAI call "reasoning" models is simply that predictive process, but in a loop packaged as a product—one of the first marvelous uses of this fascinating invention: predictive inference over all human knowledge.
When the wheel was invented, no one could have imagined that, combined with hundreds of subsequent technologies, it would enable an electric car powered by solar energy. The wheel wasn't autonomous transportation—it was a fundamental component.
I see two debates getting mixed up here:
- The debate about the current invention: A tool that makes encyclopedias "speak" by connecting patterns across all human knowledge. As a tool, that's what it is—nothing more, nothing less. Tremendously useful, but a tool.
- The debate about the future dream: What this invention might enable when combined with hundreds of technologies that don't yet exist—similar to imagining an electric car when you only have the wheel.
It seems many experts are taking positions and getting "upset" because they're mixing these two debates. Some evaluate the wheel as if it should already be a solar electric car. Others defend the wheel by saying it already IS a solar electric car. Both are right in their observations, but they're talking about different things.
LLMs are a fundamental breakthrough—the "wheel" of the information age. But discussing whether they "understand" or have "world models" is like asking whether the wheel "comprehends transportation."
On the danger of confusing capabilities: Conflating the tool with the end goal leads us to poor decisions—from over-investment to under-utilization. When we expect AGI from what is fundamentally a pattern-matching engine, we set ourselves up for disappointment and misallocation of resources. No magic, just reality.
The temporal factor: The AGI debate is a debate about the future—about what might emerge from combinations of technologies we haven't yet invented.
A pattern I noticed in a AI[sic] discussions: Handwavily declaring what intelligence is not, while not explaining what is.
Just like how a wheel moves stuff, the internet is the medium through which bits are transmitted and received.
2029: Human-level AI
2045: The Singularity - machine intelligence 1 billion times more powerful than all human intelligence
Based on exponential growth in computing. He predicts we'll merge with AI to transcend biological limits. His track record is mixed, but 2029 looks more credible post-GPT-5. The 2045 claim remains highly speculative.
Hegel thought history ended with the Prussian state, Fukuyama thought it ended in liberal America, Paul thought judgement day was so close you need not bother to marry, the singularity always comes around when the singularians get old. Funny how that works
The overwhelming majority of all gains in human life expectancy have come due to reductions in infant mortality. When you hear about things like a '40' year life expectancy in the past it doesn't mean that people just dropped dead at 40. Rather if you have a child that doesn't make it out of childhood, and somebody else that makes it to 80 - you have a life expectancy of ~40.
If you look back to the upper classes of old their life expectancy was extremely similar to those of today. So for instance in modern history, of the 15 key Founding Fathers, 7 lived to at least 80 years old: John Adams, John Quincy Adams, Samuel Adams, Jefferson, Madison, Franklin, John Jay. John Adams himself lived to 90. The youngest to die were Hamilton who died in a duel, and John Hancock who died of gout of an undocumented cause - it can be caused by excessive alcohol consumption.
All the others lived into their 60s and 70s. So their overall life expectancy was pretty much the same as we have today. And this was long before vaccines or even us knowing that surgeons washing their hands before surgery was a good thing to do. It's the same as you go back further into history. A study [1] of all men of renown in Ancient Greece was 71.3 [1], and that was from thousands of years ago!
Life expectancy at birth is increasing, but longevity is barely moving. And as Kurzweil has almost certainly done plentiful research on this topic, he is fully aware of this. Cognitive dissonance strikes again.
The merge with a machine 1 million times more intelligent than us is the same as letting AI use our bodies. I'd rather live in cave. Iirc, the 7th episode of Black Mirror starts with this plot line.
Space flight?
"When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.
That’s why these things take so long."
If you need to get to 9 9s, the 9th 9 could be more effort than the other 8 combined.
So to make predictions about general intelligence is just crazy.
And yeah yeah I know that OpenAI defines it as the ability to do all economically relevant tasks, but that's an awful definition. Whoever came up with that one has had their imagination damaged by greed.
I think you can get pretty far starting from behavior and constraints. The brain needs to act in such a way as to pay for its costs. And not just day to day costs, also ability to receive and give that initial inheritance.
From cost of execution we can derive an imperative for efficiency. Learning is how we avoid making the same mistakes and adapt. Abstractions are how we efficiently carry around past experience to be applied in new situations. Imagination and planning are how we avoid the high cost of catastrophic mistakes.
Consciousness itself falls from the serial action bottleneck. We can't walk left and right at the same time, or drink coffee before brewing it. Behavior has a natural sequential structure, and this forces the distributed activity in the brain to centralized on a serial output sequence.
My mental model is that of a structure-flow recursion. Flow carves structure, and structure channels flow. Experiences train brains and brain generated actions generate experiences. Cutting this loop and analyzing parts of it in isolation does not make sense, like trying to analyze the matter and motion in a hurricane separately.
Simulating that is a long way away - so the only possibility is that brains have some sort of redundancy and we can optimise that away. Though computers are faster than brains so its possible maybe, how much faster? So lets say a neuron does its work in a mS and we can simulate this work in 1uS, ie a thousand times faster - thats still a lot. Can we get to a million times faster? even then its still a lot. Not to mention the power required for this.
Even if we can fit a million neurons in a CPU thats still 90 million CPU's. Only 10% are active say, still 9 million CPU's, a thousand times faster - 9,000 cpu's nearly there but still a while away.
Ultimately this comes down to the philosophy of language and of the history of specific concepts like intelligence or consciousness - neither of which exist in the world as a specific quality, but are more just linguistic shorthands for a bundle of various abilities and qualities.
Hence the entire idea of generalized intelligence is a bit nonsensical, other than as another bundle of various abilities and qualities. What those are specifically doesn’t seem to be ever clarified before the term AGI is used.
A new contribution by quite a few prominent authors. One of the better efforts at defining AGI *objectively*, rather than through indirect measures like economic impact.
I believe it is incomplete because the psychological theory it is based on is incomplete. It is definitely worth discussing though.
—-
In particular, creative problem solving in the strong sense, ie the ability to make cognitive leaps, and deep understanding of complex real-world physics such as the interactions between animate and inanimate entities are missing from this definition, among others.
The important thing is that this is not going to be perpetually 20 years in the future like fusion. This is something that will happen.
SOTA models are already capable of outperforming any human on earth in a dizzying array of ways, especially when you consider scale.
Humans also produce nonsensical, useless output. Lots of it.
Yes, LLMs have many limitations that humans easily transcend.
But few if any humans on earth can demonstrate the breadth and depth of competence that a SOTA model possesses.
Relatively few (probably less than half) are casually capable of the level of reasoning that LLMs exhibit.
And, more importantly, as anyone in the field when neural networks were new is aware, AGI never meant human level intelligence until the LLM age. It just meant that a system could generalize one domain from knowledge gained in other domains without supervision or programming.
Most humans can count the occurrence of letters in a word. The word competence here is doing quite a bit of work. I think most people understand competence to mean more than just encyclopedic knowledge, with very limited reasoning capability.
> AGI never meant human level intelligence until the LLM age. It just meant that a system could generalize one domain from knowledge gained in other domains without supervision or programming.
I think it's probably correct to say that many people who seriously studied the problem had a larger notion of AGI than the layperson who only ever talked about the Turing test in the most basic terms. Also, I don't think LLMs have even convincingly demonstrated a great ability to generalize.
They're basically really great natural language search engines but for the fact that they give incorrect but plausible answers about 5-10% of the time.
So why are so many people still employed as e.g. software engineers? People aren’t prompting the models correctly? They’re only asking 10 times instead of 20? They’re holding it wrong?
I think it's fair to do it to the idea of AGI.
Moving the goalpost is often seen as a bad thing (like, shifting arguments around). However, in a more general sense, it's our special human sauce. We get better at stuff, then raise the bar. I don't see a reason why we should give LLMs a break if we can be more demanding of them.
> SOTA models are already capable of outperforming any human on earth in a dizzying array of ways, especially when you consider scale.
Performance should include energy consumption. Humans are incredibly efficient at being smart while demanding very little energy.
> But few if any humans on earth can demonstrate the breadth and depth of competence that a SOTA model possesses.
What if we could? What if education mostly stopped improving in 1820 and we're still learning physics at school by doing exercises about train collisions and clock pendulums?
There is a lot of success already in adaptive learning in elementary school for instance, my kids are blasting through math on Prodigy and it seems like Synthesis may be a great tool as well, and I believe we're just at the beginning of this wave. For that level of learning I don't think we need incredibly more capability, just better application.
I want something far more interactive that leaves me far more in control and forces me to be responsible for the choices.
For the last two months as I've been out of paid work I've been working like mad on my open source project, and fell into the pattern of heavily using Claude Code and some of the results have been amazing but some I have let my judgment and oversight lapse and come back later with a completely "WTF did it do here?" surprise.
That shouldn't be allowed to happen. A responsible SWE culture would demand that these tools engage in a way that encourages heavy oversight review and engagement.
Almost everybody does mandatory code review process these days (they didn't earlier in my career) ... despite its lower velocity... because of lessons learned -- and yet now we're allowing agent coding to produce large quantities of code that doesn't even lend itself to review by the party in charge of producing it.
Dealing with Rust's borrow checker issues, how complex C++ might be, Go's approach to language design, Java vs C#, and whatever else in the same vein, will slowly be matter of discussion to a selected few, while everyone else is promoting or doing voice dictation, creating kaban tickets for agents.
- The length of tasks AI can complete doubles every ~7 months
- In 2-4 years, AIs could autonomously complete week-long projects.
- In under 10 years, they might handle month-long software or knowledge work.
[1] https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
* At least, where we had a PM. The places I was self-directed could arguably provide an interesting comparison.
Fundamentally, AGI requires 2 things.
First it needs to be able to operate without information, learning as it goes. The core kernel should be such that it doesn't have any sort of training on real world concepts, only general language parsing that it can use to map to some logic structure to be able to determine a plan of action. So for example, if you give the kernel the ability to send ethernet packets, it should eventually figure out how to talk tls to communicate with the modern web, even if that takes an insane amount of repetition.
The reason for this is that you want the kernel to be able to find its way through any arbitrarily complex problem space. Then as it has access to more data, whether real time, or in memory, it can be more and more efficient.
This part is solvable. After all, human brains do this. A single rack of Google TPUs is roughly the same petaflops as a human brain operating at max capacity if you assume neuron activation is a add-multiply and firing speed of 200 times/second, and humans don't use all of their brain all the time.
The second part that makes the intelligence general is the ability to simulate reality faster than reality. Life is imperative by nature, and there are processes with chaotic effects (human brains being one of them), that have no good mathematical approximations. As such, if an AGI can truly simulate a human brain to be able to predict behavior, it needs to do this at an approximation level that is good enough, but also fast enough to where it can predict your behavior before you exhibit it, with overhead in also running simulations in parallel and figuring out the best course of actions. So for a single brain, you are looking at probably a full 6 warehouses full of TPUs.
Read that sentence again. Slowly.
What do you think "general language parsing" IS if not learned patterns from real-world data? You're literally describing a transformer and then saying we need to invent it.
And your TLS example is deranged. You want an agent to discover the TLS protocol by randomly sending ethernet packets? The combinatorial search space is so large this wouldn't happen before the sun explodes. This isn't intelligence! This is bruteforce with extra steps!
Transformers already ARE general algorithms with zero hardcoded linguistic knowledge. The architecture doesn't know what a noun is. It doesn't know what English is. It learns everything from data through gradient descent. That's the entire damn point.
You're saying we need to solve a problem that was already solved in 2017 while claiming it needs a century of quantum computing.
Eg for a programming LLM with an agentic agent and access to a computer, would be able to, given design-doc.md and Todo.md, implement feature X, making sure it compiles, run some basic smoke tests, write appropriate unit tests, make sure they all pass, and finally push the code and create a draft PR.
Naturally, not every call into the agent is going to take the full 10 minutes. It may need to ask questions before getting started, or stop if there's an unrecoverable error. Sometimes you'll just need to tell it "continue", but the system should be capable of a 10-minute run (hopefully longer!) given enough support.
That's an "agent" at its simplest -- a LLM able to derive from natural language when it is contextually appropriate to call out to external "tools" (i.e. functions).
>There's some process of distillation into the weights of my brain. This happens during sleep and all this stuff. We don't have an equivalent [in LLMs] (23:09)
Seems to me that's one of the big things lacking in LLMs vs human thinking. People say LLMs can't lead on to AGI but that kind of thing is an avenue they could explore.
• Text and graphs: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
• Video interview: https://www.youtube.com/watch?app=desktop&v=evSFeqTZdqs
That said, I've not seen work that looks promising to the problem of, as he phrased it: "They don’t have continual learning. You can’t just tell them something and they’ll remember it."
Saying any specific timeframe for that, 10 years or anything else, seems too certain. Some breakthrough might already exist and be unknown, but on the other hand it may require a fundamental advancement in mathematics in order to make it possible to find something at least close to optimal in a billion-dimensional (or whatever) vector space with only the first few dozen examples.
I feel LLMs are fairly capable when it comes to doing each of those steps in isolation. But not when it is all put together as a process.
We are still on trend by projections to reach human parity in many domains by 2027-2028, the only thing that would prevent this is a major unexpected slowdown in AI progress.
This is the reflexive/reflective distinction (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...). Reflexive comments—the kind that express some pre-existing feeling or opinion that happens to get triggered by association—are much faster to produce, so unfortunately they show up first in many threads.
If you’re correct, there’s not much reward aside from the “I told you so” bragging rights, if you’re wrong though - boy oh boy, you’ll be deemed unworthy.
You only need to get one extreme prediction right (stock market collapse, AI taking over, etc ), then you’ll be seen as “the guru”, the expert, the one who saw it coming. You’ll be rewarded by being invited to boards, panels and government councils to share your wisdom, and be handsomely paid to explain, in hindsight, why it was obvious to you, and express how baffling it was that no one else could see what you saw.
On the other hand, if predict an extreme case and you get it wrong, there’s virtually 0 penalties, no one will hold that against you, and no one even remembers.
So yeah, fame and fortune is in taking many shots at predicting disasters, not the other way around.
You have one decade to clean up your power use problem. If you don't you will find yourself in the next AI winter.
AGI is either more scale or differing systems, or both
They can always optimize for power consumption after AGI has been reached
I don't find it very courteous to say that you're steelmanning someone's argument. Sutton is certainly smart enough to have steelmanned his argument himself. Steelmanning : do it in your head, don't say it!
Software can already write more text on any given subject better than a majority of humanity. It can arguably drive better across more contexts than all of humanity - any human driver over a billion miles of normal traffic will have more accidents than self driving AI over the same distance. Short stories, haikus, simple images, utility scripts, simple software, web design, music generation - all of these tasks are already superhuman.
Longer time horizons, realtime and continuous memory, a suite of metacognitive tasks, planning, synthesis of large bodies of disparate facts into novel theory, and a few other categories of tasks are currently out of reach, but some are nearly solved, and the list of things that humans can do better than AI gets shorter by the day. We're a few breakthroughs away, maybe even one big architectural leap, from having software that is capable (in principle) of doing anything humans can do.
I think AGI is going to be here faster than Kurzweil predicted, because he probably didn't take into consideration the enormous amount of money being spent on these efforts.
There has never been anything like this in history - in the last decade, over 5 trillion dollars has been spent on AI research and on technologies that support AI, like crypto mining datacenters that pivoted to AI, new power, water, data support, providing the infrastructure and foundation for the concerted efforts in research and development. There are tens of thousands of AI researchers, some of them working in private finance, some for academia, some doing military resarch, some doing open source, and a ton doing private sector research, of which an astonishing amount is getting published and shared.
In contrast, the entire world spent around 16 trillion dollars on world war II - all of the R&D and emergency projects and military logistics, humanitarian aid, and so on.
We have AI getting more resources and attention and humans involved in a singular development effort, pushing toward a radical transformation of the very concept of "labor" - while I think it might be a good thing if it is a decade away, even perpetually so until we have some reasonable plan for coping with it, I very much think we're going to see AGI within the very near future.
*When I say "in principle" I mean that given the appropriate form factor, access, or controls, the AI can do all the thinking, planning, and execution that a human could do, at least as well as any human. We will have places that we don't want robots or AI going, tasks reserved for humans, traditions, taboos, economics, and norms that dictate AI capabilities in practice, but there will be no legitimacy to the idea that an AI couldn't do a thing.
Fusion research lives and dies on this premise, ignoring the hard problems that require fundamental breakthroughs in areas such as materials science, in favor of touting arbitrary benchmarks that don't indicate real progress towards fusion as a source of power on the grid.
"Full self driving" is another example; your car won't be doing this, but companies will brag about limited roll-outs of niche cases in dry, flat, places that are easy to navigate.
Not a rigorous model.
But let's be honest; software development at a modern startup is already the upper bound of applied intelligence. You're juggling shifting product specs, ambiguous user feedback, legacy code written by interns, and five competing JS frameworks, all while shipping on a Friday. Models can now do that. They can reason about asynchronous state, refactor a codebase across thousands of lines, and actually explain the difference between useEffect and useLayoutEffect without resorting to superstition.
If that's not general intelligence, what exactly are we waiting for - self-awareness?
The hubris and myopia is staggering.
The next 2-3 years are going to be incredibly interesting.
I don't know how much wish fulfilment there is in people's timelines.
I was a lot disappointed when he went to work for Tesla, and I think that he had some achievement there, butnot nearly the impact I believe he potentially has.
His switch (back?) to OpenAI was, in my mind, much more in keeping with where his spirit really lies.
So, with that in mind, maybe I've drunk too much kool aid, maybe not. But I'm in agreement with him, the LLMs are not AGI, they're bloody good natural language processors, but they're still regurgitating rather than creating.
Essentially that's what humans do, we're all repeating what our education/upbringing told us worked for our lives.
But we all recognise that what we call "smart" is people recognising/inventing ways to do things that did not exist before. In some cases its about applying a known methodset to a new problem, in others its about using a substance/method in a way that other substances/methodsets are used, but the different substance/methodset produces something interesting (think, oh instead of boiling food in water, we can boil food in animal fats... frying)
AI/LLMs cannot do this, not at all. That spark of creativity is agonisingly close, but, like all 80/20 problems, is likely still a while away.
The timeline (10 years) - it was the early 2010s (over 10 years ago now) that the idea of backward propagation, after a long AI winter, finally came of age. It (the idea) had been floating about since at least the 1970s. And that ushered in the start of our current revolution, that and "Deep Learning" (albeit with at least another AI winter spanning the last 4 or 5 years until LLMs arrived)
So, given that timeline, and the restraints in the currrent technology, I think that Andrej is on the right track, and it will be interesting to see where we are in ten years time.
I'm sure the US economy has ten more years of the data centre money, it'll be fine.
I wonder how far off the "Sell it all — today" Margin Call moment is.
From skimming the conversation it seems to mostly revolve around LLMs (transformer models) which is probably not going to be the way we obtain AGI to begin with, frankly it is too simple to be AGI, but the reason why there's so much hype is because it is simple to begin with so really I don't know.
1. This is the death knell for the the "AI" investment bubble. Agents that are useful for non-devs are 10 years away.
2. Andrej thinks that GPT5 pro is SOTA for code? Really? As a Sonnet normie.. can anyone please help me understand this?
edit:
3. You can't see any major tech developments on the GDP growth chart? Really? WTF? Have we all been smoking tech crack, this whole time? So GDP didn't grow extra from tech any single tech development, like the Internet? This broke my brain.
disclaimer: On the daily, I use LLM dev tools to add amazing LLM-enabled features to my pre-money SaaS. It's really cool and users love the features.
For extremely complex multi-step problems though - it may need some help in breaking the tasks down to more manageable chunks. But will eventually ace it. As an example, I had good success with a project that involved:
- Rewriting all internals in a dotnet/C# application to use Apache Arrow types for data through the entire pipeline - Adapting the architecture to be streaming first instead of working through entire data in each stage - Designing and implementing a complex system that creates many different projections of the data based on everything that has read in the stream so far and create multiple outputs based on that, in parallel as the stream is being read in real-time - Recreating a prototype of the entire project in Rust
(I was in college during the first AI Winter, so... I can't help but think that the cycles are tighter but convergence isn't guaranteed.)
Most of these companies value is built on the idea of AGI being achievable in the near future.
AGI being too close or too far away affects the value of these companies- too close and it'll seem too likely that the current leaders will win. Too far away and the level of spending will seem unsustainable.
Is it? Or is it based on the idea a load of white collar workers will have their jobs automated, and companies will happily spend mid four figures for tech that replaces a worker earning mid five figures?
This 2024 story feels like ancient history that everyone has forgotten: https://www.cnbc.com/2024/02/09/openai-ceo-sam-altman-report...
Therefore turning autonomous actors based on LLMs loose is a recipe for disaster.
It won’t take a decade. That’s an arbitrary statement based on a big unknown. It will take an entirely new technology. One we haven’t invented yet, one I can’t even imagine. One that is consistently accurate and reliable in ways NO EXISTING AI PRODUCT HAS EVER BEEN.
We do not know how "far away" we are from "AGI" period. It's also useless. If you're correct...so what? Someone may have been able to perfectly predict the advent of railway travel. Guess what, this gave them 0 advantage unless they already had tons of capital to invest, which is effectively what makes the realization of the predicted thing come to fruition in the first place. Bets like these are at best self-fulfilling prophecies if you are a billionaire and at worst ideal chatter that makes us all stupider the more time we waste on them and the more we let wildly unchecked claims like this dictate behaviors in the present that actually affect us.
The people heralding the emergence of AGI are doing little more than pushing Ponzi schemes along while simultaneously fueling vitriolic waves of hate and neo-luddism for a ground-breaking technology boom that could enhance everything about how we live our lives... if it doesn't get regulated into the ground due to the fear they're recklessly cooking up.
https://news.ycombinator.com/item?id=45622944
Must have been the flu-brain misfiring
People are starting to get catch on, but most non-tech people don’t use LLMs for anything more than simple questions that can be easily answered by summarizing regurgitated snippets of training data. To them, it looks intelligent. And yeah, the humans who wrote the training samples it regurgitated probably were intelligent.
It’s just a fact, one that becomes glaringly obvious when you use LLMs daily to do real work, that this is just not the tech that will lead to AGI.
They found a really clever pattern matching technique that, when combined with absurd amounts of data and compute, can reproduce plausible summaries of training data which can be stitched together in useful ways. It’s a useful tool. But the whole AGI conversation is so absurdly far away from this that it’s just clear that these guys pushing a very dishonest grift.
But nothing will make grifters richer than promising it's right around the corner.
Why is there a presumption that we (as people who have only studied CS) know enough about biology/neuroscience/evolution to make these comparisons/parallels/analogies?
I enjoy the discussions but I always get the thought in the back of my head "...remember you're listening to 2 CS majors talk about neuroscience"
I suspect the average AI researcher knows much more about the brain than typical CS students, even if they may not have sufficient background to conduct research.
Once I started to realize just how much of the brain is inscrutable, because it is a machine operating on chemicals instead of strict electrical processing, I became a lot more reluctant to draw those comparisons
well it's straightforward. First lets assume a spherical, perfectly frictionless, brain..
You can make some comparisons between how they perform without really understanding how LLMs or brains work, like to me LLMs seem similar to the part human minds where you say stuff without thinking about it. But you never really get an LLM saying I was thinking about that stuff and figured this bit was wrong, because they don't really have that capability.
Hubris.
There are two periods in history that "feel" like this time to me: - prior to Einstein's theory of relativity and - the uncovering of quantum mechanics.
In both cases bits and pieces of math and science were floating in the air but no one could connect them. It took teams of people/individuals and years of arduous effort to pull it all together.
Today there are a lot more participants. Main difference seems that a lot of them seem to be capitalists!8-))
I know it's against the guidelines to discuss the state of a thread, but I really wish we could have thoughtful conversations about the content of links instead of title reactions.
I believe this distinction is pretty fundamental to humans, so we're not likely to escape it, but the good news is that reflective comments do show up eventually if the article is substantive and the reflexive ones haven't ruined the thread. We also try to downweight the more reflexive subthreads.
Granted, a bunch of commenters are probably doing what you’re saying.
That includes anyone reading this message long after the lives of those reading it on its post date have ended.
Which of course raises the interesting question of how I can make good on this bet.
However, don't let the bandwagon ( from either side ) cloud your judgment. Even warm fusion or any fusion at all is still very useful and it's here to stay.
This whole AGI and "the future" thing is mostly a VC/Banks and shovel sellers problem. A problem that has become ours too because the ridiculous amounts of money "invested", so even warm fusion is not enough from an investment vs expectations perspective.
They are already playing musical money chairs, unfortunately we already know who's going to pay for all of this "exuberance" in the end.
I hope this whole thing crashes and burns as soon as possible, not because I don't "believe" in AI, but because people have been absolutely stupid about it. The workplace has been unbearable with all this stupidity and amounts of fake "courage" about every single problem and the usual judgment of the value of work and knowledge your run-of-the-mill dipshit manager has now.
The debate about AGI is interesting from a philosophical perspective, but from a practical perspective AI doesn't need to get anywhere close to AGI to turn the world upside down.
And real AI is probably like fusion. Its always 10 years away.
AI has now been revealed to the masses. When AGI arrives most people will barely notice. It will just feel like slightly better LLMs to them. They will have already cemented notions of how it works and how it affects their lives.
The rate depth, breadth and frequency of releases has only increased, not decreased. Meanwhile, everyone is waiting on bated breath for Gemini 3 to drop. A decade for reliable agents is not only comical, but willful cognitive dissonance at this point.