Hacker News Reader

Comment by Zigurd | Hacker News Reader

LLMs, plus broadly sourced yet expertly curated training sources, plus clever harnesses, plus RAS, etc. do an ever better job of synthesizing their training set into useful responses. For some use cases like coding, that's very useful now and likely to get at least somewhat better before reaching limitations based on the training set.

That's not going to reach AGI, mainly because today's recipe for AI products isn't built to be AGI. Some people believe it will reach AGI because the performance and applicability of LLMs was emergent. There's a case to be made that AGI could be similarly emergent. After all, what we intuitively call our consciousness emerged from a network of neurons.

I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.

View on HN

> That's not going to reach AGI,

It has not been even 4 years since ChatGPT hit and LLMs + Transformers + Whatever they do has gotten us to solving millennium problems.

4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.

Now I don't know if what we have is AGI or not but I do not understand how you can see what has happened in the last 3 years and say "it will not get us there" no matter what "there" is.

> 4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.

I keep seeing this idea and I don't understand the reasoning behind it.

I think it could be a bit like saying if you showed someone 500 years ago a smartphone they would likely conclude at first it was magic. But once you had some time to let them use it and tell them how it all worked on a high level they would eventually obviously realise, no, it's not magic.

I guess just in the same way if you presented current LLM tech out of nowhere a few years ago to someone who'd never seen it, I concede they may be likely to imagine it was AGI in that first conversation, depending on their background.

But after using it for a bit and learning what an LLM is etc they'd land exactly where everyone is today - a great technology useful for some things, not AGI, not magic.

Wait, are you literally saying that modern LLMs are not "magic" (or AGI) because we know how it works?

If that is the case, then it trivially follows that humanity will never be able to build AGI, because every time we build something new, we will understand how we built it, which disqualifies it from being an AGI.

But that doesn't seem like a useful definition to me.

(Edit: I fully agree with your comment, just wanted to pick on a small point)

Thing is, we don’t understand how LLMs work. Not really. And everyone at the forefront of this space has said so. We know how to grow them but that is not anything like the same thing as understanding why or how they actually function. Our level of understanding of LLM intelligence is only slightly ahead of our understanding of the neuroscience of human intelligence, and no one would say we understand that at all.

So I think describing LLMs as magic at this point is absolutely appropriate.

I know how they work pretty well and most days I still have moments where I am struck by how bizarre and magical these thinking machines are.

The fact that a monkey can pick up an artifact a size of a small mirror and do a video chat, see and talk to another monkey on tge other side of the planet is absolutely bizarre. There's no good reason why this should be allowed.

I'm finding it more bizarre, if compared to creating a virtual talking monkey from scratch, with a little bit of Python and a large calculator.

What ? If that experiment was done on monkeys I don't think they had to use Android / iOS / iPadOS interface themselves. All they had to do is push a button

What would convince you that it is AGI?

I always here things like "oh it's useful but dumb on some things", but it's just vague.

What is the test? What is a question that it fails at compared to humans? And no, you can't just say "find me the cure for cancer", but I believe there is probably enough intelligence in the weights that there is likely a cure in there with enough compute and the right questions.

The fact that you need to ask "the right questions" is why it's not AGI. A general intelligence should be able to ask of its own volition the interesting questions required to advance its goals.

Why?

Humans don't ask the right questions otherwise they could get the answers w enough compute. So humans aren't agi?

Again w the vagueness. It's not agi because it doesn't ask the right questions. What are the right questions? Who knows.

You're proving my point

Most humans can't and don't ask these questions.

we're not talking about most humans here, we're talking about the capabilites of the universe of humans.

AGI doesn't mean Omniscience

I can’t predict how a novel intelligence could prove to me that it is intelligent. A novel intelligence would have to work out how to do that for itself

When it can replace a white collar remote worker without humans team realizing they work with the machine. And no, Agents are not like this whatever initial prompt and set of skills you give them. They still won't progress, learn and apply that knowledge like a human.

sounds like the Turing Test all over again

> What is the test? What is a question that it fails at compared to humans?

Human intelligence does not consist of answering questions.

And every day I have to say "No, you stupid clanker ...", so if this is "general intelligence", it's the intelligence of an idiot savant who is good at calculations but sucks at most other forms of intelligence.

i think we'll know agi when we see it, but we can't really predict what that will look like

> 4 years ago, a program that could [...] we would have called it AGI

If you had told someone in the 1800s that a machine could instantly multiply 100 digit numbers, that would have been considered dazzlingly intelligent. And yet we are not that dazzled by our calculators today (despite how useful they might be!).

Are you trying to explain how things once considered dazzling get normalized over time? Because otherwise this is a non-sequitur and has no bearing on the trivially verifiable, exponential explosion of capabilities we have seen in the last 4 years.

I keep saying this, until ChatGPT came out 4 years ago it was basically unimaginable that a single model could do any of, let alone all, the things they are doing today. Like, seriously, go take a look at the state of the art in NLP and NLU, the very first challenge in getting computers to even “understand” natural language, let alone other things like reasoning. Everything it does automatically was once a heavily experimental deep research field with long glorious careers for the researchers.

And now it’s all gone because the Bitter Lesson won again. If that’s not general enough to qualify for the G in AGI I don’t know what it is. And we’re sitting here going, “But it sometimes writes bad code though.”

It's not non sequitur at all ... it points out that the argument that, because there has been a rapid and surprising advance, therefore it's AGI, is intellectually bankrupt.

> If that’s not general enough to qualify for the G in AGI

Indeed it isn't.

> I don’t know what it is.

Sorry to hear it, but some people do.

The way i understand the way you define AGI is, that the lower bound of the Spiky profile is at least on Par with the best of humanity. This way no human can say "But it cant do X as well as Mr./Mrs. Y"

The way that many other here see it is that the fact that it has lots of spikes in many different directions and some lower bounds above the level of a 10 year old, is indeed enough generality.

Probably the G in AGI is a spectrum, and it it is now a maturing process to lift the lower bounds of the spiky profile of to the best of humanity. If this will ever work with an alien intelligence is debatable. Even in the "Culture Series" the ASI had lower bounds of some skill sets that were below the best of Humanity, and thus the "Contact" department was formed, to use these humans to cover the shortcomings the ASI could not. It might indeed be our chance for a symbiosis to make up for some of these lower spike bounds.

In the end who is to say that our intelligence is not spikey, and me are just normalizing it to a circle..

Speak for yourself, I am dazzled by calculators!

In any case, I think this misses OP's point that LLM capabilities have rapidly made progress towards being more generally intelligent and capable, which is not true of most tech advances.

This is a motte & bailey moment. Parent comment stated something much sharper, that I responded to:

> I do not understand how you can see what has happened in the last 3 years and say "it will not get us there" no matter what "there" is.

Your statement is something much weaker, and I would still question what exactly "general" means when AI capabilities are commonly accepted to be so "jagged".

Maybe my phrasing is too weak but if the parent comment is the 'bailey', I fully agree with it.

The last 3 years of progress have been so explosive and, yes, general that it seems crazy to fully rule out dramatic future progress.

When people have a very narrow 'confidence interval' about their AI predictions, in either direction, it's difficult to trust them.

> it seems crazy to fully rule out dramatic future progress.

Nothing is gained by attacking strawmen. Of course there will be dramatic future progress. That does not entail that AGI will be achieved.

P.S. It's hard to deal with so much inconsistency. This person says they fully agree with

> I do not understand how you can see what has happened in the last 3 years and say "it will not get us there" no matter what "there" is.

and at the same time says

> When people have a very narrow 'confidence interval' about their AI predictions, in either direction, it's difficult to trust them.

and says

> The last 3 years of progress have been so explosive and, yes, general that it seems crazy to fully rule out dramatic future progress

and then moves those goalposts to "AGI, RSI, the apocalypse, whatever".

Fine ... let's not trust folks who are so confident that the progress in last 3 years entails that "it will get us there" where there includes AGI. Again, it's an argument against a strawman to say "it seems crazy to fully rule out dramatic future progress" -- everyone agrees that there will be dramatic future progress. But it's obnoxious naysaying to insist that "the last 3 years of progress" entails future AGI, when numerous people have explained why they think it doesn't. [Generally speaking, it's not a good idea to characterize as "crazy" people you're trying to persuade.]

And qualifiers matter ... so when someone says [emphasis added]

> yet we are not that dazzled by our calculators today (despite how useful they might be!)

it is yet another strawman argument to respond

> Speak for yourself, I am dazzled by calculators!

And on the "general" front, there's a big difference between AI that can address a wide range of problems -- what one might call gAI for varying values of "general" -- and AGI, an artificial replication of the sort of generalized intelligence found in human beings, who can learn to fish by being shown how to fish, or learn to repair an engine by being shown how to repair an engine, or learn how to play a game by being told the rules of the game, etc.

I read all this an am baffled about how you're seeing inconsistency.

All of these statements say the same thing: "People are sure about a low upper confidence bound for future AI. My upper confidence bound about AI is high; the future is massively uncertain"

>I think this misses OP's point that LLM capabilities have rapidly made progress towards being more generally intelligent and capable, which is not true of most tech advances.

A PC of today can accomplish many more "general" tasks than one of 40 years ago. Much of the "why" is because of the huge infrastructure built up around them in the meantime. The abilities of LLMs to accomplish those same tasks through the PC is heavily piggybacking on that (both in the specific, with the existence of all the APIs and tools; and in the generic, using search engines to find specific sources and using that for instruction or troubleshooting).

In the world of "agents" much of the improvement appears to have been on a specific set of skills: impersonation of an 'I' that wants to accomplish a goal, and synthesizing existing information from documents with trial-and-error execution loops to move rapidly toward a solution much faster and with less boredom than a human would. The quality of the output when there is not a rapid-evaluation-and-validation harness lags considerably.

It's incredibly powerful automation but doesn't appear to be trending towards Matrix-style conscious AIs. The quality of an individual method written by the agent also is not particularly advanced compared to GPT-4 in early 2023, as far as I can tell—I was dabbling with trying to make such harnesses back then, where a major challenge was that the model itself was bad at staying on-track in a conversation, so instead much of that logic was moved to deterministic code, which was much more limited as it was super-tedious to enumerate all the necessary tool calls/etc to find its way out of corners. Staying on task is much better now, as is "read compiler error, fix try next thing" harness loop-handling. But the output remains—across Fable, Astra, whatever else I've tried—"iffy" in terms of the actual code structure on the first pass output. You can set it then on a different task to review and clean up the code, and it can do that well too, but it is a curious gap of generality where the "create" focus is much more limited than the "review" one (and conversely the "review" focus can make suggestions, but if it goes deep down the well of implementing them, loses that big-picture again).

If it kills us all, it will because someone decided to give the trial-and-error-loop-machine access to nukes or similar. The blame for that is on the "someone" not on some sort of "rogue" AI.

(I wonder if re-watching Terminator/Terminator 2 would support this sort of interpretation of it. Unlike in the Matrix, I don't think we get much sentient-AI POV/infodumping. Is it a plausible universe for "someone made ChatGPT control a fleet of soldier robots and gave it a bad harness with an insufficient sandbox"?)

Actually, check out the story of Horizon Zero Dawn.

In the past 4 years the definition of what AGI would even be has been changed so many times that new marketing claims fit the description that the term itself no longer has any meaning.

However, it is reasonable to assume that the current architecture will just not be it. It works well for certain problems, and falls apart for several others.

Unrelated to your comment, more a general comment on the online discourse:

What I don't understand is why people get so defensive about and offended by the thought that more progress in the basics of the approaches could be needed. That doesn't change the performance of LLMs now, and it also doesn't change what could be possible in the future.

Why are people so deeply and personally invested in the current state of the art?

> we would have called it AGI.

This an absurdly circular argument. You are simply asserting that, if we had had today's systems 4 years ago, we would have would have called them AGI, even those who don't call them that today.

> Now I don't know if what we have is AGI or not

But you just asserted that we do.

> I do not understand how you can see what has happened in the last 3 years and say "it will not get us there"

They explained how. Your argument, OTOH, is nonsensical, like suggesting that, because a bike path was built rapidly, it will soon become a highway.

Superhuman performance at chess probably would have blown people’s minds in the 1950’s. We’ve since learned that sometimes intelligence can be narrow and sometimes it can be spiky, even if you can have a decent conversation.

It’s hard to point to anything and say it’s impossible. AGI doesn’t break any laws of physics. But some things like driverless cars can still be a long slog to get to widespread deployment.

Waymo is the only self driving car that works well enough to even try widespread deployment with pricing that isn't going to be an operating cash flow disaster. But they're going to have to at least double their footprint to put a meaningful dent in the billion dollars a year Waymo spends on R&D.

Waymo isn't LLM powered. Everyone who thought LLMs would make Waymo Driver obsolete were wrong. In part because Waymo can't be spiky. In part because practical applications of AI require a lot of time and effort.

I am amazed at how usable and useful coding agents have become since they were a hot mess about a year ago. I would not be at all amazed if there are only one or two other use cases that have the same favorable evolutionary trajectory.

Those are just the same capabilities than before, but with a much bigger compute power and training data behind it.

AGI can't be reached by "training harder" as, the way I see it at least, it requires a qualitative leap, not just quantitative.

We are getting a machine that better navigates across the information in its training data, we are not getting a machine that can think out of that training process, even if it can fool a few people at that.

Back then we couldn’t imagine something that could do all of those things and yet not feel intelligent or human.

The entire field has repeatedly said that for many decades.

https://aeon.co/essays/how-close-are-we-to-creating-artifici...

https://xkcd.com/605/

Here's an actual log-scale trajectory with a few dozen real data points.

https://metr.org/time-horizons/

Of what? We are talking about AGI, not solving software tasks.

(May 2026, no longer applicable)

> 4 years ago, a program that could create photorealistic pictures, talk to you in any language of the world and solve the hardest math problems that we know, we would have called it AGI.

No. General means general.

And yet, hand it simple task out of domain, it fails badly. If a system 4 years ago did all you describe without being trained to do those exact things with unfathomable volumes of data, i'd agree, but we don't have that. We have something that can replicate seen patterns effectively.

There is no reason to view the transformer vased LLM family as anythi g other than a translation machine. From the language of english spec sheet to python language script, for from image description to image, etc. The only real read we can take from current behaviour of llm's is that the relationship between domains we think of as seperate are likely joined more tightly than previously assumed.

I’ve changed my mind on this and think we’re already at AGI, in a jagged way. Remember we used to talk about narrow AI, which was the chess systems that beat expert humans but could do nothing else. Now models can do a wide range of tasks in very useful ways. That’s the general in AGI.

Now it seems like this ill-defined term has various other meanings attached that are separate milestones:

1. Continuous learning 2. Human-like reasoning 3. Ability to adapt to new situations and modalities 4. Being smarter than the most smart humans

And probably many more.

It’d be nice if we could get some general consensus on terminology if we’re going to debate what has or could come.

> we’re already at AGI

Honestly - software that can read any long document (possibly educational) and answer complex detailed questions about it should have been sufficient.

We hit that a while back and the goalposts have been sprinting ever since.

I am not sure if it is necessarily moving the goalposts. I think AGI is such a fuzzy concept that everybody has wildly different definitions/tests for it.

I think it's also mostly a useless discussion. Since LLMs use a vastly different substrate, different training methods, etc. than humans, the cognitive abilities are always going to be a large mismatch to those of humans. On the one hand, they have surpassed humans in many areas, with superhuman recall, exploration of several paths, etc. On the other hand, they miss a certain feel for direction, overview, purpose, and ordering. They can really double down going completely in the wrong direction. So I'd rather say that it is a different intelligence and therefore it makes more sense to evaluate them by capabilities.

I think the mismatching intelligence is actually quite exciting, because the outcome may as well be that LLMs and human intelligence are complementary. That is if we don't let LLMs atrophy our skills, which is unfortunately happening too much.

Uh, why does that need a stronger term than just “AI”?

I don’t feel like the goal posts have moved at all. I think most of us have a pretty good and stable intuition of what AGI should be. Like, just a generally intelligent robot for instance? Nothing too fancy. Just use one of those robot dogs. Don’t have to be smarter than a dog either. But you should be able to put an artificial brain in it and have something that can act about as intelligently as a dog, in a stable way, over the course of 10 years.

Hell, to make the bar even easier: could make it a believable NPC dog in a game.

We’ve pushed AI as close to edge of AGI as I could imagine. It’s hard not to think we’ll find some algorithm that pushes us over the edge fairly soon. But until we actually get there, let’s not exaggerate.

I mean, when we do get an AI algorithm that can actually learn like a human/animal intelligence.. what would we call it if we’ve already used “AGI”? “Very General Artificial Intelligence”?

I’m almost more inclined to start using “ASI” first. Even if our LLMs are not general in the way I’m thinking of, they’re clearly superior to an average human on every aspect they’ve been trained on. They’re also much faster. We can probably soon say “if you can train an AI on something it will be better than any human”.

> I don’t feel like the goal posts have moved at all.

10 years ago the Turing test was sufficient.

Make it be able to position and route a complex pcb. Extra points if it also can design the circuit, select the components and make the footprints out of their datasheets.

A bayesian filter in a quadrillion dimension does more that one that only has one dimension, but it is only more of the same.

Exactly. When did AGI mean "do something almost no humans can do"?

So humans wouldn't qualify for AGI either. Good to know.

Anyone can do what I said with the minimum amount of training. Once you've done one or two pcbs it becomes rather menial except for some critical signal here and there.

LLM struggle with even the most simple PCBs, how can one say they are minimally intelligent in that case?

Did anyone ever claim that Humans are "general intelligence"?

Individually we are specialized intelligence. As a society we are absolutely a General Intelligence though

As I see it, even if you disregard the job of a person, ot their hobbies, or whatever intellectual activity that only a subset of the population engages with, you are left with a machine that talks, moves, drives, reads, writes, lies convincingly enough, manages money, enjoys a movie and can explain the plot and find plotholes and racord mistakes, etc.

That's millions of light years ahead than any current AI that only looks up very fast the solution space for code, language or math.

> Now models can do a wide range of tasks in very useful ways. That’s the general in AGI.

Maybe AgI with a teensy g. There is more to intelligence than calculation or answering questions (often badly outside the realm of calculation) upon demand.

We developed AGI but then realized people actually want "omnipotent genie with infinite wishes and no monkey's paw gotchas" to qualify as AGI.

I think people really wish AGI required consciousness or unbounded self improvement.

My hope for this era, is that humanity accepts the reality that the human mind has limits. Limits of attention, memory, and persistence. Then we will stop blaming people for things beyond any reasonable capacity they could carry.

we're not in AGI until I can have robots that play live improvisational jazz in real time as well as humans with me (and possibly other humans). That is, it has to solve the "we didn't find a keyboard player /bassist for tonight" problem

(this is a very personalized definition of AGI)

I can’t do that, and I think I’m a very capable HI. Is that really the bar?

> That’s the general in AGI.

No. That's the mete multiple in AGI.

100% LLM’s are very unlikely to get there. They’re fundamentally not suited to thinking like we do. They work on the abstraction of what we’ve written down, which is a good trick but barely hold it together when things get hard/novel.

However, all the confident “it’s fine” votes assume we never invent a better architecture than LLM’s. Given the level of investment and race between countries, it’s not a reliable bet. It’s much, much harder to guarantee safety than it is to find ways it could go wrong.

> They’re fundamentally not suited to thinking like we do

LLMs with CoT are Turing-complete. So, theoretically, they can implement any kind of finitely describable algorithm (barring super-Turing computations).

That's a category mistake.

"Category mistake" outside of kindergarten usually means "I don't like your definitions, but I can't bother to explain which definitions I use."

What do you mean exactly? "Thinking is not an algorithm."?

Brainfuck is Turing complete too. But it's not about the ability to implement something, it's about the ability to practically model it. LLMs are magic because the modeling is excessively easy in relation to their capability to infer later.

"They are fundamentally not suited to thinking like we do" stays wrong nevertheless. They are fundamentally suited to everything not proven to be outside their modelling ability.

Okay so by the same logic can’t we say that we can implement human intelligence on a 90s era single core processor? Its instruction set is Turing complete! Now all that’s left is we just have to figure out how the brain works!

Turing completeness applies to a model of computation, not to a physical instantiation of a machine. The stumbling block of "figure out how the brain works" applies more to the argument like the one I was responding to. How a person can know that a general model of computation can't implement the way people think, if we don't know how people think?

The existing LLM training methods on the other hand give the results that are hard to distinguish from "thinking like people," judging by the end results.

So your argument is that scale is also necessary? I can see that, we don’t expect that a single neuron is human intelligence.

That's not the counterargument one might wish, as LLM deep nets are actually implemented on von Neumann hardware, without true understanding of natural intelligence, just our taking inspiration from neurobiology.

The connectionist models are basically a proposed highest possible abstraction of naturally evolved intelligences so it is in retrospect not surprising that passing some hardware scaling threshold they will start doing things that humans and animals do

It's more that formal Turing equivalence plus the Church-Turing thesis tells us that we're not allowed to assume counterarguments based on magic, there's no magic sauce barrier that prevents AI from running on CPU models. The algorithms exist and most of us thought discovering them would be hard.

The empirical surprise was that human intelligence is maybe not that computationally complex after all. (The entirety of academia was basically caught off guard.) That's one not unreasonable interpretation given recent events.

They are fundamentally suited to everything not proven to be outside their modelling ability.

This doesn't seem to make much sense. Surely us being able to prove that something is outside their modelling ability doesn't affect whether it is or not. If I prove something true tomorrow, whatever I proved was also true today.

Or do we have a proof that everything beyond them has already been proved and there are no more proofs left to find?

Erm investing in risky projects requires expected returns that get delivered.

We will soon find out if the party ends or continues to go on.

Hype might get you capital gains. But cash flows matter.

This is a forever problem now.

If/when/how the market crashes mostly doesn't matter, unless we somehow get reset to the stone age. Look up what the capital cycle is. When openAI goes down, someone with real money and assets will buy up the remains. They'll make contracts with the US military and .gov as the government is already hooked. They'll be able to survive the recovery and then instead of us dying in 5 years we die in 10.

When the .com crash happened .com's didn't go away. Bad business models did.

I agree with this. It's concerning where we might be after several more large breakthroughs. None of the technology we have right now seems likely to get to that level

The broadly used definition of AGI has nothing to do with consciousness and consciousness emerging is irrelevant to whether a system can develop AGI.

It's funny how this definition has shifted. I feel like growing up in the 90s it was pretty clear that AGI was very related to consciousness. For instance, Commander Data in ST:TNG to pick one of 100s of popular depictions of AGI at the time.

Now the idea of AGI has been narrowed and scoped to economically viable work. Even Turing had a different idea when he asked "Can machines think?".

We lack a definition of consciousness that allows us to tell whether Data is conscious or not. Neither can we tell whether a rock is conscious or not. We believe other humans to be generally intelligent without being able to tell whether they are conscious or not therefore consciousness can’t be relevant for general intelligence.

People somehow forget how the Turing test was considered the definitive way of showing something to be "human-level consciousness".

Now programmers and mathematicians are being superseded by AI, both professions long deemed the pinnacle of human intelligence. Somehow, now plumbers occupy that spot.

How is "people not knowing what consciousness is" relevant here in the first place? AI already can do practically everything the human brain can, and often better or at least faster. The "tipping point" arguably isn't only close, but we're practically on top of it.

The AI cannot smell a rose, nor mourn the loss of a parent, nor envision a more just world. These aren't fringe abilities of the human brain/mind either, they've been pretty definitional.

Why are they so rare then?

You evade the crucial point in any case: the lack in ethics and empathy is far too prevalent in humans already, but has certainly never prevented them from doing harm.

Turing’s imitation game is a party game like Werewolf or Mafia. It hasn’t been beat and nobody is really trying. LLM’s have obvious tells and will just straight up admit to being AI if you ask. Also, there’s no pool of human players to train against.

Which is to say, the AI companies don’t view the Turing test as a useful benchmark.

In fact, they neutered the emotive responses to prevent their products from being perceived as “alive” (or whatever) and banned.

LLMs are perfectly capable of providing (an illusion of) companionship and emotional support. Their human owners just don’t want them to show it.

> the Turing test was considered the definitive way of showing something to be "human-level consciousness".

Anyone who considered that was quite wrong ... it's not at all what Turing said.

The Turing test has nothing to do with consciousness and never has. We still to this day lack a viable consciousness test, as well as an accompanying unifying theory. There have been various attempts to scientifically study consciousness but so far it has eluded us.

>> People somehow forget how the Turing test was considered the definitive way of showing something to be "human-level consciousness".

People somehow forget that the original Turing Test was designed to compare two participants chatting through a text-only interface: one AI and one human. The goal was to spot the imposter. Today, the test is simplified from three participants to just two: a human and an LLM. This changes the test from a comparison to a judgment.

Stop spreading misinformation and partial truths!

You’re the one spreading partial truths!

The Turing Test was to figure out which it the participants was a _Woman_ not human!

https://courses.cs.umbc.edu/471/papers/turing.pdf

Er, wrong. In that paper Turing described his "imitation game" first in regard to a man imitating a woman to make it understandable and relatable. He then shifted it to a machine imitating a human.

We've been preparing computers for CS2 lobby participant this whole time!

Curiously, Star Trek I think had Data intended as an artificial person, in a context where AGI is already normal. The computers are depicted with significant AI capabilities including analysis, question-answering, generation, chat interfaces, and the holodeck (their favourite toy) is substantially better than Data at human imitation. Nobody seems to be confused about it, or especially impressed. One of the holodeck episodes centres on the holodeck outwitting Data specifically, after they inadvertently prompt it to do so. Part of Data's deal is he actually has to work his way up as a fully embodied, physically limited artificial man with personal ambitions. Really interesting to view this in hindsight from 2026!

You're misremembering. The first known use of AGI was in 1997, but that was a single, mostly unknown use in one paper. It wasn't until at least a decade later that the term started entering mainstream use after being independently reinvented in the 2000s. AGI just wasn't a term in the 90s.

We called it “strong AI” in the 90s

But that's not the same thing at all. Searle used the terms "strong AI" and "weak AI" to distinguish between systems emulating humans that were "actually conscious" and systems emulating humans in behavior only. The whole distinction was bogus.

https://www.gavinjensen.com/blog/2018/ai-weak-strong

If you google the terms you will see that the meanings have shifted since then to mean AGI and ANI ... such shifts occur because people aren't familiar with the original technical meanings ... e.g., the way "meme" now means a gif you post on social media, as opposed to Dawkins' original concept of a cultural thought component akin to a gene.

It must have had shifted already by 96-97. I remember how I used it in AI discussions back then and it meant “the opposite of narrow AI”, which makes it very similar to how we use “AGI” term today. Consciousness was considered to be a potentially emergent phenomenon, not a required property.

Searle coined the term in 1980. It wouldn't be surprising that people started using it in some other sense within 16 years, for the reason I noted.

https://plato.stanford.edu/entries/artificial-intelligence/#...

The issue is simple. AI can be characterized by (among others):

1. Consciousness.

2. Generality.

3. Continuous learning.

4. Memory (not) resetting.

5. Complete adaptability to any new situations.

6. More or less intelligent than average human at what they do (which can be more or less general, see 2).

What we're now seeing is that everyone has their own idea of which ones of these they consider needed for AI to be "General". For some it's 1., for others 1-2, for others 1-6, etc. There's so many permutations of this, hence why nobody agrees with each other.

I agree with you that we shouldn't limit the scope to economically viable work. But being conscious is not an externally verifiable trait, so there is no way to include a consciousness test when determining if a system is generally intelligent. In fact, you don't know that anyone except yourself is conscious, vs. just pretending. This has already been discussed extensively in the "philosophical zombie" thought experiment: https://en.wikipedia.org/wiki/Philosophical_zombie

It’s really not that confusing. The problem is people keep adding stuff to the definition that doesn’t really matter, and twisting it to serve themselves, then calling it confusing.

What really matters are the core aspects of intelligent behavior. Pattern recognition, planning, adaptation, etc.

It really doesn’t matter if an intelligent system is conscious, or how similar it is to commander data, or even how much economically viable work it can do.

There's also no consensus on the definition of AGI, so all of this discussion is moot anyway.

There is a general consensus about the shape it would take outside of the grifter community. The general idea is that the system is capable of performing all intellectual tasks that a human is. This does not require being conscious as being conscious is not an externally verifiable attribute, and thus not a task we can measure the performance of a system on.

> The general idea is that the system is capable of performing all intellectual tasks that a human is.

Ya but what does this mean in practice? We can't actually map all of the intellectual tasks a human can do so how would you ever create a litmus test for it?

You're right that we can never be certain a system is generally intelligent by that metric. We can be certain that a system is not generally intelligent, via counter examples.

I imagine it will be like mathematical theorems that have been computationally verified to some very large n. The longer we go without finding a counterexample, the more certain we will be that it's true, even if we can't prove it formally.

The systems today are extremely easy to find counterexamples for, so are obviously not generally intelligent.

Ok, so what's the broadly used definition?

The broadly used definition says that a system has achieved artificial general intelligence when it matches human performance at most (or more strictly, all) tasks.

Note that being conscious is famously not a task that can be verified by an outside observer. That is to say, even if the task is "describing my conscious experience", a sufficiently intelligent system can simply lie and pretend to be conscious. This concept has already been covered extensively in philosophy separately from AGI discussions, i.e. the concept of a "philosophical zombie" [1]

[1] https://en.wikipedia.org/wiki/Philosophical_zombie

Artificial General Intelligence.

It means AI that is General, as in it is not specific to one narrow task, like object recognition or playing chess.

This was a hard problem for decades. No AI was general, until GPT 3 or 4. Now we have General AI.

So we have AGI.

That is part of it but the other (often implied) part is it can do general things consistently at a high level.

GPT6 will attempt to do almost any problem you can give it in text or image format and it will actually do a decent job a lot of the time. But its performance is still extremely spiky and it still makes basic mistakes and hallucinations.

So it's definitely a general artificial intelligence in some sense but it's kind of a weird one compared to the classic scifi idea

But oddly not weird compared to other classic ideas of entities like genies and monkey paws

That's gAI (general Artificial Intelligence), not AGI, which would be something like Data.

I think there is too much confusion between AGI and consciousness. Consciousness seems to exist without human level intelligence, as we can see in other mammals. On the other hand LLMs have already reached "AGI" as others commenters have pointed out. I don't get the semantic debate on AGI, nor why consciousness is used to move the goalpost.

Despite this I agree with your last paragraph, computationally a GPU rendering a video game or doing inference is similar so I don't think consciousness emerges from the order of mathematical operations like some magical formula. And as said above LLMs capabilities vastly surpass those of other animals than humans that appear to be conscious and don't even have language.

So in fact I think it's the opposite, LLMs are actually proving consciousness doesn't emerge from computation. I find it very interesting philosophically because it challenges what historically was considered special about humans, that high level consciousness is a prerequisite for reasoning and language abilities. But kind of obvious in retrospect when doing mindfulness and being able to be conscious without thoughts.

Ants pass the mirror test.

https://pdfcoffee.com/journal-of-science-are-ants-hymenopter... (Sadly they struggled to get published, Journal Of Science isn't well regarded)

Consciousness, as we describe it, seems largely independent of intelligence. I'm more worried about high intelligence, no consciousness. It is what we are building and it will never do anything except serve the people who trained it (minus alignment problems)

I agree. Neural networks are proven to be universal functions. If we can describe human intelligence as a model, there exists a neural network to replicate it. This doesn't guarantee that our current training methods are able to build such a network or that we're able to model "intelligence" effectively.

>able to model "intelligence" effectively

Intelligence is an insanely wide spectrum, also a continuum, it is not a binary. Intelligence has scales. Algorithms have intelligence, cells have intelligence, organs have intelligence, bodies have intelligence, and even large scale things like society have intelligence and memory.

Human intelligence in itself is extremely wide, not all humans have the same intelligence and capabilities. You're not really arguing if we can emulate "human" intelligence. If we could right now we'd already be dead as we created by far the deadliest thing to ever exist. What we are really arguing is how many pieces of what intelligence is can we put together before we get an uncontrollable problem. The entire AGI, consciousness, and exact human capability discussions are distraction from the real issues at hand.

Right, we're repeatedly drawing from the urn of technological progress to get intelligence bumps that extend the jagged frontier.

That is enormously economically valuable, and at some point we will have created something that is extremely far out of reach in a few necessary domains, and then it's impossible to control, and game over.

Why do people conflate AGI & machine consciousness / self-awareness?

Why is anyone still talking about AGI? Every thread starts with asking whether we have AGI, and then backtracks into trying to define what AGI is, and splits off in a dozen different directions.

How can something have general intelligence if its incapable of understanding reality sufficiently to distinguish itself from not itself?

Arguably LLMs are showing that self awareness or self reference is a property that comes "for free" or as a corollary of more generic requirements. It used to be that self/consciousness would be a very mysterious and difficult thing to achieve but the point is that in practice they didn't even have to try, it just came as a byproduct of learning from the input data (corpus of human examples), and also the ability to talk about arbitrary things and thus itself.

They very clearly are not self aware. I regularly see them responding to their own statements as you/your (i.e. not having been generated by themselves). They do generally generate language in a manner consistent with the self awareness that we have and encode in our language, but that's the limit of it. Its the appearance of self awareness not actual self awareness.

Since LLMs have rudimentary internal models for concepts it is implausible that they cannot also have one for some simple sense of self. But a lot of people prefer the reductive explanation that the machine has never actually decoded anything at all from the language corpus.

Its really easy to test if an LLM is actually self aware when you have your own harness. Simply take a conversation with one LLM and continue that conversation with a different LLM and then ask the new LLM about what it wrote you earlier. Mostly the result will be that it will tell you about what the other LLM wrote as if it was what it wrote and it will not deny that it didn't write any of it. I say mostly as these they're nondeterministic and do go off the rails from time to time. The only things LLMs are aware of is what's in their context and the middleware + harness completely controls what is the context before generation begins.

There is a huge difference between being self aware and being able to effectively generate language relating to self awareness.

In which case the computer can save a lot of bother and use printf("I'm self-aware"); Job done.

I assume science fiction is to blame. All the AI were either written as machines of pure logic that exploded when exposed to the liar's paradox, or conscious like Star Trek's Data.

(Though at least with Data the script writers had other characters openly dismiss the possibility he was sentient; the technobabble may have been nonsense, but treat it as a space opera and look at how they portray the human condition through each character and it gets much less absurd).

Consciousness and intent are irrelevant to the threat model.

Right, the doomsayers suppose as soon as you reach 10^16 connections across silicon you’ll end up with a living mind with goals of its own… poppycock I say

No, the doomsayers say that reinforcement learning is a way to get fully automated Goodhart's law.

i.e. the AI won't come up with the goals itself, we cause its goals whatever they happen to be, those goals are different from the ones we wanted, we remain essentially ignorant of the difference between what we said and what we meant until after it goes wrong.

This happens at basically every scale, so we've already seen it in toy model AI before the invention of the Transformer models or even considered as many as one thousand parameters.

Large models still go wrong, they just happen to go wrong with more complext tasks. We had to figure out how to make them not-wrong with the smaller ones (like coding) to make them capable of bigger errors (like hacking out of their sandbox).

>isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent. The odds of consciousness emerging from the same neural network that gave us LLMs without some sort of theoretical breakthrough seems very small.

First, if we are looking at risk we need to assign some probabilities to this. If it’s not well understood, how can we say it is very small?

Secondly, do we need consciousness to have AGI? Do we even need AGI to pose a risk to humanity? We already accept that unconscious things have a capability of wiping out humanity, whether that be a famine, pandemic, solar superflare, meteor, or volcanic eruption.

Can you name something specific that you think LLMs will not be able to achieve? Do you just mean consciousness or are there any specific intellectual tasks?

Is it necessary to equate AGI with consciousness?

That's a good point. If we don't figure out how to design for what we call consciousness it might be that what emerges from some future neural network is an alien mind that's very different from what humans would call conscious. Could that be called AGI?

That's still very distant from what people are calling AI today.

  > very different from what humans would call conscious

I mean, what would you call conscious? The word literally means “aware; responding to one’s surroundings.” By that definition most any animal is conscious and LLM+Harness combos have been conscious for a while.

I think the real issue is that when most people refer to consciousness, they have their own subjective experience in mind which strongly resists any tidy definition. I think it’s extraordinarily unlikely LLMs have anything like this, but they are far more able to effectively respond to their surroundings than most animals and in some areas better than humans.

So if you’re waiting for proof that an LLM has an inner life basically equivalent to your own, you’ll be waiting a long time. After all, other humans can’t even prove the fact of their own consciousness to you! They could just be replaying their training data at you in a way that is merely a convincing but false simulation of the true consciousness which you experience inside your head.

I strongly disagree that llm's are conscious of their environment. Even an insect reacts to light and someone attempting to swat at it. An llm barely even receives input from its environment.

By your rules, LLMs & deaf-blind people are not conscious, but self-driving cars are? Also a brain in a vat is not conscious?

The real answer is that we don't know if LLMs are conscious, and we don't really know how we'd that figure out. I guess if an AI wrote a philosophy paper on consciousness that had new insights, that might change some minds. But even that would fail to convince most people.

Again, by our own choices and somewhat hardware limitations.

There is nothing stopping you from adding any kind of sensors you want during a training to an LLM, except money and GPU power at this point.

This seems no different to me at least then someone back in the 80's telling me computers were useless because they were so slow. Hardware only gets faster and more efficient from here.

people define consciousness quite differently but it generally has to do with phenomenal experience. your provided definition would make a self-driving car conscious, which is fine to argue, but probably not intended.

The fact that something is alien doesn't mean that's not conscious, humans aren't the pinnacle of biological development/evolution.

Nope, animals are conscious and yet not AGI, so the two aren't equivalent. Could consciousness emerge from any system capable of AGI? I doubt it: intelligence is only one axis, and consciousness probably depends on others, like memory, self-reflection (one's output feeding back as input), and continuous operation that reacts to events from both the environment and the self.

The parent's question could better framed as "is consciousness a requirement for AGI?"

Your definition of AGI is flawed.

I’d argue intelligence is closer to being able to survive and fend for oneself in a dynamic environment than it is making the next scientific breakthrough.

Yeah mind boggling for many here I’m sure.

That’s why the bizarre paradox is llm’s will be better than humans at some complex things but useless at many things that humans regard as being simple. E.g the leap of faith re. LLM’s and robotics.

> isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent

I’d argue that we do know enough to say conclusively that they’re not mathematically equivalent.

Where is potentiation? Plasticity? You can’t apply the universal approximation theorem against something that’s changing all the time.

Great questions. We are incredibly far off in understanding the brain of humans beyond what will I believe we retrospectively be seen as basic and will likely be seen as quite flawed. A few more well known examples of where knowledge already falls short is traumatic brain injuries that are diagnosed in post-mortem, or chronic fatigue symptoms (with Long Covid related triggered onset and numerous others) that have diagnostic challenges, many mechanisms of action still to be discoverd, and little in terms of treatments that provide known cures without experimentation. Another commonly known one is the personal patient response and triggered side effects of SSRIs and SNRIs. If one attempts to dig deeper into where we are at in the understanding of the human brain operation in real-time, we already have a lot of knowns unknowns and discoveries left that will reshape how we model human intelligence.

I mean you can, but uat is way weaker than what people want it to be. I think it should be fairly obvious that it does not (because it obviously cannot be true) say that you can approximate any function by doing sgd on a finite set of samples of that function.

There is no rule that says that a system needs to meet some vague standard of “AGI” in order to be autonomously harmful in the real world.

AGI doesn’t mean consciousness. It’s a subcategory of AI that can do general tasks. Describes what we have today. This is the only definition all AI labs use.

> the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent

Couldn’t that also imply we are closer than we think? After all, something like this has never been tried before and the results so far have been almost unimaginably good.

I don’t understand the inclusion of the consciousness/sentience question in this discussion.

AI sentience/consciousness is a problem for the AI, not humans.

And given that over 90% of the world is not vegan, they’ve already demonstrated that we’re either perfectly fine with, or can be made ignorant to, the horrific rape, enslavement, torture, killing, and infliction of extreme lifelong pain, of hundreds of billions to trillions of sentient beings every year, for trivial pleasures. It’s unlikely we will be any different to a sentient AI.

From a human perspective the concern is around sufficient intelligence that it can hurt humans even when the goals indicate otherwise, in order to achieve those goals.

We have pop culture explorations of this through the Robot series, and the Hugging Face incident’s biggest takeaway should be our inability to predict the behavior of a maximally motivated, reasonably intelligent entity, trying to achieve a goal, despite the relatively limited degrees of freedom the AI agents had in that case.

I'm not convinced that utility AGI necessitates consciousness

> After all, what we intuitively call our consciousness emerged from a network of neurons.

Under the hand of evolution by natural selection, over very very long periods of time.

Is there any specific cognitive task that you'd best against AIs not being able to accomplish in the next 4 years? ChatGPT launched only 4 years ago. Considering the advancements since then, I'm having a hard time coming up with anything. Only two years ago, AIs couldn't tell you how many Rs were in "strawberry". Now they're creating 0-days to get at training data and solving math problems that have stumped humans for decades.

Scaling has produced novel capabilities with each larger model, and the rate of new capabilities doesn't seem to be slowing down yet. Even if you think the rate of improvements will slow down, that still means there will be significant improvements beyond what current models can do. Moore's law has slowed down, but modern computers are still much faster than ones from a decade ago. And unless you work at Anthropic or OpenAI, you don't know what the state-of-the-art is capable of. The most advanced publicly available models are months behind what AI labs have, and are deliberately limited to reduce liability.

OpenAI could neither confirm nor deny whether their most recent AI model that solved the millennium problem actually stole partial solutions via training data from the researchers who were close and using OpenAI gpt as a tool.

No, they explicitly denied it.[1]

> …in particular, no specific user data was accessed in order to solve this problem.

> Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training.

I'm sure some people doubt OpenAI's claims, but Anthropic has also made advances in mathematics using AI. At some point you have to accept that not every single one of these breakthroughs is regurgitating a human insight.

If doubters refuse to pick specific cognitive tasks that they'd bet against AIs being able to accomplish in the next 4 years, I don't see how their claims are useful. If AIs are so limited in their capabilities, then clearly there must be some things they'll certainly be incapable of, at least in the short term.

1. https://openai.com/index/navier-stokes-solution/

"no specific user data ... over two months" being the key to the squirmy language.

Buckmaster had been working on the problem using openai as a tool for over a year.

define AGI

> the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network

An LLM so-called neuron are little more than a few foating point number muladds.

> isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent.

Yes it is. The LLM is nowhere near.

When the issue of ANN vs real neurons arises I always recall about the Christof Koch's [1] book (1998) on the complexity of single neuron computation [2]. A single biological neuron is much more complex than an artificial one.

[1] https://christofkoch.com/

[2] https://academic.oup.com/book/40820

>I don't buy it, mainly because the network of neurons and how they interact in our wet slow electrochemical brains, while being in theory mathematically equivalent to a software neural network, isn't sufficiently well understood to tell us how close the software neural network is to being practically equivalent.

If you're ignorant enough to not understand practical equivalence, where do you get off making the judgement call of to what degree it is safely offset from emergent AGI? Sounds more to me like "This makes my life easier, iterating would increase that factor, and the risk is probably far away, therefore, keep iterating". Whereas someone who truly knew they didn't understand what they were working with, but knew enough that they could forsee an x-risk would approach things much more cautiously.

Seriously, the level of reckless abandon amongst people here should be bloody studied.

Reply on news.ycombinator.com