Hacker News Reader

By this definition, even most humans would not qualify as having AGI though.

However most humans can do at least some of the things given they spend the required effort.

Some problems presented needs a very large context and some are not much solvable (e.g. trading) since market responds to traders' actions, as well, making it effectively an oracle problem (of computation).

On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them. However, brains in nature never stops. Wonder, daydream, sleep, self-evolve, clean up and eliminate memories and views and much more.

But this is assuming the model is the entire story. The original comment you were replying to pointed out that the harness is just as important.

The hardware of human intelligence is not a singular thing that is uniform throughout. You cannot take the prefrontal cortex white matter out of someone's head and say you are holding a person. Much of the parts of our brains that enable much of our intelligence, is made of different specialized stuff. The visual cortex and sensorimotor regions aren't only there for input and output, they are used by the more thinky parts of the brain to do visualization and spatial reasoning. The cerebellum contains billions of neurons making little oscillator circuits and PID-like self-regulation machines that help make muscles do what they're supposed to, but also provide attention and time perception.

Heck, our brains contain language models, that train themselves up based on a glut of data over a span of about 10 years, and then they become more or less set in stone for the rest of our lives. Of course we can learn languages, but the "Critical Period" is a very real thing that produces a permanent architecture for some grammatical structures, or things like the ability to partition a lexicon by gender for faster lexical access which cannot be learned as an adult if your native language did not have gender.

I'm not trying to make a direct analogy, the point is that the language model doesn't need to be fully "generally intelligent" all on its own for there to exist a general intelligence, because the language model can be part of a generally intelligent system, which can do things like form, recall, and manage memories which are by now a standard feature in basically every chatbot.

That's true, but my general gripe is the static nature of the whole system. Even if humans' neuroplasticity reduces with age, it never goes to zero. I started to learn English at a very early age, yet my ability to communicate with it soared around 20, because I started to use it more and more.

Same with instruments. I started to play instruments at an early age, but started to play guitar around that age. Well, I'm not a virtuoso, but can play and more importantly can improve.

These AI systems we built are static things. We generally try to make them more intelligent by augmenting the context they can see, but the model doesn't evolve in every turn, for example.

Intelligence is a multi-faceted and multi-input construct, that's true, and GPT-6 may do amazing things w.r.t. other models, I didn't try it yet. OTOH, my main call is to remember that these are still static algorithms fed with enormous amount of data. They are more rooted on statistics rather than fixed inputs. In short, they are still fitting to the frame of "advanced search".

In general, I'm not against the tech, but the hype. I have other gripes about how AI is being built, but that's not subject of this comment.

> On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them

The parent commenter noted:

"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"

Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.

However, this doesn't change the fact that you are pumping more and more tokens to a static model's context window, even if you do compaction, the model is not more intelligent than previous turn.

Nature doesn't work that way.

Actually, the world kinda does work like that.

Most intelligence researchers would agree that people seem to have a genetic cap on their intelligence. While someone can underperform their intellectual potential with an upbringing that doesn't adequately enrich their minds, it's near-impossible for humans to become more intelligent through reading, studying, etc.

When humans learn we gain knowledge, not intelligence.

I think the only real difference is that we humans are born lacking a lot of initial knowledge/data which means we have to go through a decade or more of education to reach our potential intelligence. LLMs on the other hand come pre-loaded with that knowledge.

Passed this point, wherever knowledge is passed in as context or stored in the neural net I don't think is that significant personally. I'm of course not suggesting we're exactly the same as LLMs and there is no noteable difference, I just don't think continual learning is as important as some suggest it is – at least assuming a model is deployed with adequate training such that it reaches its potential given it's size + architecture.

I'm guessing their defn of AGI is something like the sum total of all humans' abilities? Still though some of those tasks (e.g. beat an index fund) may very well be impossible, and worse yet a lot of those tasks are not coherently defined.

I haven't met a person who doesn't wonder about things.

They can't, by definition, have the A part btw.

AGI has always been expected to outperform humans or else what is the point of it?

The point is replacing humans, and for that it only has to equal them at lower cost (including factors like not needing sleep and being easily clonable). The outperformance lies in the cost savings, not necessarily in the intelligence.

An AI does not have to be AGI to replace humans, that is whole another topic I think.

Humans are AGI simply because if they want to, they can achieve it (it just takes some effort).

Humans are general intelligence, not artificial general intelligence. :)

Let's start by giving "most humans" the long term safety net (food, lodging) from early age to allow to focus intelligence on high level problems.

Machines aimed at running AGI are provided that, but the money spent there could as well go towards enhancing humans.

To me all this makes the label of AGI completely meaningless.

What AGI has always meant (eg. in 2019) is Artifical General Intelligence.

Artificial -- something made by humans instead of occurring naturally

General -- not confined by specialization or careful limitation

Intelligence -- the capacity to learn, reason, solve problems, think abstractly, and adapt to new situations

Basically, the metric was that any healthy adult human on the planet represents a general intelligence. This has certainly long been reached.

Also some of the stuff you're listing has long been solved as well, such as listing what it knows and what it doesn't know, and what information it would need. Other is just poorly defined: "be able to argue persuasively". AI can certainly write an argument on almost any topic that would pass any University homework in 2019.

Your examples are things that most humans cannot do, or things that AI can already do. For example most humans, even most intelligent humans, could not write a well-received book, run a successful company, or make a popular game. On the other hand, AI can absolutely sort through research, draw conclusions on complex topics, and argue them persuasively. Likewise, I don't know what you mean by a "decent" conversation, but millions of people converse with chatbots daily, so I don't know why you say AI fails to meet that bar.

I fail to see how what you describe is any better than the old autocomplete-on-steroids comparison. Could the human mind learn to spell every word properly? Yes. Do most (or any) do it? No. Does that mean a human can't?

I think what OP was drawing a comparison to is that AI right now could not come up with an award winning novel from the spark of some creative notion and working up from there, as opposed to just mashing together what has already been done and calling it a day.

> AI right now could not come up with an award winning novel from the spark of some creative notion

I agree, but in this field we value evidence. So there needs to be some test of novel-writing abilities.

Once there is, AI companies will be out to score highly on it.

Wait for a resurgence of Philip K Dick-style novels as humans desperately try to write things LLMs cannot.

> Wait for a resurgence of Philip K Dick-style novels as humans desperately try to write things LLMs cannot.

I myself can't wait for Finnegan's Wake 2

It also cannot do tasks it wasn't trained for. It can extend texts, read images and click on a desktop, but only because it's made for that.

I don't think that's strictly true, as I can give it a new gui or tui program it wasn't trained on and it will learn it. Unless you're talking about general abilities like sight, but the same is somewhat true of humans.

If you consider the data on which an LLM was trained on to be points on a very highly multidimensional object, the claim is that the LLM can interpolate a convex hull spanned by those points, therefore recovering a subset of consequences attainable from those points. Obviously this hull includes completely novel points that were not present in the initial data set, so the output of the LLM goes beyond its initial training. And yet, there are clearly points outside a convex hull spanned by any finite number of points, such that we can imagine not all possible outputs are attainable using this method.

The claim is furthermore that truly original thinking, the infamous leaps in understanding and creativity, happen by attaining points outside such a convex hull.

It's hard to rigorously verify or disprove this claim. Hopefully this helps build an intuition of why the claim is not as shallow and obviously wrong as it may seem initially.

I'd call it interpolation on a high dimensional manifold. Convex hull is too simple a shape.

But yes, metaphorically I think that's right.

This makes me realize there is a higher bar we need to achieve with AI still. The ability for the model to evolve through interactions more on a hourly or daily basis. The models are accelerating but inference doesn’t modify the model.

Humans cannot do tasks they are not "trained" for.

speak for yourself

This is false, unless you think the same for humans

> come up with a new company idea, Run that company

So far nobody's even shown an LLM succesfully running a high-traffic vending machine for as much as 30 days at a time.

Most of humans don’t reach any of these levels.

But you have to acknowledge how uneven the playing field is. The AI has read every book that's ever been written, and can spend hours of compute time in a few seconds.

I think if a person had those same advantages (e.g. could spend 5 hours thinking about what to say next) we could all hold outstanding conversations, or if we had read every book ever written I think many of us could write a very popular book, if we could read every singe company's P&L statement in a few seconds we could invest better than an index fund.

What I'm pointing out here is that these models appear to be intelligent when they really are simply unimagineably knowledgeable. When you drop the time-constraints it starts to become more and more apparent that human intelligence scales better with time than AI does (much in the same way AI can burp out tons of code but make your codebase entirely illegible within a matter of months).

Perhaps to simplify: my notion of intelligence is how much can you deduce with a constant set of starting context

> I think if a person had those same advantages (e.g. could spend 5 hours thinking about what to say next) we could all hold outstanding conversations, or if we had read every book ever written I think many of us could write a very popular book, if we could read every singe company's P&L statement in a few seconds we could invest better than an index fund

I’m not so sure of that - to get average outcomes in these fields it’s a matter of time, to get above average or extraordinary, you need talent/intelligence/taste.

And the bar the parent set is at extraordinary.

So now the bar is not only to be at the level of a human, but to achieve it without the natural advantages of being a machine, while still having the disadvantages (lack of embodiment, etc.).

If you went back to when I started working on AI stuff 20+ years ago and described the capabilities of GPT-3 to people, the overwhelming majority would say it's a form of AGI. It's incredible how fast and far the bar moves.

I would be willing to bet that any human for which we spend $100billion - $3 trillion (depending if you want to count single corporations or global totals) on in an attempt to make them as capable as possible would be able to reach all of those levels.

I think I view humanity fairly positively, but I admit I would gladly take the other side of that bet

A quick search suggests that the most expensive education in the world is something like $100k.

So like you spend a million times more than that and you still think you're not going to see some results?

except you are missing the one versus many argument here. sure we could make one human much smarter, could we make endless copies with the same intelligence? no

For $100 billion we could pay ivy-league level tuition for a million people. You don’t think investing that much in education would yield some good research or companies?

That's not what the parent comment said though

> any human for which we spend $100billion - $3 trillion...would be able to reach all of those levels

Some results, sure.

But that list is extremely ambitious. Write a best seller, make a popular game, come up with a truly novel theory, consistently outtrade index funds.

That's top 0.001% human stuff, I don't think you can take just any person and get there through education alone, it takes extreme talent and dedication. There's also diminishing returns when spending on education, it doesn't just improve linearly.

> A quick search suggests that the most expensive education in the world is something like $100k.

I have a kid in an American university right now, and a quick search of my bank account statements confirms that there are far more expensive educations in the world.

Wow, an increasing number of US universities are going over $100k per year. That's a crazy amount. That's more than enough to hire an entire person.

The marginal return on education spending decreases fairly quickly, but obviously becomes zero at the point by which there are not enough hours in the day/year/decade to cover every single topic that humans know about - no matter the talent or resources available to the student.

Are you imagining artificial augmentation somehow? Purely through tutors or training programs we seem pretty limited. Otherwise billionaires (or even multimillionaires) could have far more consistently successful kids.

Don't the children of the wealthy famously have a tendency to be successful? Or have I badly misinterpreted the last several thousand years of human history.

To really drill down into that I would think you would need to figure out how many millionair children get tutored vs how many get spoiled.

Successful, sure.

Gold medal Olympic athletes who are also brain surgeons AND astronauts, no.

You'd get rapidly diminishing to zero returns after the cost of university a few times over. Every dollar past that would produce no performance gain beyond that.

>be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)

i'm not sure what makes you think AI cannot do this already. in my experience, this sort of deep research is something AI is quite good at.

example i just tested: https://chatgpt.com/share/6a9a20e3-1d20-83ea-a125-31aa240c74...

I understand that what it came up with sounds impressive (especially since I know 0 about Myanmar), but on the topics I do know about its analysis routinely have very fundamental problems (even this Myanmar analysis has % that add up to > 100). There's a chance it's just parroting the majority opinion on Myanmar, or making stuff up (and perhaps you could ask it to write a strongly worded opinion in the other direction that would sound equally plausible).

For example I asked it to do a full analysis on the AI bubble, and a full analysis on the risks of Glyphosate, and it came up with a lot of things that sounded credible, but within a few minutes of questing was admitting it hadn't even really checked for internal consistency in its positions, and even doing a 180. It certainly was much faster at gathering sources and reading but it fundamentally doesn't seem very effective at creating a consistent worldview.

And of course the funny thing is it says it did a 180 on one of these topics, great, except whatever it concluded will be discarded because it cannot learn. It's just bonkers to me pretend this is AGI, it probably couldn't even hold its own in this very discussion.

I claim that the percentages add up to more than 100% because the first described case overlaps with the second.

Regarding this specific case, the analysis is sound. It is the majority opinion on Myanmar, but that majority opinion is held for good reason, e.g. China's desire to keep Myanmar together, complicated ethnic boundaries, desires of current EAOs.

The % responses adding up to over 100 makes sense, because the 25% outcome (formal secession attempt) and the 10% outcome (completed secession) could both occur, so it's implying that if secession is formally attempted, there's a 40% chance it will be completed. The numbers are broken down more clearly at the end - it is a bit confusing at first glance though.

You want a computer program to be able to take a single phrase and execute decade long journies?

Who will be responsible for the outputs and side effects of such a closed loop system?

Half of those the agent fleet systems can do right now.

These are things it cant do and will not be able to do without human labor and long running human vision:

https://rcsnyder.github.io/open-frontier-curriculum/05-front...

> You want a computer program to be able to take a single phrase and execute decade long journies?

In my opinion that is exactly the point missing from AGI: the fact that you still need to prompt it. As long as you have to ask for something, is not general.

It sounds like what you're saying is that AGI should have some sort of free will. I'm not sure why you would add that as a requirement. Could you expand?

I think they merely want something with a functioning long term memory. Something that can exhibit growth past the first 5 to 10 human-equivalent hours working on something.

Current LLMs are worse than most dementia cases, reaching "peak domain skill" pretty much immediately.

Autonomy is not the same as general intelligence. We already have all kinds of fully autonomous technologies that are nowhere close to generally intelligent. Plus, at a certain level of abstraction, human beings also need to be “prompted” to some extent by stimuli. And this is the funny thing about general intelligence as a concept: most of the definitions that come close to internal coherence rely on references to human intelligence, a concept we feel like we understand because we all live it all the time, but whose actual nature and structure is extremely slippery.

Yes everything needs an initial condition.

You could have the smartest human political operator, but if he has no context, no motivation, not much is going to happen.

I don't get it, human employees frequently need to ask for directions too?

They often act on their own, too, and get things wrong a lot. The reason it works is because of all the systems of laws and institutions we have built around humans, not so much because human minds are special.

You (as a human) don't need to prompt AI. Why do you think it is required?

>You want a computer program to be able to take a single phrase and execute decade long journies? > Who will be responsible for the outputs and side effects of such a closed loop system?

Itself. That's the point. We can do it. Until it can met that bar, it ain't AGI. That's always been the bar.

"Being a person in all of its aspects" isn't the same as "generally intelligent". The latter is at best subset of the former, and it's also easy to imagine a system that is more generally intelligent than humans, without being a person. See also discussions of the personhood of various animals who are less intelligent than average humans.

Have whatever definition you want, but I'm not moving my goalposts. They started out that way and that's where they're going to stay.

That's fine, but they've also always been objectively wrong :-)

That's also fine.

Most of the list reads more like ASI than AGI.

- Amazon is full of AI books, and they're clearly making money. AI has won multiple literary and artist awards.

- Okay, it's not a "new company" idea, but VendingBench is all about ability to run a company

- Plenty of people disagree with you on conversational quality; see "AI Boyfriends" etc.. (and it's not hard to find people who consider it uniquely valuable for discussing mental health)

- "come up with its own ideas or theories that nobody else has presented" C'mon, seriously? Solving a half-dozen hard open math problems wasn't enough there? What the heck counts as "it's own ideas or theories" at this point?

- plenty of evidence that custom models are starting to do well on the stock market, although I'll admit we're a year or so from any solid proof, since you need a track record to really make the claim

- LLMs have been capable of being a GM for a TTRPG for over a year (although like humans, they make mistakes)

- Okay, conceded, but humans tend to take years and large teams to make a game. Even if the capability existed today, it would take a while to actually build, test, market, etc.. - all made much more complicated by gamers being largely opposed to AI art styles, etc..

- "be able to sort through research and come to conclusions on complex geopolitical/sociological topics" - uh... did you mean to say something else, because "come to conclusions" is... like, LLM 101?

- Hahaha, have you met humans? We definitely cannot do that.

- Uh... thinking about it's own thinking is trivial. Most LLMs these days are built using LLMs, so uh, self-optimization seems nailed, too? We just don't let them do it unsupervised.

- LLMs fucking love to wonder about things

- "observe contradictions and ironies in the social-consciousness" really seriously have you actually used an LLM recently? I think you would find it remarkably enlightening.

I would bet that llms have talked plenty of people both into and out of suicide at this point. That nitpick aside, I think that's an excellent list. Especially being able to articulate what it does and doesn't know, or how confident it is. That's something that naively sounds pretty simple, but clearly isn't. And it's something humans aren't great at either (see: Dunning-Kruger), but so far LLMs don't even really have the capability to attempt it.

Tell a funny joke.

The bottomless pit supervisor was quite funny.

I don't agree with you how measure how intelligence is, because why ??? those list is not easy even for expert human to do it either

or are you miss the part "general intelligence" is ????

That’s ASI, not AGI.

When taken together, that is ASI.

That would be Artificial Super Intelligence

This is more like ASI instead of AGI

The bar must be underground if things humans do all the time are considered "superintelligence"

How many times have you done each of the following?

- write a well-received book, write a best-seller

- come up with a new company idea, Run that company

- actually have a decent conversation, maybe someday talk somebody out of suicide effectively

- come up with its own ideas or theories that nobody else has presented

- understand the stock market well enough to trade better than an index fund

I just picked the first few from the top of the list. The average human has probably not done any of them.

I couldn't do any of those, but then ~$10k of education later I was able to accomplish one of those things.

I think LLMs are really impressive, but I suspect that we might have overpaid just a bit.

It costs money to train each single human, who is then only productive for a number of years until age takes its toll. Once you have trained one software system, the marginal cost of producing a copy approaches zero. Every subsequent improvement can be broadcasted in a matter of seconds across thousands of data centers. In addition, software does not get sick, age, or die.

Ordinary people do these things all the time. There are new companies made every day, new books top the charts every week/month/year, same for music. People have decent conversations every day. Ordinary people sometimes do have to talk someone out of suicide.

Yes, average humans are not beating the stock market. But the average human is a bit better than you give credit to.

Please consider the context of the question. An artificial intelligence only needs to have the cognitive abilities of a random average human in order to be "AGI".

The average human has never published a bestselling book. A person who has published a bestselling book is an above-average writer. And, therefore, an artificial intelligence capable of writing a bestselling book would be above an average human at the task of writing books. Therefore, somewhere beyond an AGI.

Attempting to redefine AGI to "being better than most humans at most tasks" is moving the goalposts towards artificial superintelligence.

Are you mistaking "average" for the literal average number of people? I am referring to average as in ability. Most authors are pretty average people, sure, the above average ones write war and peace etc. But most books these days are not from these extraordinary people.

Most authors do not write bestsellers.

This subthread was about the root list of claims being "more like ASI instead of AGI." By definition being superhuman means doing things no human can.

Indeed. And there is a wide gap between "being as cognitively capable as the average human" and "doing things that no human can", as you put it. So it is important to be realistic about what things the average human is capable of doing, and writing bestselling books is not one of them. Just as an example.

You're arguing for calling it AGI, but again, this tread was about calling it ASI.

And in order to describe the capabilities of what is AGI and ASI, it is useful to look at examples of what humans can do at a range of capabilities, with some nuance. It is not black and white.

So the goalposts have moved to include continual learning.

In a sense I think no one will agree on a definition of AGI until it becomes impossible to construct any benchmark under which an AI underperforms "average" humans. That or it's defined retrospectively, after it's overwhelmingly obvious it met any such definition.

I hardly think it’s fair to label an objection so old that Turing included it (and discussed it at length) in the list of objections to thinking machines in 1950 “moving the goalposts.”

> These arguments take the form, “I grant you that you can make machines do all the things you have mentioned but you will never be able to make one to do X”. Numerous features X are suggested in this connexion. I offer a selection:

> Be kind, resourceful, beautiful, friendly (p. 448), have initiative, have a sense of humour, tell right from wrong, make mistakes (p. 448), fall in love, enjoy strawberries and cream (p. 448), make some one fall in love with it, learn from experience (pp. 456 f.), use words properly, be the subject of its own thought (p. 449), have as much diversity of behaviour as a man, do something really new (p. 450). (Some of these disabilities are given special consideration as indicated by the page numbers.)

(emphasis added).

Just because a condition is new to you doesn’t imply moving the goalpost. People have been putting forward continual learning and similar conditions like autonomy since 1950s.