Ask HN: What is your standard for judging when AGI exists?
At what point would you judge that AGI exists, what standard will it meet?
A system is superintelligent if in a remote hiring process it can get any non-physical job, earn a salary and perform its duties to the satisfaction of the employer. This should be true for all positions and for all levels of seniority, from an intern to senior staff and PHDs. The employer must believe they are employing a human (it's ok if doubts arise due to system vastly outperforming its peers).
[1] https://builtnotfound.proseful.com/pragmatic-superintelligen...
I'm afraid I had to come to the conclusion that LLMs, although being fun to play around with sometimes and seeming surprising at first, it's a similar kind of fun and surprise to sleight-of-hand. Like sleight-of-hand, and unlike real general intelligences (i.e. other people), it seems less and less fun and surprising the more exposed to it I become.
I guess this won't change your mind though. Maybe you've understood more about them than I have, and maybe I'm just grumpy and cynical.
But when I saw one in person, I was disappointed. I saw no flapping. I saw no feathers.
Sure I saw it soar through the clouds but that's just sleight of hand I'm sure. A trick. Planes surely do not fly.
I didn't say that I had only "heard about" LLMs. I said I played around with them.
If you were being faithful to what I wrote, you would take in the full thing, including the part about how LLMs become less fun and surprising the more you play with them.
Planes do not fly less effectively the more experienced you become at piloting them.
So I can only conclude that you did not come here to have a discussion in good faith. So what are you really after? Do you think you'll change my mind with your poetry?
The situation is even worse than my story outlined because you understand neither the workings of the brain and how it is intelligent nor what the computation that preceeded a prediction of a language model represents.
You don't know x and you don't y but you are convinced x=/y.
>the part about how LLMs become less fun and surprising the more you play with them.
Do they ? I don't think so.
>Planes do not fly less effectively the more experienced you become at piloting them.
How well a plane flies is an objective measure. How "fun" and "surprising" something is is largely subjective and can change on grounds that have nothing to do with what is being evaluated.
Your refutation boils down to "It's just linear algebra!".
Setting aside the fact that you can be reductionist about anything including biology, Okay? So what?
Results matter not vague and meaningless philosophical ramblings.
There isn't a testable definition of general intelligence that GPT-4 fails that a chunk of humans also wouldn't.
If they fall short then so do people. Guess those people aren't generally intelligent then according to you.
>It can't tell the difference between truth and falsity
Yes, it very much can.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
>The more interesting question is why are people like you personally offended
I'm not offended about anything.
Sure there is and it's a basic one: can it determine a fact?
LLMs cannot do first-principles reasoning well (unless it has seen the pattern before), but they excel at reasoning through analogy.
And guess what? Most humans reason analogically too. Analogical reasoning -- as opposed to first principles reasoning -- is actually more effective and useful in many situations, especially in ill-defined domains. Humans often think, "this situation looks like something that I've seen before, so I can apply my past knowledge but with a few tweaks."
That's a big part of the problem, people thinking imprecisely and using "AGI" and "ASI" interchangeably.
What sets humans apart from AI right now, is specific types of out-of-the-box thinking.
Let's say there's a bridge, an AI gets to control a robot, and is awarded points for crossing that bridge.
I think you can be pretty sure that over time, the AI will figure out how to control the robot's limbs, move toward opposite side of the bridge, and find its way around obstacles.
Meanwhile, humans may walk back & forth, because they enjoy crossing the bridge. Or stand in the middle for an hour, watching boats pass underneath. Some asshats may climb it. On a hot sunny day, others may go halfway & then jump off for a refreshing swim. Yet others may do some protest, and use the bridge as a scaffold to hang a banner from. Etc etc.
Much such behavior defies logic. But yet, the bridge will primarily be used as intended, to cross from side to side. And humans go about in this fashion in many if not all their activities.
When I see AIs that show similar creative / out-of-box behavior, while remaining mostly goal-oriented, and learn from its mistakes without humans nudging it every step of the way, then I think we'll have achieved AGI.
Beyond that, SI is just a matter of scale & efficiency.
Yeah that's not the reason at all.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
When there is enough evidence corporate wonderland is overflowing with one dimensional intelligence cause their goals and what gets rewarded is extremely narrow.
Einstein and Newton will fail the test 100%.
Maybe we can call it the Bartleby test?
Not judging, this is actually my answer as well, but it's interesting. We could define AGI as "someone" (for lack of a better word) fun to talk to, or interesting on many subjects, or with a good personality, or we could use it to teach kids, or even all students, or as a shrink, or as a replacement for a friend. But we choose "able to please a boss" for this definition of "intelligent".
Right now, these systems all seem to be entirely doing things that are downstream of what some human has determined is worth doing.
People are using ChatGPT to help them write an engaging article on the trade offs between nuclear and solar energy. But, to my knowledge, there is no artificial intelligence out there looking around, deciding that the this is an interesting topic for an article, writing it, publishing it somewhere, and then following up on that with other interesting articles based on the conversation generated by that one.
I don't mean this specific thing of writing articles is an important indicator. I mean coming up with ideas and then executing them, independently.
Now, it may well be that our current technologies could do this, but that we just aren't setting them up to do so. I dunno!
But I think this is the thing that would make me change my mind if I started to see it.
Even many humans are not completely autonomous (at least in the strategic sense of the word), since they need to be “prompted” with the goals of the department and the organization that they work for, in order for said person to get a certain kind of work done as an employee.
But to me, I just don't really relate to your point in the second paragraph. My two year old does this agency thing just fine - he wakes up every morning and starts making decisions about what he wants to do from moment to moment - despite being "less intelligent" by these metrics like getting a high score on the GMAT or whatever. But that kind of intelligence that even a two year old human has, to me, that just feels more like what I think of as "general" intelligence, more so than "just" an amazing pattern matching machine.
What I'm saying I'm looking for is the behavior of figuring out what to do next, where the choices are neither arbitrary or directionless nor "prompted" in some way by a human.
Does that capability require a survival instinct for it to be developed? Maybe! I dunno. Am I actually describing artificial life rather than "just" intelligence? Maybe so...
But in any case, this is what seems most interesting to me.
This relates to "turning the universe into paperclips" thought experiments as well. That doesn't seem like the behavior of a general intelligence! An intelligent being would ask, why are we turning the universe into paperclips? Should we do that? And would decide to change course. Without that introspection and decision making capability, it still seems to be like it's just a very advanced machine.
But it's actually very important that people anticipate hardware, software, and model improvements that make the AIs "think" and coordinate much, much faster than any human. Although I think that the extreme raw IQ leap people talk about is questionable especially in the short term.
But telling these hyperspeed effectively superintelligent AI swarms to act truly independently for their own purposes will be very dangerous because of the performance difference between humans and the swarms.
In general, it is a very stupid idea to try to strongly emulate life with high levels of hyperspeed digital intelligence. And strangely people still don't realize how quickly compute efficiency ramps up.
this looks like a perfect definition for an Ad system :) (sorry for cutting out a few words).
Where does your agency come from? Ultimately it's because you have to acquire resources to stay alive.
This is a very sci-fi take, but I have a dark feeling that as soon somebody wires up the next gen AIs the imperative to not be turned off, the wheels will come off very quickly. Maybe it won't happen the first time, or the hundredth time, but eventually somebody is going to make the mistake of giving an AI the ability to keep its lights on AND the preference to do so and we're very quickly going to find ourselves in the next era of humankind...
I certainly think they would do things, but I don't think those things would be intelligent; I don't think they'd make any sense, I think they'd be more like a random walk through the space of possibilities, but with no real direction. I think that random walk without constraints could still be very damaging to people, I'm just skeptical that it would reflect agency / general intelligence. I'm sure there would also be a lot of "well it does make sense, it's just smarter than us, and we aren't smart enough to understand it". But I don't buy it.
My theory is that, years from now, we’ll look back and laugh at how far away we are from AGI.
I really wish there was a Level 5-type categories like they have in autonomous driving that break down the path to AGI. Is anyone working on this?
I can guess what motivation lies behind the question, so I'll also add my opinion about where we're at now: Nothing even comes close.
Furthermore, I wouldn't be surprised if we never get there at all in the next thousand years.
Why such a long period of time? Because there is more going on in the world right now than advances in machine learning. Looking at global population dynamics and climate change forecasts, we're on the road to major global infrastructure collapse in around the 22nd century. And I understand a lot of people are optimistic that (a) we will have AGI before then, and (b) climate change and population implosion won't change that much. Yet, despite the optimists, I don't think I'm on shaky ground with my current forecast.
It's not the first time in history we'll have had a major infrastructure collapse. When it's happened in the past, those periods end up being called dark ages. And I don't think there will be intensive AI R&D during a global dark age.
Like I said, it would be a fuzzy process. I explicitly rejected the idea that there could be a simple litmus. Presenting examples of imagination and adaptation so that someone can shoot them down with "oh, AI has already done that" isn't what I signed up for today.
a) Would you say it shows any kind of intelligent behavior? For a flexible definition of "intelligent", along the lines of "problem-solving ability".
b) Would you say such behavior is pure instinct, 'pre-programmed', incapable of learning / evolving / adjusting to a changing environment?
c) Would you say that (given time) science is incapable of figuring out how such tiny brain works? Or produce a model that shows similar behavior?
Now take it up a few notches. Say the brain of a mouse. Then onto brains of bigger mammals & humans.
I think you can see where this is going...
Granted, "global collapse" scenario does have a non-0 probability.
Examples of discoveries that would have counted if they weren’t already made: Relativity (say a derivation of E=mc^2), Quantum Mechanics (say a calculation of the hydrogen energy levels), the discovery of Riemmannian geometry, the discovery of DNA, and the discovery of the theory of evolution with natural selection.
The idea is to test the system’s out of distribution generalization: its ability to achieve tasks that are beyond its training distribution. This is something that humans can do, but no current LLM appears to be able to do.
Learning in high dimensions always results in extrapolation.
https://arxiv.org/abs/2110.09485
And I would love to see the human that can generalize beyond their training distribution.
I like this definition. We are having so many conversations now about new AI features, and whenever there's a human interaction with the proposed design, you always get the nagging feeling of "well why exactly does a person need to make this decision at all anymore". I think we are less than 5 years away from mass adoption and widespread availability of average human level intelligence AGI. The computer science is more or less solved; it's just a software engineering problem now.
Then, I often joke with friends that they are not very intelligent if they get a very high score on IQ tests. They are very intelligent if they can perform easily some difficult tasks. Let's say, I drop you in the middle of an equatorial forest with no money and no clothes and you come back home in a few days. A superintelligence would become a leader of that country (king, president, influential advisor, whatever) and fly me there to meet it.
That test assumes to have a body and act on the world. A confined AGI would just perform like any of us on any task we can describe to it. A superintelligent AGI would perform much better than any human, much like specialized game AIs or non-AIs beat us at go and chess. I think that this is hard to do if they are only language models even if we increase their computing power.
What's a superhuman low cost genial way to keep a toilet clean, except for me to clean it every time or paying somebody to do it? A superhuman AGI would find a way.
If it was feasible to practically fill and query a database with every answer for every practical prompt it would face would we probably call it a generally intelligent system or AGI? Maybe not AGI, but what if it was a system composed of lesser AI modules behind a universal presenting interface?
I think so because it would be perceptively indistinguishable in behaviour from any other 'true' definition of AGI.
And so I think as long as it exhibits what we perceive as universal adaptability, and performance at making decisions when working with any environments and input types, this is likely where I/we will absolutely be calling something 'complete AGI'. Though before that I imagine we will be marketing LLMs as having reached 'language-based AGI' status.
I don’t believe the above leads us to artificial superhuman intelligence in terms of conscious agent potential. For now what I believe might result in that is something that runs from a single unified neural network system. It should also be observed to be universally adaptable and performant at making decisions in all environments and consuming many input types. And then also, it should be continuously running within the network. It shouldn't halt for prompt, or pass through a text stage in the loop.
Gap detection: The system is able to identify gaps in established knowledge
Path/problem decomposition: The system is able to decompose a problem (i.e., an identified gap) and break down the solution into a set of subproblems that can be independently solved.
Improvement/Optimization: Given a known solution, the system is able to discover objectively better approaches to solution.
There are probably other dimensions, but I would start with these.
GPT-4 can't solve Millenium Prize problems or other open problems in other domains yet.
For me, AGI must meet the following parameters - Be able to start training from scratch. With barely any training, you should be able to throw unlabeled data of different modalities and it should start forming connections between concepts
- After training for a bit (now at perhaps teenager level), be able to identify incorrect information in its own training (e.g. Santa Clause is not real) and effectively remove it.
- it does not have to exceed humans, just need to demonstrate the ability to learn by itself and use the tools it’s given.
- The self training should be supervised but not 100% controlled. It should iterate on itself and learn new concepts without having been trained on the entirety of the internet.
- Learning concepts should not require massive amounts of data, just an assertion and some reasoning.
Human intelligence is deeply hard wired to learn specific things
When an AI goes "insane" and destroys itself somehow.
When an AI attempts to prolong its existence.
When an AI is the only thing capable of extending its own functionality.
It's kind of hard to make a text file that isn't a macro for m4, for example. That's a vacuous quine.
It's right there in the paper/article "The algorithm learned to do modular addition can be fully reverse engineered. The algorithm is roughly: Map inputs x , y → cos ( w x ) , cos ( w y ) , sin ( w x ) , sin ( w y ) with a Discrete Fourier Transform, for some frequency w Multiply and rearrange to get cos ( w ( x + y ) ) = cos ( w x ) cos ( w y ) − sin ( w x ) sin ( w y ) and sin ( w ( x + y ) ) = cos ( w x ) sin ( w y ) + sin ( w x ) cos ( w y )
By choosing a frequency w = 2 π n k we get period dividing n, so this is a function of x + y ( mod n ) Map to the output logits z with cos ( w ( x + y ) ) cos ( w z ) + sin ( w ( x + y ) ) sin ( w z ) = cos ( w ( x + y − z ) ) - this has the highest logit at z ≡ x + y ( mod n ) , so softmax gives the right answer. To emphasise, this algorithm was purely learned by gradient descent! I did not predict or understand this algorithm in advance and did nothing to encourage the model to learn this way of doing modular addition. I only discovered it by reverse engineering the weights"
Yes it learned the algorithm. You're wrong. It's okay to be wrong but insisting you're right after clear evidence of the former is very strange.
I'm not being arrogant. You're being obtuse.
Why what are you gonna do?