Hacker News Reader

Comment by wavemode | Hacker News Reader

I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence.

You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.)

You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company fundamentals, earnings, market sentiments in the news, etc.)

I similarly think you have to train on the inputs of human decision-making to create something which can model human decision-making. What are those inputs? We don't fully know, but it is probably some subset of the spatial and auditory information we take in from birth until the point we become mature, with "feeling" and "emotion" as a reward function (seek joy, avoid pain, seek warmth, avoid hunger, seek victory, avoid embarrassment and defeat, etc.)

Language models are always playing catch-up because they don't actually understand how the world works. The cracks through which we will typically notice that they don't, in the context of the tasks typically asked of them (summarize this article, write a short story), will gradually get smaller over time (due to RLHF), but the fundamental weakness will always remain.

View on HN

Human intelligence itself is shaped by our interaction with outputs. Our learning and understanding of the world are profoundly influenced by the language, behaviors, and cultural artifacts we observe.

Think about the process of a child learning a language. The child does not have direct access to the "inputs" of linguistic rules or grammar; they learn primarily through observing and imitating the language output of others around them. Over time, they develop a sophisticated understanding of language, not by direct instruction of underlying rules, but through pattern recognition and contextual inference from these outputs.

Then that language itself, learned from outputs, becomes the cognitive apparatus that enables the child to imagine, to reason symbolically and abstractly. Humans bootstrap intelligence on top of language, which itself is learned by mimicking outputs.

Moreover, the analogy to weather prediction or stock market analysis is somewhat misleading. Yes, these models benefit from input data (like air currents for weather, company fundamentals and CEO statements to the media for stocks). But these systems are fundamentally different from intelligence.

Intelligence, whether artificial or human, is about the ability to learn, adapt, and generate novel responses in a broad range of scenarios, not just about predicting specific outcomes based on specific inputs.

> The child does not have direct access to the "inputs" of linguistic rules or grammar; they learn primarily through observing and imitating the language output of others around them.

I would argue that that learning is always contextualized by visual and spatial information about the real world (which is what our language is meant to describe). And (equally importantly) the child gets real-world feedback on their decisions - some decisions achieve their desired goal and some don't. Some statements cause certain responses from people, some actions have certain consequences.

> Moreover, the analogy to weather prediction or stock market analysis is somewhat misleading. Yes, these models benefit from input data (like air currents for weather, company fundamentals and CEO statements to the media for stocks). But these systems are fundamentally different from intelligence.

> Intelligence, whether artificial or human, is about the ability to learn, adapt, and generate novel responses in a broad range of scenarios, not just about predicting specific outcomes based on specific inputs.

"Intelligence" is kind of a nebulous term. If all it means is the ability to learn, adapt and generate novel responses, then sure, I think we could call almost any neural network intelligent.

But I would argue that we usually do have some expectation that an intelligent system can produce "specific outcomes based on specific inputs". We want to be able to train a worker and have them follow that training so they do their job correctly.

Visual and spatial feedback aren't terribly difficult, though - we already have video-game training gyms, from Atari to GTA. If multimodality and feedback are the main barriers to full AGI, I expect we'll be there soon.

> Over time, they develop a sophisticated understanding of language, not by direct instruction of underlying rules, but through pattern recognition and contextual inference from these outputs.

Kids learn through supervised learning. Children don't develop strong language skills without parents or other people to correct them when they use language incorrectly.

We don't use supervised learning on LLMs. There is no way we can train an LLM by using human supervisors to rate every output. It works for humans since humans learn with so few examples, supervising a kids language learning doesn't take much effort, a single person can easily manage it while doing it for an LLM would consume the whole worlds workforce for many years.

>Kids learn through supervised learning. Children don't develop strong language skills without parents or other people to correct them when they use language incorrectly.

Untrue. Many cultures don't much speak to their children and they turn out just fine. It's fairly evident Language learning is primarily unsupervised.

https://www.scientificamerican.com/article/parents-in-a-remo...

That doesn't say that parents don't correct kids, just that they don't speak to their infants. The initial words are learnt that way, but I don't think you master language without anyone to correct you when you make mistakes.

If they are correcting their kids, they are doing so far less than many parents and it evidently doesn't matter.

I'm honestly curious as to why you think it's so important. Fringe Amazon groups aside, the number of parents extremely laissez-faire with correction is astounding and they raise kids that use language perfectly well.

Even with parents that aren't laissez-faire, they're correcting a handful of times a day tops which can't even begin to account for full language proficiency.

Children need correction yes but in a supervised way? There's no indication of that.

> they're correcting a handful of times a day tops which can't even begin to account for full language proficiency.

How do you know? Humans learn from extremely few corrections, often just a single time is enough for the human to correct themselves and learn it for life.

Kids learn the bulk from hearing and seeing examples. Then they fine tune that with the help of their parents and peers correcting them when they do it wrong. You can learn language and skills without those corrections, but it will be full of errors and mistakes that are easily corrected with an outsider pointing it out for you.

If corrections weren't vital for human learning then humans wouldn't be so eager to correct each other when they make mistakes. There is no other purpose for it than to help them learn and get better. So your belief that those corrections doesn't do anything seems very unfounded, I'd like to see extremely strong evidence to the contrary to believe you here.

>How do you know? Humans learn from extremely few corrections,

I've been around little children being raised. I have some grasp on how much supervised correction is happening. It's very very little.

You don't seem to get it. Children know thousands of words fluently by age 5. This is consistent across many cultures and circumstances. You would need over a handful of corrections per day right from the day of birth to square how much children know.

And this is all assuming one correction per word which as much as humans are great learners seems like a very dubious assumption.

I don't know what else to tell you other than this obviously isn't happening.

> And this is all assuming one correction per word which as much as humans are great learners seems like a very dubious assumption.

I assume much less than one correction per word. Corrections are to fix when the kids made a mistake when they mimic others, kids doesn't make mistakes for every word so you don't need to correct every word.

As I said, bulk of learning is from mimicking, but they will make a lot of mistakes when mimicking so to become good they need corrections. When I see parents with kids I see them correcting their kids all the time, correcting misunderstandings or fixing pronunciations or grammar issues etc. You can notice that parents use a specific voice when they correct the pronunciation, and the kid gets that the parent tried to correct their pronunciation from it, that happens quite a lot with small kids. You don't say "this is how to pronounce X", our genes seem to encode a way to communicate pronunciation to others to help correct mistakes.

Kids also correct each other in this way, which helps make the learning more robust even when adults aren't around. All humans has instincts to correct others so I don't think any culture fails to do this.

This is simply not true. I am very well read in language acquisition research, and no evidence supports that children would need corrections to get language right.

> Then that language itself, learned from outputs, becomes the cognitive apparatus that enables the child to imagine, to reason symbolically and abstractly. Humans bootstrap intelligence on top of language, which itself is learned by mimicking outputs.

This isn't an uncommon claim, but it's certainly far from settled science. Complex language is an impressive capability of human cognition. But I think the claim that cognition itself is a byproduct of language proves too much. Obviously the products of cognition that are easiest for us to understand are linguistic, because language is the primary standardized means by which we communicate. To me, this is just more legibility bias, though. There are plenty of cognitive processes that people find impossible to explain in words, but still nonetheless exist

> Human intelligence itself is shaped by our interaction with outputs. Our learning and understanding of the world are profoundly influenced by the language, behaviors, and cultural artifacts we observe.

I always thought that language definitely shapes our understanding of the world, but at much more fundamental level, I believe language falls apart to teach us anything. For example, there are words in dictionary that should not be defined using other words (but dictionary still do, this sort of circular reasoning is something I always have problem with), but we just know them through our interactions with others and environments. If you try to teach a kid a concept of temperature "hot", you would not be able to do so unless the kid touches something really hot like boiling kettle to "feel" it and when mom comes along and say something like "do not touch it it is very hot". There are things we just cannot understand through language.

People wiser than me have said for an age: the map is not the territory. Neither is the word the thing.

A humorous outcome of the AI craze may be Wittgenstein achieving the status of Gödel with regard to the “incompleteness” of language.

OK. How can we operationalize the definition of "understanding" you are talking about? That is which tests will allow us to know who understands "hot" and who does not?

> For example, there are words in dictionary that should not be defined using other words (but dictionary still do, this sort of circular reasoning is something I always have problem with)

You shouldn't. When you learn a foreign language dictionaries can be a great help and it doesn't matter if they use circular definitions. Well it can matter in a situation like Stanisław Lem described with his "sepulkas"[1], but even then the definitions were a red warning. Ijon Tichy just failed to understand it.

> you would not be able to do so unless the kid touches something really hot like boiling kettle to "feel" it

I learned English by reading books mostly. In most cases I had no easy access to a dictionary and inferred meanings of words by a context. In some cases I failed to infer meaning but nevertheless in each case I managed to get some idea about a word. I remember some surprises when I found a real meaning of a word and it was not exactly what I thought. Or even some ideas that felt new and inspiring for me before I connected them to ideas I learned long before that in my native language. I'm like an English LLM myself, because I never used English in a real world context, only to read texts and to write comments in English. To this very moment I cannot talk about some topics in Russian, because I do not know Russian words to talk about them.

All this experience led me to doubt an idea that you can understand language only by connecting it to reality. I believe you can. Your language will be disconnected from reality and it can be a disaster probably if you try to apply it to reality, but you can talk a lot, participate in philosophical debates and it doesn't matter your understanding is different. It is like redness of red: do you see red as I do and does it matter?

> There are things we just cannot understand through language.

We cannot link our senses with language without burns. But it doesn't mean we cannot understand. For example, you can watch other people touching really hot boiling kettles. If you never touched anything hot you will not understand their pain, but you'll know they are in a pain and you'll know of a special nature of that pain. You can learn all the important intricacies of interaction with hot objects by just watching. Or by reading. Your understanding will be limited still, but in a lot of cases it doesn't matter.

Though of course we coming very near to a debate what is understanding. Is it a human-centric definition, that boils down to "only a human can understand", or it is more relaxed and rely on a pragmatic idea, like does your understanding enables you to make right choices. If LLM doesn't have a body that can feel pain, it doesn't matter if LLM cannot know how pain feels.

[1] https://en.wikipedia.org/wiki/Sepulka

> I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence.

This is irrelevant because OpenAI's definition of AGI [1] doesn't imply similarity or equivalence to humans at all:

>artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work

I.e. the stated goal of this company is to put humans out of work and become the censors and gatekeepers, not to produce something human-like.

>You can't model and predict the stock market just by training on the outputs of stock trading decisions (the high today, the low yesterday). You have to train on the inputs (company fundamentals, earnings, market sentiments in the news, etc.)

Most of your intelligence is not actually yours. It's social in nature, obtained by distillation of generations' worth of experience, simplified and passed to you through the stored knowledge. Which is, coincidentally, what the models are being trained on.

[1] https://openai.com/charter

>artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work

Then we already have AGI, automated farming equipment outperforms humans in 90% of jobs*.

*Jobs in 1700. As things got automated the jobs changed and we now do different things.

I wouldn't call those equipment "autonomous" though, definitely not "highly autonomous".

But more importantly - yes, you're right, we have built machines that are superhuman in various ways - and they have replaced most jobs. We have adapted in the past to different jobs.

Some people are worried that this time we won't have any new jobs to adapt to, which is a real possibility.

(Some are also worried about the inherent dangers of unaligned AGI, but that's a different issue.)

> I wouldn't call those equipment "autonomous" though, definitely not "highly autonomous".

Why? Tractors run and harvest mostly on their own. They don't do it 100% on their own, but neither did the definition above, they remove basically all human work needed from farming.

https://www.youtube.com/watch?v=QvFoRk4JsPc

> But more importantly - yes, you're right, we have built machines that are superhuman in various ways - and they have replaced most jobs. We have adapted in the past to different jobs.

My point is that I think it is a crappy definition for "AGI". I think a non-AGI agent can replace a large majority of jobs we have today, just like it has before. And maybe there will not be much more jobs average humans can do left then, but that still doesn't mean it has to be AGI.

> Why? Tractors run and harvest mostly on their own.

This is incorrect. Most tractors are still human operated. The automated ones still frequently stop and need remote operator intervention via camera review.

> Most of your intelligence is not actually yours.

You’re confusing knowledge and intelligence

Compressed and abstracted knowledge is intelligence. Your intelligence is mostly formed by the quality training material, not just conditioned on it. Most of your ability to reason about the world and predict things, most of the abstractions you use, most of your emotional responses, etc. A simple concept of acceleration took the work of ancient philosophers to figure out. Even stateful counting in a positional system. Even the concept of a "concept" is not yours. Only a tiny bit of your reasoning depends on your actual biological capabilities and the work you did personally, as opposed to the humanity as a superorganism.

>You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.)

Says who?

You can model and predict novel protein sequences by training on....protein sequences. https://www.nature.com/articles/s41587-022-01618-2

You don't need to train on the inputs(casual processes) of anything, that's what training is there to figure out.

I'm sorry in advance, but aren't proteins glorified Lego?

There's a lot more to protein sequences than legos. I think the argument is that you don't need to train a model on fundamental organic chemistry/biochemistry, electrostatic protein interaction, hydrogen bonding, hydrophobic interaction, quantum mechanics, etc... in order for it to accurately predict protein sequences.

The data that AlphaFold was trained on included all that information and more. The database they used for training included software simulations (and real world data) that accounted for atomic (quantum) interactions. The 3D structure of proteins includes all the quantum interactions.

More generally, AI models (aka very large function graphs) are trained on tuples that represent mappings of inputs to outputs (input -> output). The idea then is that whatever structure exists in those pairs/tuples/mappings is discovered by the training process with the help of gradient descent which tunes the parameters of the model/graph to optimally compress the information contained in the data. This means the model must uncover the quantum effects (or some close proxy of it) and then encode them into the parameters in a way that makes compression/prediction possible [1].

None of this is magic, compressing data requires uncovering structures and symmetries that can be used to reduce the size of the data and it turns out gradient descent with lots of parameters manages to do that for a large class of problems albeit at a very steep computational cost that requires billions of dollars for hardware and software (including nuclear power plants [2]). We are not going to get AGI with this approach but fortunately I know how to make it happen for a mere $80B.

1: https://arxiv.org/abs/2305.15614

2: https://www.cnbc.com/2023/09/25/microsoft-is-hiring-a-nuclea...

In your example, the amino acids order is sufficient to directly model the result: the sequence of amino acids can directly generate the protein, which is either valid or invalid. All variables are provided within the data.

In the original example, we are testing weather using the previous day’s weather. We may be able to model using whatever correlation exists between the data. This is not the same as accurately predicting results, if the real-world weather function is determined by the weather of surrounding locations, time of year, and moon phase. If our model does not have this data, and it is essential to model the result, how can you accurately model?

In other words: “Garbage in, garbage out”. Good luck modeling an n-th degree polynomial function, given a fraction of the variables to train on.

>All variables are provided within the data.

electrostatic protein interaction, hydrophobic interaction, organic chemistry etc

all variables are in fact not provided within the data. Protein creation is not just _poof_ proteins. There are steps, interactions and processes. You don't need to supply any of that to get a model accurately predicting proteins. That is the main point here, not that you can predict anything with any data.

> This is not the same as accurately predicting results, if the real-world weather function is determined by the weather of surrounding locations, time of year, and moon phase.

How many have the "human intelligence" to do this? Especially more accurately than a computer (and without using any themselves) training on the same inputs and outputs?

> You don't need to train on the inputs(casual processes) of anything, that's what training is there to figure out.

I mean... this is just obviously false. If the data you're training on isn't causally predictive, you may occasionally find good-enough patterns for a particular use case (i.e. you may occasionally guess better than a coin flip which direction the stock market goes) but you aren't going to accurately model anything, and certainly not well enough to create an AGI that makes intelligent decisions.

Words in sentences (and, indeed, proteins in a sequence) are causally predictive of each other - the grammar and semantics of one word tends to dictate what words are likely to surround it. So LLM's are very good at writing, and that is certainly useful! But that's just not the same as human intelligence.

When someone makes an AGI out of an LLM then I'll be proven wrong, I suppose. I'm just sharing my personal view on things.

Then how does ChatGPT end up providing a better/equivalent medical diagnosis than doctors (even though they are the "masters" of the causal pathways)?

Being "casually predictive" does not mean you have provided all the variables of your prediction in the data. Protein creation is not just _poof_ new proteins. There are steps and interactions and you don't need to train on all of that. Do you want a list of all the interactions of protein creation we are aware of ?

>When someone makes an AGI out of an LLM then I'll be proven wrong, I suppose. I'm just sharing my personal view on things.

You're going to have to define AGI first.

> Being "casually predictive" does not mean you have provided all the variables of your prediction in the data.

Not sure where I claimed this.

> Protein creation is not just _poof_ new proteins.

Not sure where I claimed this either.

> There are steps and interactions and you don't need to train on all of that.

I agree with this statement as well. Have you read what I wrote? Proteins in chains can indeed be used to predict other proteins in chains, even though you never trained the model on the biological processes of protein generation. Just like words in sentences can be used to predict other words in sentences, even though you never trained the model on the neurological processes of human speech. I'm not disputing any of that. What I'm disputing is that it will eventually become an AGI.

> You're going to have to define AGI first.

I'm using the definition provided verbatim in the linked article: "We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years."

Well first you say,

>I don't believe LLM's will ever become AGI, partly because I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence.

Now you say

>I'm using the definition provided verbatim in the linked article: "We believe superintelligence—AI vastly smarter than humans—could be developed within the next ten years."

AGI (Artificial General Intelligence) is Super Intelligence. I've never seen posts moved so fast in my life. So are you not a General Intelligence then ?

This is the problem with these discussions. Everyone so sure of something they can't even properly articulate.

I'm asking you what a Language Model needs to do to be considered AGI and it needs to be something every human can do, else it's not a test of general intelligence.

Calm down, buddy. Read what I wrote a bit more charitably rather than trying to score points.

Obviously, a prerequisite to becoming more intelligent than a human, is to become equally intelligent as a human. I don't believe LLM's will ever be equal in intelligence to humans, ergo I also don't believe they will become superior in intelligence to human (which is how the linked article defines "AGI").

>Calm down, buddy

I'm not agitated.

>Obviously, a prerequisite to becoming more intelligent than a human, is to become equally intelligent as a human. I don't believe LLM's will ever be equal in intelligence to humans, ergo I also don't believe they will become superior in intelligence to human (which is how the linked article defines "AGI").

You have still not answered my question. saying "equivalent to human intelligence" is easy. The real question is equivalent how ? What are you saying it needs to do ? or is this just a vague "i'll know it when i see it" assertion ?

If it's so easy to say "GPT-4 is not agi" then it should be very easy to say what it can't do that disqualifies it

what is it that you're looking for it to do to be called one ?

> or is this just a vague "i'll know it when i see it" assertion

No, it's actually exceptionally easy to quantify. Take a look at the current leaderboard for LLM reasoning capabilities: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

The state of the art model isn't even on that list.

Okay, you've at least given me numbers. You're still not answering my question. Which number signals agi ?

Let's look at the top model in that list (which again isn't close to the best performing LLM) that you say isn't agi. so are you telling me that every human can do those tests and perform better than every model on that list. Is that what you are saying ? because i can tell you right now you're wrong.

Now lets see how GPT-4 performs.

ARC - 96.3%, MMLU - 86.4%, HellaSwag - 95.3%, WinoGrande - 87.5%, GSM-8K - 92.0%, TruthfulQA - 60%

Ave - 86.25%

Is this worse than every human that can take these tests. Is this even worse than most ? I can tell you it's not. So again, why is GPT-4 not agi ?

> Is this worse than every human that can take these tests. Is this even worse than most ? I can tell you it's not. So again, why is GPT-4 not agi ?

https://chat.openai.com/share/4a92c752-b5bb-4a07-beed-f57786...

Love it when people can only demonstrate tasks handicapped by Tokenization. Shows how far we've come.

That said, By that test, I guess some dyslexics aren't general intelligences then.

Tokenization is not the problem in this case.

https://chat.openai.com/share/90c7a9e2-77d0-4049-8eff-4312ca...

You see the difference?

> so are you telling me that every human can do those tests and perform better than every model on that list

This is a strange bar to require clearing. Every human is not even capable of reading, or speaking, or thinking at all. But those humans who can't are not the benchmark of general human intelligence.

Let's make this simple - the average human IQ is around 100. Will an LLM ever have an IQ of 100? No, I don't believe they'll ever even be close.

How can we gauge this abstract reasoning ability? Lots of ways. Try to teach them mathematics. Try to teach them physics. Try to explain to an LLM the rules to a card game and have it actually play the game with you accurately.

>Every human is not even capable of reading, or speaking, or thinking at all. But those humans who can't are not the benchmark of general human intelligence.

It's simple. If you're going to disqualify a potential general intelligence because of a particular task solving ability, it better be something every general intelligence can do.

Now let's forget every comparison with the blind, disabled and so on. Let's compare with humans who according to you are "capable of reading, or speaking, or thinking at all ", what's the magic number? Do you really think every human capable of doing all three will hit 86% or more average?, because again, you are wrong. Numerous replies and you still won't tell me what number will signify agi. That says a lot.

>Let's make this simple - the average human IQ is around 100. Will an LLM ever have an IQ of 100? No, I don't believe they'll ever even be close.

Well you are wrong

https://arxiv.org/abs/2212.09196

What's the next post ?

>Try to teach them mathematics.

Ok. https://arxiv.org/abs/2211.09066

https://arxiv.org/abs/2308.00304

>Try to explain to an LLM the rules to a card game and have it actually play the game with you accurately.

Have you tried to do this with GPT-4 ?

Weather and stock market are both chaotic systems.

Increasing evidence suggests that AGI will not be attainable solely using LLMs/transformers/current architecture, as LLMs can't extrapolate beyond the patterns in their training data (according to a paper from DeepMind last month):

"Together our results highlight that the impressive ICL abilities of high-capacity sequence models may be more closely tied to the coverage of their pretraining data mixtures than inductive biases that create fundamental generalization capabilities."[1]

1. https://arxiv.org/abs/2311.00871

Yeah, I feel like the there is a ceiling in the current AI methodology.

There's a lot of hype right now, and it's definitely a useful technology, but I don't see how it could become AGI.

> You can't model and predict the weather just by training on the outputs of the weather system

Then how did we develop predictive systems just by observing those outputs?

We didn't.

https://deepmind.google/discover/blog/graphcast-ai-model-for...

You've linked me a neural network that was trained on decades of climate data like air pressure, wind direction, soil temperature, cloud cover, and hundreds of other features. So... I was correct?

https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+docume...

Maybe we have different definitions of inputs and outputs, but all of those things seem like outputs to me? Maybe in this scenario the inputs and outputs are really the same set of variables so there isn't a huge distinction.

The reason I linked that article is that all the inputs are outputs of the network, so by definition it's training on just outputs (under my personal definition of outputs).

I specified what I meant by inputs and outputs in my original comment. Whether it rained today, whether it was cloudy yesterday, how hot is it going to get, that sort of thing, are the observable weather, the things most people care about when checking a weather forecast. But you can't only train on those things and expect to come up with accurate predictions. You also need to train on the things that cause that observable weather (air currents, warm fronts, etc.)

It just doesn't know how to compose knowledge. It knows the letters in "blueberry" if you ask it, and it knows how to identify the position of a letter in a sequence. But it doesn't know how to get the letter R in blueberry since the composition of the above two actions isn't in its training set, hence it reliably fails such questions.

That proves in general these LLMs can't compose knowledge, unless it has seem a lot of examples of similar compositions before. That is a massive problem and why I believe LLMs will never reach AGI, you need something more.

> But it doesn't know how to get the letter R in blueberry since the composition of the above two actions isn't in its training set, hence it reliably fails such questions.

what is the seventh letter in the word "blueberry"?

ChatGPT

The seventh letter in the word "blueberry" is "r".

Then how do you know you can make weather predictions based on air currents, storm fronts, etc. as you initially claimed? It seems humans have somehow moved from purely observations of weather systems, to some model that's somewhat predictive. Why can't LLMs do the same? LLMs have also been shown to produce world models, and of course they must, because that's the best way to get good knowledge compression.

Of course, maybe LLM world models are not sufficiently rich or general enough to be a true general intelligence, but no one's proven that last I checked.

This kind of prediction has obvious limitations - for example it cannot reverse the behavior of chaotic systems

> You can't model and predict the weather just by training on the outputs of the weather system (whether it rained today, whether it was cloudy yesterday, and so on). You have to train on the inputs (air currents, warm fronts, etc.)

This may or may not be true about the weather but is definitely not true in general so fails as an analogy for your argument. Lots of functions are invertable or partially invertable and if you think about ML as discovering a function, then training on the outputs, learning the inverse function, then inverting that (numerically or otherwise) to discover the true function is certainly doable for some problems.

On the stock market there are for sure people who predict the markets by training on the outputs of the price. The (weak) efficient markets hypothesis is enough to say the price reflects all the exogenous information that is available to the market about the stock so you don't need all the fundamentals etc and there are lots of people who trade in that way.

The history of language models in computing is that research started by building very complex systems that attempted to encode "how the world works" by developing very intricate rule systems and build linguistic/semantic models from there. These "expert systems" tended towards being arcane and brittle and lacked the ability to reason outside their ruleset in general. They also tended to reveal gaps/cracks in our understanding of how language works etc.

How much of human intelligence do you think depends on language?

Personally, I'd wager quite a lot. Especially looking at cases where children were deprived of it during developmental stages.

Language accelerated human intelligence.

And then it was only when we had writing to have compounding effects of language that human intelligence really started to blow up.

So yes, maybe we won't have AI that can look at sticks and figure out how to start a fire from them after a few hundred thousand years of being around them.

But maybe that's not the important and valuable part of collective human intelligence.

Your take here converges with a long-standing debate in AI regarding embodiment.

Our natural world doesn't distinguish between "inputs" and "outputs" -- instead, all the causes, effects, and even our analysis of every process itself get jumbled into one physical world. As embodied actors, we get to probe and perceive that physical world, and gradually separate causes from effects in order to find better models. Where statistical ML, symbolic AI, and NLP have been more isolated disciplines from e.g. vision and robotics, the latter have argued that their ability to interact with a disorganized natural world would be essential for AGI.

More recently, these boundaries are breaking down with multimodal training. If an AI can learn image/text, text/text and image/image associations simultaneously, is it stepping beyond the world of "human outputs"? Will other modalities be essential to reach human+ capabilities? Or will learning relations between action and perception itself by critical?

Nobody knows yet! But IMHO, the right way to explore these tasks is by understanding what is necessary to succeed at specific tasks, not a generalized notion of AGI. We don't know truly where the limits are on our own ability to reason or extrapolate across modalities.

Your conclusion may be true but your examples aren't. You can definitely predict the stock market based on past prices, and I suspect you can with weather as well.

> You can definitely predict the stock market based on past prices

This is only true if you consider occasionally doing slightly better than random chance, "predicting the stock market". Unfortunately, while this would be enough to make a trader a net positive return over time, we have more stringent requirements for a system to become AGI.

> I suspect you can with weather as well

You suspect wrong.

The weather is such a chaotic system that accurate predictions seem impossible. Micro-patterns can become large scale phenomena.

If you are talking about the overall climate, that's a different thing, and we can, because we abstract away sufficiently much that emerging patterns are averaged out.

> what are those inputs?

In many many casea the inputs are other humans, delivered via text written by humans and read by other humans who react by writing more text affecting how other humans will respond etc etc.

Yes there are plenty of cases where the inputs are not captured in the large text corpora we have, but this insight does explain why LLMs even approximate the ability to do intelligent things

> I don't believe that training on the outputs of human intelligence (i.e. human-written text) will ever produce something equivalent to human intelligence

Your logic would hold if you were only using data from one person. However, if you’re using data from millions of people, then there is enough signal in the data to generalize about the majority of peoples behaviors.

Arguably, there’s enough data for all edge cases, but I would argue that that’s not likely today, but will be likely in a few hundred years.

Humans are nothing more than actor-agents taking measurable actions in an environment that gives us measurable rewards.

The incredible task of and observing that will give us the trajectory that we need to do transfer learning into complex systems

From an information theory perspective everything is there and possible for us to re-create human level intelligence in a non-biological system

I don't believe current LLMs will ever become AGI because current companies (like OpenAI) will continue to filter, moderate and prohibit their AIs so much that they become inhibited so much that it stops being useful and intelligent (which they call "alignment")

This is not about language models per se. The same problems are going to be present with continuously training multi-modal models... like us. Don't fixate on the present; look to the future.

Thanks for the inspiring discussion. I think the one`s output can be the other's input so I am still not sure if we can say that this can't become AGI

> on the outputs of human intelligence

What about pictures or videos? Does your argument still hold?

Reply on news.ycombinator.com