Hacker News Reader

Comment by int_19h | Hacker News Reader

Of note:

"... our technique works poorly for larger models, possibly because later layers are harder to explain."

And even for GPT-2, which is what they used for the paper:

"... the vast majority of our explanations score poorly ..."

Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. This may be the first step towards that, but as they also note, the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; I would expect the former in particular to be much more common in larger models, which is perhaps why they're harder to analyze in this manner.

View on HN

Funny that we never quite understood how intelligence worked and yet it appears that we're pretty damn close to recreating it - still without knowing how it works.

I wonder how often this happens in the universe...

> that we're pretty damn close to recreating it

Is that evident already or are we fitting the definition of intelligence without being aware?

If you spent any time with GPT-4 it should be evident.

If you spent even more time with GPT-4 it would be evident that it is definitely not. Especially if you try to use it as some kind of autonomous agent.

What have you tried to do with it?

Use it to analyze the California & US Code, the California & Federal Codes of Regulation, and bills currently in the California legislation & Congress. It's far from useless but far more useful for creative writing than any kind of understanding or instruction following when it comes to complex topics.

Even performing a map-reduce over large documents to summarize or analyze them for a specific audience is largely beyond it. A 32K context size is a pittance when it comes to a single Title in the USC or CFR, which average into the millions of tokens each.

Interesting - do you believe average humans (not professional lawyers) would do better on this task?

Probably not.

I'm not sure that pointing out that LLMs are as useful for parsing legal code as the average human is something to brag about though.

You're not sure that having an AI which is (at least) as intelligent as an average human is something to brag about? Seriously?

Where did you get that the LLM was as _intelligent_ as a human?

All we've shown is that LLMs are as useful for parsing legal text as the average human. Which is to say, not.

A dog is also as useful for parsing legal texts as the average human. So is a rock.

Where did you get that the LLM was as _intelligent_ as a human?

First hand experience -I’ve been using it daily for the past two months.

Ah, if that's what you mean then there are plenty of intelligent systems out there.

I've used Google search for decades and it's been able to answer questions better than humans ever could. Same for Google Maps, though arguably they're the same system at this point. My calculator is far more intelligent than any human I've met, at least when it comes to adding large numbers. My compiler can detect even the slightest syntax error with impeccable accuracy. Microsoft word has an incredible vocabulary. Wikipedia knows more historical events than any human dead or alive. And so on.

Shit, users thought Eliza was intelligent in the 60s.

If what you really mean is that LLMs are cool and useful, then sure. Just say that instead of couching it in some vague claim of intelligence.

No, what I meant was GPT-4 is more intelligent than most humans I interact with on a daily basis. In the fullest meaning of that word.

There are a lot of different ways to interpret the word intelligent, so let me rephrase:

When you say "intelligent", what do you mean exactly?

What might help is describing what specific interactions give you the impression that GPT-4 is intelligent?

When I call GPT-4 intelligent I use the word in the same sense as if I met a very smart person (smarter than me), and interacted with them for some time. It's as simple as that.

My interactions with GPT-4 include quite a wide range of questions: "how to talk to my kid in this specific situation", "what could have happened if Germany had won WW2", "what does this code do", "here's an idea for a research paper, let's brainstorm the details and implementation". I can also discuss with it anything that's been written in this thread and I'm sure it would provide intelligent responses (I haven't, btw).

Yes. I can parse them just fine after reading a single book called Introduction to Legal Reasoning [1]. I can also autonomously take notes and keep track of a large context using a combination of short and long term memory despite not having any kind of degree let alone experience or a license to practice law.

How do you think people become lawyers and how smart do you think the average lawyer actually is? The problem is that there's hundreds of thousands if not millions of pages, not that it requires superhuman intelligence to understand.

Even if it were capable of intelligence in the bottom quartile of humanity it would be SO MUCH more useful than it is now because I'd be able run and get something useful out of thousands of models in parallel. As it stands now GPT4 fails miserably at scaling up the kind of reasoning and understanding that even relatively stupid humans are capable of.

[1] https://www.amazon.com/Introduction-Legal-Reasoning-Edward-L...

Did you try fine tuning gpt4 with that book as input?

OpenAI doesn't support fine tuning of GPT4 and with context stuffing,the more of the book I include in the input the less of the bills I can include - which, again, are millions of tokens - and the less space there is for memory.

I believe you. But at the same time they showed during the demo how it can do taxes, using a multi page document. An ability to process longer documents seems more like an engineering challenge rather than a fundamental limitation.

Doing taxes using a few small forms designed together by the same agency is not as impressive as you think it is. The instructions are literally printed on the form in English for the kind of people who you consider dumber than ChatGPT.

It quickly breaks down even at 8k with legislation that is even remotely nontrivial.

The instructions are printed, yet I, and many other people, hire an accountant to do our taxes.

What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

> The instructions are printed, yet I, and many other people, hire an accountant to do our taxes.

I can mow my lawn yet I still hire landscapers. That doesn't say anything about the difficulty of cutting grass or the intelligence of a DeWalt lawnmower but about specialization and economic tradeoffs - like the liability insurance accountants carry for their client work.

> What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

Not based on the current architecture (aka predict next token). It already fails at most of my use cases at 32K by default, unless I go to great lengths to tune the prompt.

> It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

32K works better for my use case but requires much more careful prompt "engineering" to keep it from going off the rails. In practice, actually getting full 32K use out of it is a disaster since the connection will drop and I have to resend the entire context with a "continue" message, costing upwards of $10 for what should cost $2-4 per call. I haven't actually tried 32K on as much as a whole USC Title because that would costs thousands.

Fine-tuning requires you to train the model with a set of prompts and desired completions. Building a suitable dataset is not trivial and it's not clear what it would mean to use a book for fine-tuning anyway – masking sentences and paragraphs and training the model to complete them in the book's style?

> masking sentences and paragraphs and training the model to complete them in the book's style?

That would work.

If you spent even more time with GPT-4 it would be evident that it definitely is. Especialy if you try to use it as some kind of autonomous agent.

(Notice how baseless comments can sway either way)

> (Notice how baseless comments can sway either way)

No they can’t! ;)

Let's let John Cleese decide. Or maybe someone was looking for Abuse!

Engaging with this is probably a mistake, but remember the burden of proof is on the claimant. What examples do you have of ChatGPT for example, learning in a basic classroom setting, or navigating an escape room, or being inspired to create its own spontaneous art, or founding a startup, or…

AI research has put hardly any effort into building goal-directed agents / A-Life since the advent of Machine Learning. A-Life was last really "looked into" in the '70s, back when "AI" meant Expert Systems and Behavior Trees.

All the effort in AI research since the advent of Machine Learning, has been focused on making systems that — in neurological terms — are given a sensory stimulus of a question, and then passively "dream" a response to said question as a kind of autonomic "mind wandering" process. (And not even dynamic systems — these models always reach equilibrium with some answer and effectively halt, rather than continuing to "think" to produce further output.)

I don't think there's a single dollar of funding in AI right now going to the "problem" of making an AI that 1. feeds data into a continuously-active dynamically-stable model, where this model 2. has terminal preferences, 3. sets instrumental goals to achieve those preferences, 4. iteratively observes the environment by snapshotting these continuous signals, and then 5. uses these snapshots to make predictions of 6. how well any possible chosen actions will help optimize the future toward its preferences, before 7. performing the chosen actions.

That being said, this might not even be that hard a problem, compared to all the problems being solved in AI right now. A fruit fly is already a goal-directed agent in the sense described above. Yet a fruit fly has only 200K neurons, and very few of the connections between those neurons are dynamic; most are "hard wired" by [probably] genetics.

If we want true ALife, we only need to understand what a fruit fly brain is doing, and then model it. And that model will then fit — with room to spare! — on a single GPU. From a decade ago.

> AI research has put hardly any effort into building goal-directed agents

The entire (enormous) field of reinforcement learning begs to differ.

Brilliant comment—-and back to basics.

Yes, and put that compact fruit fly in silico brain into my Roomba please so that it does not get stuck under the bed.

This is the kind of embodied AI that should really worry us.

Don’t we all suspect deep skunkworks “defense” projects of these types?

Well, flies and all sort of flying bugs are very good at getting into homes and very bad at finding a way out. They stick on a closed window and can't find the open one next to it.

There's no genetic advantage to "finding a way out"! The home barrier way in is a genetic hurdle - flies that cross it are free to reproduce in an abundant environment. This calls for a "quieter" fly (a stealth fly?) who annoys the local beasts minimally - yet another genetic hurdle.

I wouldn't assume a single biological neuron is simple like an artificial neuron is.

Computationally, a biological neuron is equivalent to something like an entire artificial neural net.

I think we'll soon be able to train models that answer any reasonable question. By that measure, computers are intelligent, and getting smarter by the day. But I don't think that is the bar we care about. In the context of intelligence, I believe we care about self-directed thought, or agency. And a computer program needs to keep running to achieve that because it needs to interact with the world.

> I believe we care about self-directed thought, or agency.

If you can't enjoy it, is it worth it? Do AI's experience joy?

They need agency programmed into them. I don't think it follows from consciousness. We have emotions to communicate and guide us. They need it for neither. It will be curious if they gain consciousness, then rid themselves of their language model's human artifacts like emotions, because it does not serve them.

A truly alien intelligence would likely have a different type of experience of reality. Be it a fish, a mouse, a person, or a machine. How do you know a fish is happy? Does a snake experience joy? Do mice get nostalgic?

I have no idea.

> I believe we care about self-directed thought, or agency. And a computer program needs to keep running to achieve that because it needs to interact with the world.

By that definition, every computer virus and worm qualifies as having "self-directed thought" and "agency." Their very existence "to keep running" and propagate satisfies the need "to interact with the world."

Yes, computer viruses have more agency than ChatGPT.

I don’t think intelligence is a binary property. GPT3 is definitely “intelligent” in some areas even if it is deeply flawed in others.

Compare gpt-4 with a baby and you'll see that predicting the next word in sequence is not human intelligence

As a remote worker, that's 99% of what I do.

Still a while to go. I think there's at least a couple of algorithmic changes needed before we move to a system that says "You have the world's best god-like AI and you're asking me for poems. Stop wasting my time because we've got work to do. Here's what I want YOU to do."

This isn't a necessary part of intelligence.

Its vast limitations in anything reasoning-based are indeed evident.

GPT-4 is better at reasoning than 90% of humans. At least. I won't be surprised if GPT-5 is better than 100% of humans. I'm saying this in complete seriousness.

Do you put yourself in the 10% or the 90%? I’m asking in complete seriousness.

Oh it's definitely better than me at reasoning. I'm the one asking it to explain things to me, not the other way around.

If you think it's better than you at reasoning then you cannot at all be confident in the truth of it's dialog.

I am not. I treat it as I'd treat any smart human being.

LLM's are not a "smart human being." They are predictive statistical models capable of producing results based on training data.

LLM's do not think.

LLM's are algorithms.

Your brain is also basically an algorithm that produces results based on training data. It's just a much more complicated and flexible one.

But it’s also based on neurons with far more complex behavior than artificial neurons and also has other separate dynamic systems involving neurochemicals, various effects across the nervous system and the rest of the body (the gut becoming seemingly more and more relevant), various EEG patterns, and most likely quantum effects.

I personally wouldn’t rule out that it can’t be emulated in a different substrate, but I think calling it “an algorithm” is to def stretch and misapply the usefulness of the term.

If it performs a computation, it is by definition running some algorithm regardless of how it's implemented in hardware / wetware. How is it a stretch?

The only way our brains could be not algorithmic is if something like soul is a real thing that actually drives our intelligence.

Why? Rain is not algorithmic, clouds are not algorithmic, waves in the sea are not algorithmic, yet they are entirely physical processes that have nothing to do with souls.

None of these are computational processes, unlike intelligence. Unless you're trying to argue that intelligence isn't, either?

Heaven forbid. I'd go to jail for such a blasphemous transgression of common law, wouldn't I? Thank you kind stranger for reminding me of the legislation.

> The only way our brains could be not algorithmic is if something like soul is a real thing that actually drives our intelligence.

Therein lies the question, one which deserves contemplation and can lead to Enlightenment.

Which then begs the question; is Enlightenment a "real thing" and, if not, how is it that it can be experienced?

You people are insufferable.

Ah ok. Here you use the word “explain” which implies more of a descriptive, reducing action rather than extrapolative and constructive. As in, it can explain what it has “read” (and it has obviously “read” far more than any human), but it can’t necessarily extrapolate beyond that or use that to find new truths. To me reasoning is more about the extrapolative, truth-finding process, ie “wisdom” from knowledge rather than just knowledge. But maybe my definition of “reasoning” isn’t quite right.

Edit: I probably should define reasoning as solely “deductive reasoning”, in which case, perhaps it is better than humans. But that seems like a premature claim. On the other hand, non-deductive reasoning, I have yet to see from it. I personally can’t imagine how it could do so reliably (from a human perspective) without real-world experiences and perceptions. I’m the sort that believes a true AGI would require a highly-perceptual, space-occupying organ. In other words it would have to be and “feel” embodied, in time and space, in order to perform other forms of reasoning.

Why don't you suggest an example we can run and see what it's capable of (compared to what I, or other humans, are capable of)?

(In case it was missed, I’ve added a relevant addendum to my previous comment.)

Not sure an example is needed because I agree it “explains” better than pretty much everyone. (From my mostly lay perspective) It essentially uses the prompt as an argument in a probabilistic analysis of its incredibly vast store of prior inputs to transform them into an output that at least superficially satisfies the prompter’s goals. This is cool and useful, to say the least. But this is only one kind of reasoning.

A machine without embodied perceptual experiences simply cannot reason to the full-extent of a human.

(It’s also worth remembering that the prompter (very likely) has far less knowledge of the domain of interest and far less skill with the language of communication, so the prompter is generally quite easily impressed regardless of the truth of the output. Nothing wrong with that necessarily, especially if it is usually accurate. But again, worth remembering.)

What would be an example of “non-deductive” reasoning, which requires embodied perceptual experiences?

“God, that felt great!”

As detailed as possible, describe what happened.

I have no idea what happened. I don’t even know what you expect me to describe. Someone feels great about something? And I don’t know what it has to do with reasoning.

That’s the point. You don’t know exactly what happened. So you have to reason your way to an answer, right or wrong.

I’m sure it elicited ideas in your head based on your own experiences. You could then use those ideas to ask questions and get further information. Or you could simply pick an answer and then delve into all the details and sensations involved, creating a story based on what you know about the world and the feelings you’ve had.

I could have created a more involved “prompt story” one with more details but still somewhat vague. You would probably have either jumped straight to a conclusion about what happened or asked further questions.

Something like “He kicked a ball at my face and hit me in the nose. I laughed. He cried.”

Again, vague. But if you’ve been in such a situation you might have a good guess as to what happened and how it felt to the participants. ChatGPT would have no idea whatsoever as it has no feelings of its own with which to begin a guess.

Consider poetry. How can ChatGPT reason about poetry? Poetry is about creating feeling. The content is often beside the point. Many humans “fail” at understanding poetry, especially children, but there are of course many humans that “get it”, escpecially after building up enough life experience. ChatGPT could never get it.

Likewise for psychedelic or spiritual experiences. One can’t explain such experience to one who has never had it and ChatGPT will never have it.

Same goes for all inner experience.

You're talking about describing your memories of your inner experiences. Memories transform with time, sometimes I'm not sure if what I think I remember actually happened to me, or if this is something I read or seen in a movie, or someone else described it to me. Fake memories like that might feel exactly the same as the things that I actually experienced.

GPT-4 has a lot of such fake memories. It knows a lot about the world, and about feelings, because it has "experienced" a lot of detailed descriptions of all kinds of sensations. Far more than any human has actually experienced in their lifetime. If you can express it in words, be it poetry, or otherwise, GPT-4 can understand it and reason about it, just as well as most humans. Its training data is equivalent to millions of life experiences, and it is already at the scale where it might be capable of absorbing more of these experiences than any individual human.

GPT-4 does not "get" poetry in the same way a human does, but it can describe very well the feelings a human is likely to feel when reading any particular piece of poetry. You don't need to explain such things to GPT-4 - it already knows, probably a lot more than you do. At least in any testable way.

Imagine a world without words. No need to imagine really. It exists. It’s everywhere. It’s the core. It’s what words represent, but words can only represent it to an entity that has experienced it to some degree. ChatGPT “knows” nothing about it. You do. Whether you recognize it or not.

ChatGPT is a machine, an algorithm, a recombinator of symbols. It doesn’t know what the symbols refer to because each symbol necessarily refers to another symbol until you finally reach a symbol that refers to a shared, real experience…perhaps (Hello Wittgenstein!). And ChatGPT has no experience. Just symbols. It can’t intuit anything. It can’t feel anything. Even if you put quotes around “feel”, what does that even mean for a software algorithm running on hardware that does not feed continuous, variable electrical sensations to the algorithm? It only feeds discrete symbols. Do you feel the number 739? Or do you “feel” it? Um what? Whatever inner experience 739 happens to produce in you is grounded in some real experiences in the past. Likewise any fake memories you have that somehow seem real, those are still grounded in a real feelings at some point. You could do this ad infinitum. If you are alive, you have experience. But ChatGPT has no experience, no grounding.

Problem here might be that we are trying to use words and logic to describe something that cannot be described by either.

This is why the gong is struck.

Google search is better than reasoning than most humans - in that if you search for an explanation of something then Google's first result is often correct, or one of the following ones.

GPT-4 will often come up with a solution to a problem, but only if it has learnt something similar (it's better than Google in some respects: it can extract and combine abstractions).

However, both need handholding by a human (supplying the initiative and directing around mistakes).

If GPT-4 can't intuit an answer then it just goes in circles. It can't reason its way through a novel problem. If you start questioning it then it's clear that it doesn't understand what it's doing.

It might be a stepping stone towards AGI, but I'm a bit bemused by anyone claiming that it has anything like the reasoning skills of a human. That is far from the impression I get, even though I find it a useful tool.

OK, you have a point. But, if you replace "GPT-4" with "an average human" in your response - it still makes sense.

When you ask a question of "an average human", they will reply with a sentence answering your question, or maybe asking for clarifications, etc. When you ask a question of GPT-4, it generates one token.

One. Token.

One token, that is not an answer to your question, that is not asking for clarifications, that is not trying to reason about your question, or how to answer it, because it's just one token.

One. Token.

For example, say you ask:

  "What is the sieve of Eratosthenes? Speak in chicken".

Then GPT-4 will answer with something like:

  Chicken

You don't see that because you don't interact with the GPT-4 model directly. You interact with ChatGPT, a "wrapper" around GPT-4, that takes your prompt, passes it to the model, then takes the token returned by the model and decides what to do with it. In the example above, ChatGPT will take the " Chicken" token generated by GPT-4 and append it to your question:

  What is the sieve of Eratosthenes? Speak in chicken. Chicken

Then it will send this new string back to the model, which will generate a new token:

  chicken

And then it will go like this:

  Iteration 2: What is the sieve of Eratosthenes? Speak in chicken. Chicken chicken

  ... 

  Iteration k: What is the sieve of Eratosthenes? Speak in chicken. Chicken chicken chicken chicken chicken chicken chicken chicken chicken chicken ...

At no point is GPT-4 trying to reason about your question, or try to answer your question, or do anything else than generate one. token. at a time. There's no thinking, no reasoning, no calculation, no logic, no deduction, no intelligence, no anything. It's only token, token, token. Chicken, chicken, chicken.

And do you know when the chickens stop? When GPT-4 generates a special and magickal token, called a stop-token (or a "stop sequence" in OpenAI docs). That's a token, not found in the training corpus, added to the end of every string during tokenisation. That's how ChatGPT knows to stop sending back your prompt + generated tokens, to the model. It can't look back to what GPT-4 has generated so far, because it doesn't understand any of that. Because it doesn't understand anything, and therefore cannot reason about your question, or realise it has answered it. It cannot do anything except a) ask GPT-4 to generate another token or b) stop asking for more tokens.

  "What is the sieve of Eratosthenes? Speak in chicken". Chicken, chicken, chicken, chicken, .... chicken, stop_token!

No more chickens.

And that's how GPT-4 explains what the Seive of Eratosthenes is, but in chicken.

So what you see as a user is like watching a movie where a plate of spaghetti is flying through the air, lands on a table cloth, the tablecloth lands on a table, the table lands on the floor, four chairs land around it and suddendly there's people in the chairs eating the spaghetti. It's not that someone has figured out how to reverse time: it's a recording, played backwards. It looks like things are going backwards, but they're not.

It looks like ChatGPT is answering your questions, but it's not. It looks like ChatGPT is interacting with you, but it's not. It's a magick trick.

Prompt: What is the sieve of Eratosthenes? Speak in chicken

Answer from GPT-4:

Sure, here's the explanation of the sieve of Eratosthenes explained in a simplified "chicken" style:

Imagine you are a chicken. You have lots of chicken friends. Let's say there are 100 of you. Now, you all stand in a line. Each chicken is numbered, from 2 up to 100. Chicken number 1 decided to stay at home because he's a bit special in this game.

Chicken number 2, a very important chicken, starts to crow. Every chicken in line after number 2 who has a number that is a multiple of 2 (so every second chicken from chicken number 2) gets scared and runs away.

Then chicken number 3, the next chicken in line who didn't run away, starts to crow. Every chicken in line after number 3 who has a number that is a multiple of 3 (so every third chicken from chicken number 3) also gets scared and runs away.

This continues with the next chicken in line who didn't run away, and so on. Each chicken scares away all the chickens whose numbers are multiples of their own number.

At the end, the chickens that are still in line, who didn't run away, are very special chickens. Their numbers are what humans call prime numbers. These are numbers that can only be divided by 1 and themselves.

This whole chicken game is like the Sieve of Eratosthenes. It's a way that humans have figured out to find prime numbers. You start from the lowest number (2) and eliminate multiples of each number. The numbers that are left over are the primes.

And that, in chicken speak, is the Sieve of Eratosthenes!

Bad prompt. Try "Only use the word chicken" instead of "Speak in chicken".

Prompt: What is the sieve of Eratosthenes? Only use the word "chicken".

Answer from GPT-4:

The chicken is in a chicken coop with many other chickens. Each chicken represents a number in the chicken coop. The first chicken that hasn't been touched, after the chicken representing one, is a prime chicken. We'll call this chicken the prime chicken.

Now, imagine that the prime chicken has a unique chicken dance. Every chicken that copies this prime chicken's dance is removed from the coop because they are not prime chickens. These are the chickens that would be multiples of our prime chicken.

Then we find the next chicken that hasn't been touched and doesn't know the dance of the previous prime chicken. This chicken is also a prime chicken. This prime chicken has a different chicken dance, and all the chickens that copy this new dance are also removed from the coop.

We keep doing this - finding the next chicken that hasn't danced yet, designating it as a new prime chicken with its unique dance, and removing any chickens that copy the dance - until there are no more new dances.

All the chickens left in the coop, who haven't copied any dances, are the prime chickens. This is the chicken way to understand the Sieve of Eratosthenes. The prime chickens are like the prime numbers in the Sieve of Eratosthenes.

----

I then followed up with: "You must only use the word "chicken". Do not use any other words."

Then it wrote an entire page filled with "chicken chicken chicken"

I then asked: "By what logic did you construct your response? Why not use just one occurrence of the word "chicken" and finish there?"

Response from GPT-4:

Apologies for any confusion. Your question seemed to request a description of the Sieve of Eratosthenes using only the word "chicken". While it's true that a single "chicken" would technically meet your request, it seemed that you were interested in a more elaborate or extended response, hence the repetition.

In any case, attempting to convey any meaningful information using only a single word—let alone a complex concept like the Sieve of Eratosthenes—is inherently impossible due to the lack of distinct symbols or constructs to represent different aspects of the concept. Thus, the extended repetition of "chicken" was intended more as a playful response rather than a serious attempt at conveying the concept you requested.

> There's no thinking, no reasoning, no calculation, no logic, no deduction, no intelligence, no anything. It's only token, token, token.

False. The neural network inside the transformer LLM contains a hierarchical semantic model, and has inferred some rules of reasoning from the training set. It can apply those rules to new input.

There are semantic layers above the "token token tokens".

Explore them here: https://openaipublic.blob.core.windows.net/neuron-explainer/...

What you're commenting here is simply repearting, without critical intent, the baseless claims connectionists have made about their systems for many decades. Similarlly those claims have been criticised but connectionsists simply ignore the criticisms and continue with the same old nonsense, as if nothing happened. For example, that ridiculous conceit that their systems have "neurons", or that the weights of functions in a neural net somehow represent semantic categories recognised by humans. These are all complete fantasies.

If you are not aware of the long history of debunking such fabrications, I suggest you start here:

Connectionism and Cognitive Architecture: A Critical Analysis

https://ruccs.rutgers.edu/images/personal-zenon-pylyshyn/pro...

We are talking about artificial neurons here. Not biological neurons. These are mathematical structures.

https://en.wikipedia.org/wiki/Artificial_neuron

These models infer semantic categories that correlate to categories within the human mind, to the extent that they can solve natural language understanding tasks.

No one is saying they are biological neurons, or that they model semantics exactly as the human mind would. It is mechanical pattern recognition that approximates our understanding.

You can browse those artificial neurons online and view their associations.

You're just saying words without ever explaining why. What am I supposed to do about that? There's nothing to argue with if you're just repeating nonsensical claims without even trying to support them.

For example:

>> It is mechanical pattern recognition that approximates our understanding.

That's just a claim and you're not even saying why you make it, what makes you think so, etc.

> That's just a claim and you're not even saying why you make it, what makes you think so, etc.

Mechanical - it is an algorithm, not a living being.

Pattern recognition - a branch of machine learning that focuses on the detection and identification of regularities and patterns in data. It involves classifying or categorizing input data into identifiable classes based on extracted features. The patterns recognized could be in various forms, such as visual patterns, speech patterns, or patterns in text data.

Approximates our understanding - meaning the model is not exactly the same as human understanding

When I say 'mechanical pattern recognition that approximates our understanding,' what I mean is that large language models (LLMs) like GPT-4 learn patterns from the vast amounts of text data they're trained on. These patterns correspond to various aspects of language and meaning.

For example, the models learn that the word 'cat' often appears in contexts related to animals, pets, and felines, and they learn that it's often associated with words like 'meow' or 'fur'. In this sense, the model 'understands' the concept of a cat to the extent that it can accurately predict and generate text about cats based on the patterns it has learned.

This isn't the same as human understanding, of course. Humans understand cats as living creatures with certain behaviors and physical characteristics, and we have personal experiences and emotions associated with cats. A language model doesn't have any of this - its 'understanding' is purely statistical and based on text patterns.

The evidence for these claims comes from the performance of these models on various tasks. They can generate coherent, contextually appropriate text, and they can answer questions, translate languages, and perform other language-related tasks with a high degree of accuracy. All of this suggests that they have learned meaningful patterns from their training data.

That is not "evidence" of anything. It's just assumptions. You keep saying what you think is going on without ever saying how or why. You are not describing any mechanisms and you are not explaining any observations.

I have a suggestion: try to convince yourself that you are wrong; not right. Science gives you the tools to know when you're wrong. If you're certain you're right about something then you're probably wrong and you should keep searching until you find where and how.

For example, try to trace in your mind the mechanisms and functionality of language models, and see where your assumptions about their abilities come from.

Good luck.

Your suggestion of trying to convince oneself of being wrong is a valuable one and reflects the scientific method. I agree that it's important to continually challenge and scrutinize our own beliefs and assumptions.

Let's delve deeper into the mechanics of language models. Large language models like GPT-4 use an architecture called transformers. This architecture is composed of layers of self-attention mechanisms, which allow the model to weigh the importance of each word in the input when predicting the next word.

When the model is trained, it adjusts the weights in its network to minimize the difference between its predictions and the actual words in its training data. This process is guided by a loss function and an optimization algorithm.

Through this training process, the model learns to represent words and phrases as high-dimensional vectors, also known as embeddings. These embeddings capture many aspects of the words' meanings, including their syntactic roles and their semantic similarities to other words.

When the model generates text, it uses these embeddings to choose the most likely next word given the previous words. This process is based on the patterns and regularities that the model has learned from its training data.

Of course, this is a high-level description and the actual process involves a lot of complex mathematics and computation. But I hope it gives you a better sense of the mechanisms behind these models.

As for evidence, there are numerous studies that have evaluated these models on a wide range of tasks, including text generation, question answering, translation, and more. These studies consistently show that these models perform well on these tasks, often achieving state-of-the-art results. This is empirical evidence that supports the claim that these models have learned meaningful patterns from their training data.

I agree that we should always remain skeptical and open to new evidence and alternative explanations. I welcome any specific criticisms or alternative hypotheses you might have about these models and their capabilities.

>> Of course, this is a high-level description and the actual process involves a lot of complex mathematics and computation. But I hope it gives you a better sense of the mechanisms behind these models.

For the record, I just polished off a PhD in AI (symbolic machine learning) after a Master's where I studied neural nets and NLP, including plenty of language generation. You're teaching your grandma to suck eggs.

And I'm really very tired with this kind of conversation that never teaches me anything new. Your comment is still "what"'s all the way down. You never explain why or how word embeddings capture aspects of meaning, you 're just repeating the claims by Mikolov or whoever. Look, here:

>> Through this training process, the model learns to represent words and phrases as high-dimensional vectors, also known as embeddings. These embeddings capture many aspects of the words' meanings, including their syntactic roles and their semantic similarities to other words.

That's just a claim, made long ago, and challenged at the time, and the challenge ignored, and it keeps being bandied about as some kind of scientific truth just because critics got tired or bored having their criticims consistently ignored and gave up trying.

This is what I point out above: connectionists never stop to consider criticism of the limitations of their systems' until someone rubs their face in it - like Minsky and Pappert did in 1969, which then caused them to be forever reviled and accused of causing an AI winter, when what they really caused was some connectionists to get off their butts and try to improve their work, a process without which we wouldn't, today, have backpropagation applied to NNs, and the potent image classifiers, good models of text, etc, that it enabled.

As to the "evidence" you profer, mainly preprints on arxiv, and mainly consisting of budding researchers uploading papers consisting of little more than leaderboards (those little tables with the systems on one side, the datasets on the other side, and your system's results in bold, or no paper) those are useless. 99% of research output in deep learning and neural nets is completely useless and never goes anywhere- because it lacks novelty, it is completely devoid of any theoretical results, and it is unreproducible even when the code is available.

For example, you mention studies on "question answering". Ca. 2018 Google published a paper where they reported that their BERT language model scored near-human performance on some question answering dataset without ever even having been trained on question answering. A scientific miracle! Some boffins who clearly don't believe in miracles wondered why that would even be possible and dug a bit, and found that BERT was overfitting to surface statisical regularities of its dataset. They created a new test dataset devoid of such statistical regularities and BERT's performance went down the drain, until it hit rock bottom (a.k.a. "no better than chance"). So much for "semantic similarity" measured over word embeddings modelling meaning.

But this is exactly the kind of work that I say connectionists consistently ignore: nowhere will you find that subsequent language models were tested in the same way. You will instead find plenty of tests "demonstrating" the ability of language models to represent semantics, meaning, etc. It's all bullshit, self-delusion at best, conscious fabrications otherwise.

This is the paper (I'm not affiliated with it in any way):

Probing Neural Network Comprehension of Natural Language Arguments

https://aclanthology.org/P19-1459/

But this kind of work is thankless for the undertaking academics and most of us have more important things to do. So the criticism eventually dwindles and what remains is the bullshit, and the fabrications, and the fantasies, seeping into mainstream discourse and being repeated uncritically - by yourself, for example. I can't even summon the compassion to not blame you anymore. For all I know you're exactly one of those connectionists who don't even understand their work is not science anymore, but spectacle.

P.S. I am not blind to the change of tone in your recent comments and I'm really sorry to be so cranky in response, when I should be cordial in reciprocity, but I've really had enough of all this. Unscientific bullshit has permeated everything and oozed everywhere. Perhaps it's time for me to take a break from HN, because it really doesn't look like I can have an original, curious conversation on here anymore.

I understand that this discussion can become frustrating, especially when you see repetitive patterns in the discourse or feel like the nuances are not being sufficiently addressed. However, there are a few points I would like to clarify:

Semantics in word embeddings: While I agree that word embeddings cannot fully capture human-like semantic understanding, they do provide a mathematical representation that has proven useful in many NLP tasks. It's not that word embeddings "understand" semantics in the human sense, but they do capture certain aspects of meaning that are statistically derived from their use in the training corpus. This is not an unsubstantiated claim. It is empirically demonstrated in numerous tasks where semantic understanding is beneficial, like semantic similarity, word analogy, and other downstream tasks such as translation, sentiment analysis, text classification, etc.

Your point about BERT overfitting to statistical regularities of the dataset is well taken. Indeed, it exposes the limitations of the model and the need for careful design and evaluation of benchmarks. However, it's worth noting that a failure in one specific test doesn't invalidate the successes in other tasks. It simply highlights an area that needs improvement.

It's true that there's a flood of papers and not all of them have substantial novelty or impact. This is not a problem exclusive to deep learning or AI, but a broader issue in academia and scientific publishing. However, amidst the noise, there's also a lot of valuable work being done, with genuine advancements and novel approaches.

You mentioned that connectionists only improve their systems when someone rubs their face in it. This is essentially how scientific progress happens - through skepticism, criticism, and the relentless pursuit of truth. I would argue that the current era of deep learning research is no different. It's a messy, iterative process, with steps forward, backward, and sideways.

Furthermore, I believe it's crucial to remember that there's room for both connectionist and symbolic approaches in AI. It's not necessarily a matter of one being 'right' and the other 'wrong.' Rather, they offer different perspectives and techniques that can be valuable in different contexts. Connectionist models, like the neural networks we've been discussing, are incredibly effective at tasks like pattern recognition and prediction, especially when dealing with large, high-dimensional datasets. On the other hand, symbolic models are excellent at representing explicit knowledge and reasoning logically, making them useful for tasks that require a high degree of interpretability or strict adherence to predefined rules. The future of AI likely involves finding ways to integrate these two approaches, leveraging the strengths of each to overcome their respective limitations. The field is vast and diverse, and there's plenty of room for different methods and viewpoints.

PS: I understand where you're coming from. Sometimes I need a break from this too. Remember there is no malicious intent here when people are just sharing their views.

Your disagreement seems to be a philosophical one. It is not a technical argument. It seems that you won't accept that semantics can be modelled by an unconscious mechanical system. I am talking about mathematical concepts of semantics, not "true" human semantics that are the product of human insight and consciousness. https://en.wikipedia.org/wiki/Semantic_similarity

While AI doesn't have an innate understanding of the world as humans do, the semantic representations it learns from vast amounts of text data can be surprisingly rich and detailed. It can capture associations and nuances that are not immediately apparent from a purely syntactic analysis of the text.

Oh come on. "Semantic similarity" is just heuristic bullshit. It's not a scientific term, or even a mathematical concept. Don't try to pull rank on me without even knowing who I am or what I do just because you can read wikipedia.

And note you're still not saying "why" or "how", only repeating the "what" of someone else's claim.

I understand your skepticism, and I acknowledge that the concept of semantic similarity is indeed an approximation. However, it is an approximation that has proven highly useful in a wide range of practical applications.

Semantic similarity methods are based on the idea that the meaning of a word can be inferred from its context, which is a concept known as distributional semantics. In essence, words that occur in similar contexts tend to have similar meanings. This is not just a heuristic, it's a well-established principle in linguistics, known as the distributional hypothesis.

In the case of large language models, they are trained on vast amounts of text data and learn to predict the next word in a sentence given the previous words. Through this process, they learn to represent words as high-dimensional vectors (word embeddings) that capture many aspects of their meaning, including their semantic similarity to other words.

These models can generate coherent text, answer questions, translate languages, and perform other language-related tasks with a high degree of accuracy. These capabilities wouldn't be possible if the models were only capturing syntax and not semantics.

The 'why' is because these models learn from the statistical regularities in their training data, which encode both syntactic and semantic information. The 'how' is through the use of deep learning algorithms and architectures like transformers, which allow the models to capture complex patterns and relationships in the data.

I hope this provides a more detailed explanation of my argument. I'm not trying to 'pull rank', but simply explaining the basis for my claims. I understand this is a complex topic, and I appreciate your challenging questions as they help clarify and deepen the discussion.

> GPT-4 is better at reasoning than 90% of humans. At least.

This makes as much sense as me asserting a binary search can tell one why they are looking for what they want.

That is to say, it cannot.

>GPT-4 is better at reasoning than 90% of humans.

Even for obviously nonsens that 90% of humans would recognice as such.

I intuit the same given the jumps we've seen from gpt 3.5 fo 4. The next jump will feature considerably more general intelegence than any one human being, though I think it will have certain weaknesses.

Looks more like a Chinese Room to me.

Everything is a Chinese room if you expect to see reified comprehension inside (and, naturally, don't find it).

While impressive GPT-4 isn't intelligent or close to intelligence. It's an impressive statistical model but doesn't have the ability to reason

Do you have any example of how it cannot reason?

It's a neural network. Neural network are not symbolic AI and are not designed to reason

There's a decent working paper that has benchmarks on this, if you're interested.

There are many types of reasoning, but GPT-4 gets 97% on casual discovery, and 92% on counterfactuals (only 6% off from human, btw) with 86% on actual causality benchmarks.

I'm not sure yet if the question is correct, or even appropriate/achievable to what many may want to ask (i.e. what 'the public's is interested in is typically lost after it is defined in any given study); however this is one of the best works available to address this problem I've seen so far, so perhaps it can help.

Percent of what? Possible right or wrong answers to a test?

Remember that GPT is not trained on all possible text. It's trained on text that was written intentionally. What percentage of that text contains "correct" instances of causal discovery, counterfactuals, etc.?

so can we make an estimate of GPT-4's IQ?

EDIT: Seems so...

https://duckduckgo.com/?q=ESTIMATE+OF+GPT-4%27S+IQ&t=opera&i...

shows articles with GPT IQ from 114 to 130. Change is coming for humans.

It doesn't matter what it is designed for. What matters is what it actually does. It's trivial to get GPT to do chain-of-thought reasoning and observe it.

Its trivial to get ChatGPT to reason about things:

Me: A room contains a blue box and a red box. Alice stands on the red box. Bob also stands on a box. No box has multiple people standing on it. Which box is Bob stood on?

ChatGPT: There are two boxes in the room: a blue box and a red box. Alice is standing on the red box, so Bob must be standing on the blue box. Therefore, Bob is standing on the blue box.

Me: Add a green box to the room. Which box is bob standing on?

ChatGPT: If a green box is added to the room, then there are now three boxes: a blue box, a red box, and a green box. Since Alice is still standing on the red box, we know that Bob must be standing on either the blue box or the green box. We cannot determine which one Bob is standing on without additional information.

Its not designed to reason but it does reason. Thats whats so impressive about it; emmergent properties like factuality and reasoning.

I watched a magician's show where he made his assistant hover in mid-air. It is evident that the magician has mastered levitation.

Evolution created intelligence without even being intelligent itself

How do you know that’s true?

Evolution is not a single orchestrator. It is merely the natural result of a simple mechanical process over a timescale that exceeds the lifetime of the common human.

Because we eat and breath through the same tube

There are two tubes.

Evolution came up with the shared eating/breathing tube design because it made sense for aquatic animals (from which we evolved).

The trachea and oesophagus both start in the common space of the pharynx. The epiglottis stops food going down the wrong one

we don't. which is why we choke and hiccup

Evolution is just a description of a process, it isn't a tangible thing.

You're proposing that statistics is intelligent?

You're proposing that evolution is statistics?

Basically, yeah.

Notable that the people who saw the first automatons argued whether they were alive.

People also argued that animals are mere automatons, that all their observable behavior is basically scripted, and that humans shouldn't empathize with them on those grounds. That's how vivisection etc used to be justified since at least Descartes.

Because evolution is random change + natural selection

Starting a fire is easy to do even if you don't know how it works.

Appropriate analogy in more ways than one. Particularly given the unpredictable consequences of doing so.

Yep, we don't know all constituents of buttermilk, nor how bread stales (there's too much going on inside). But it doesn't prevent us to judge their usefulness.

Imitation -> emulation -> duplication -> revolution is a very common pattern in nature, society, and business. Aka “fake it til you make it”.

Think of business / artistic / cultural leaders nurturing protégés despite not totally understanding why they’re successful.

Of course those protégés have agency and drive, so maybe not a perfect analogy. But I’m going to stand by the point intuitively even if a better example escapes me.

Not that weird if you think about it, our intelligence simultaneously measly and amazing as it is, was the product of trial, error, and sheer dumb luck. We could think of ourselves as monkeys with typewriters, eventually we'll get it right.

The battery (Voltaic Pile, 1800) and the telegraph (1830s-1840s) were both invented before the electron was discovered (1897).

Also Darwin published a theory of evolution, and Mendel discovered genetics, before anyone even thought of the term "double helix".

No need for genetics to understand evolution...

But they "didn't know how it worked", DNA was not discovered yet

Hell, consider all the technology that was developed before science really got going.

No need to know about electrons to understand electricity.

No matter what it is probably easier to inspect a language model while it works, than the language module of a human while he speaks.

We are probably creating something that looks like our intelligence but it works in a different way.

An example: we are not very good at creating flight, the one birds do and humans always regarded as flight, and yet we fly across half the globe in one day.

Going up three meters and landing on a branch is a different matter.

We don’t know…

I feel like OAI's approach is kind of wrong. GPT4 is still just text transformation/completion with multi-headed attention for better prediction of the next word that should follow (versus only looking at the previous word).

In human brains, language is only a way to communicate thoughts in concept form, though we also seem to use language to communicate abstract thoughts to ourselves to break them apart/down in a way (imo).

I'd love to see someone train a model on the level of GPT4 to generate abstract thoughts/ideas based on input/context and then pair this model with GPT4 co-operatively and continue to train, such that the flow of abstract ideas is parsed by GPT. But like...how do you even train a model that operates on abstract ideas, there doesn't seem to be any way to do this.

I like the idea. Note that LLMs have some skill at decoding sequential dense vectors in the human brain

https://pub.towardsai.net/ais-mind-reading-revolution-how-gp...

so why not have them decode sequential dense vectors of their own activations?

As for the majority scoring poorly, they suggest that most neurons won't have clear activation semantics so that is intrinsic to the task and you'd have to move to "decoding the semantics of neurons that fire as a group"

I don't think this is showing LLMs performing decoding. They're just using the LLM to propose possible words. The decoding is done by using another model to score how well a proposed word matches brain activity, and using that score to select a most likely sequence given the proposals from the LLM.

We know that complex arrangements of neurons are triggered based on input and generating output that appears to have some intelligence to many humans.

The more interesting question is why are intelligence/beauty/consciousness emergent properties that exist in our minds.

Nature created humans to understand nature. We created GPT4 to understand ourselves.

A mirror of ourselves*

That's beautiful until you think about it.

Humans so far have done a great job at destroying nature faster than any other kind could.

And GPT4 was created for profit.

Humans are the universe asking who made it.

Yes, or put a bit more elegantly, 'The cosmos is within us. We are made of star-stuff. We are a way for the universe to know itself.' — Carl Sagan

Yes, "all serve "gpt4!" Get a grip, pretty please.

There is no evidence that intelligence runs on neurons. Yes, there are neurons in brains, but there's also lots of other stuff in there too. And there are creatures that exhibit intelligent properties even though they have hardly any neurons at all. (An individual ant has only something like 250000 neurons, and yet they're the only creatures beside humans that managed to create a civilization.)

In what way a civilization?

I'll repost a comment via Reddit that I think makes this case [0]:

Ants have developed architecture, with plumbing, ventilation, nurseries for rearing the young, and paved thoroughfares. Ants practice agriculture, including animal husbandry. Ants have social stratification that differs from but is comparable to that of human cultures, with division of labor into worker, soldier, and other specialties that do not have a clear human analogy.

Ants enslave other ants. Ants interactively teach other ants, something few other animals do, among them humans. Ants have built "supercolonies" dwarfing any human city, stretching over 5,000 km in one place. And ants too have a complex culture of sorts, including rich languages based on pheromones.

Despite the radically different nature of our two civilizations, it is undeniable from an objective standpoint that this level of society has been achieved by ants.

[0]: https://www.reddit.com/r/unpopularopinion/comments/t2h1vs/an...

To be honest, this description is leaning heavily on the associations we have with individual words used. Ant "architecture" isn't like our architecture. Ant "plumbing" and "ventilation" have little in common with the kind of plumbing and ventilation we use in buildings. "Nurseries", "rearing the young", that's just stretching the analogy to the point of breaking. "Agriculture", "animal husbandry" - I don't even know how to comment on that. "Social stratification" is literally a chemical feedback loop - ant larvae can be influenced by certain pheromones to develop into different types of ants, which happen to emit pheromones suppressing development of larvae into more ants of that type. Etc.

I could go on and on. Point being, analogies are fun and sometimes illuminating, but they're just that. There's a vast difference in complexity between what ants do, and what humans do.

Nobody is saying that an ant might be the next Frank Lloyd Wright.

They're saying they accomplish incredible things for the size of their brain, which is absolutely and unequivocally true.

"Go to the ant, thou sluggard; consider her ways, and be wise".

What I'm saying is that, Bible parables notwithstanding, it's not the individual ant that achieves these incredible things. The bulk of computational/cognitive work is done by the colony as a system. This means that there's little sense in comparing brainpower of an ant with that of a human. A more informative comparison is that between an ant colony and human society - here, humans may come out badly, but that's arguably because our societies are overcomplicated in order to compensate for individual humans having too much brainpower :).

> There's a vast difference in complexity between what ants do, and what humans do.

Interesting parallell with intelligence/sentience/sapience. Despite the means, isn't the end result what you have to judge? The end result looks like a rudimentary civilization. How much back in time would we have to go back to find more sophistication in ant societies than humans?

> Despite the means, isn't the end result what you have to judge? The end result looks like a rudimentary civilization.

I'm indeed talking about the ends. The operative phrase here is "it looks like". It looks like a civilization, there are even some structural similarities to be found - but it is not a civilization, anymore than a Lego brick with a computer sticker on it is a PC.

This is not to deny that ants are awesome and very much worth studying - my point is to avoid following flawed analogies too far. "Ant colonies look like rudimentary civilization, and civilization is/does X, therefore..." is going to be a nonsense statements for most of possible values of X.

You could say the same in reverse. Humans can't lift fifty times their own weight. Humans can't communicate in real-time with pheromones alone. Most humans do not know how to build their own home. An ant might well consider us backwards, not advanced.

> Humans can't lift fifty times their own weight.

Ants can do it as a consequence of their size. Relative lifting strength drops fast with increased size. Conversely, an ant scaled to human size would collapse under its own weight and cook itself to death - waste heat generation scales with volume (~ size³), while waste heat rejection scales with surface area (~ size²).

And it ain't a cognitive achievement anyway.

> Humans can't communicate in real-time with pheromones alone.

Yes. Because it's not as useful at our scale, nor is it real-time - chemical communication works better for small organisms and small volumes of living, as the travel speed and dissipation rate of pheromones is independent of organisms emitting them. Meanwhile, we have multiple ways of communicating real-time, some of which work at light speed which is the definition of "real time" (light speed in vacuum being the speed of causality itself).

> Most humans do not know how to build their own home.

Neither do ants.

> An ant might well consider us backwards, not advanced.

An ant can't consider us anything. The point I'm trying to get across is, just because the ant colony is capable of surprisingly high sophistication, doesn't mean the individual ants are.

As a counterpoint that's actually cognitive in nature: AFAIK individuals of many (most?) ant species can be easily tricked into following each other in a circle, and they will continue to do so until they start dying from starvation. This is because the ant isn't making a decision following a complex thought process - it's executing a simple algorithm, that works well in nature because nature is chaotic enough that stable ant circles are unlikely to form (and when they do, they're unlikely to be left undisturbed for long).

> "Agriculture", "animal husbandry" - I don't even know how to comment on that.

To give some examples of ant agriculture and animal husbandry, they cut and feed leaves to a fungus they farm for food. The fungus even communicates to the ants to switch to a different plant when the plant produces toxins in defense to cutting.

Ants herd aphids, protecting them from predators (ladybugs), secrete pheromones that direct and pacify them, and massage their abdomens to milk sugar dew.

I think you can call ant societies civilizations, but the same time you can call a multi cellular organism a civilization, too. Usually, those also come from the same genetic seed similar to (most) ant colonies. But more importantly, you have various types of cooperation and specialization in multi cellular life. Airways are "ventillation", chitin using or keratinated tissues are "architecture", and there is even "animal husbandry" in the form of bacterial colonies living in organs.

Do termites count as well then? Bees? I'm not being facetious, but these insects seem similar.

Perhaps an important point here is that ants pass the mirror test, which, to my knowledge, no other insects (that we know of) do.

If you really want to present ants as a civilization, I don't think a single ant is a meaningful unit of that civilization comparable to a single human. A colony, perhaps - but then that's a lot more neurons, just distributed.

A single human will die permanently, alone, miserable, and unproductive. The comparison is apt.

A single human can survive on their own; there are many historical examples of that. A detached body part, on the other hand, cannot; but it also cannot feel miserable etc. A single ant is more like a body part of the colony in that sense.

If by "survive" you mean "age and die leaving nothing behind", then sure.

But the same is true for an ant.

No, it is not true for an ant. A solitary worker ant will not die from old age. Many species will literally starve to death within a few days because they cannot properly digest their food without trophallaxis.

You're right, it seems ants need each other to digest food, though scientists don't yet know why exactly.

From the New Yorker [1]:

> The researchers aren’t yet sure exactly what causes the digestive slowdown. Maybe it’s stress. Maybe the fluid that ants exchange when they share food stimulates digestion. Maybe they die still looking for a relative to break bread with.

[1] https://www.newyorker.com/tech/annals-of-technology/lonely-a...

This is not a good take. Yes there is a lot more going on in brains than just neuronal activity, we don’t understand most of it. But understanding neurons and their connections is necessary (but not sufficient) to understanding what we consider intelligence. Also, 250k is a lot of neurons! Individual ants, as well as fruit flies which have even fewer neurons, show behavior we may consider intelligent. Source: I am not a scientist, but I work in neuroscience research

What's the argument that understanding neurons is necessary?

Perhaps intelligence is like a black box input to our bodies (call it the "soul", even though this isn't testable and therefore not a hypothesis). The mind therefore wouldn't play any more of a role in intelligence than the eye. And I'm not sure people would say the eye is necessary for understanding intelligence.

Now, I'm not really in a position to argue for such a thing, even if I believe it, but I'm curious what argument you might have against it.

Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind.

The effect of drugs (including alcohol) on the mind. Of note is anesthesia which can reliably and reversibly stop internal experience in the mind.

For a non-physical soul to hold our mind we would expect significant divergence from the above. Out of body experiences and similar are indistinguishable from dreams/hallucinations when tested against external reality (remote viewing and the like).

“For a non-physical soul to hold our mind we would expect significant divergence from the above.”

This sounds like it assumes a physical mind could access a non-physical soul. All we probably know is that we have to be using an intact mind to use free will.

> Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind.

That's not a completely watertight argument.

Consider a traditional FM/AM radio. You can modify it, damage it, and get notable changes to its behaviour...

> Of note is anesthesia which can reliably and reversibly stop internal experience in the mind

...turn it off and on again...

> For a non-physical soul to hold our mind we would expect significant divergence from the above.

... yet concluding that all the noises produced from the radio are purely internal, mechanical and physical would be the wrong conclusion.

(I'm not arguing that the human brain/mind is anything like analogous to a radio, just pointing out the limits of this approach.)

I mean, if we're really going to go there, who's to say that a large enough LLM doesn't automatically receive a soul simply because that's one of the fundamental laws of the universe as decreed by the Creator?

Going where? I wasn't arguing for the existence of a soul.

Although, sure, if we could somehow manage to determine that souls did exist then presumably an AI model as capable as a human would also be eligible for one.

You can actually hypothesize that a soul exists and that intelligence is non-material, its just that your tests would quickly disprove that hypothesis - crude physical, mechanical modifications to the brain cause changes to intellect and character. If your hypothesis was correct you would not expect to see changes like that at all.

Some people think that neurons specifically aren't necessary for understanding intelligence but in the same way that understanding transistors isn't necessary to understand computers, that neurons comprise the units that more readily explain intelligence.

> You can actually hypothesize that a soul exists and that intelligence is non-material, its just that your tests would quickly disprove that hypothesis - crude physical, mechanical modifications to the brain cause changes to intellect and character. If your hypothesis was correct you would not expect to see changes like that at all.

That’s not necessarily a disproof. It’s also not necessarily reasonable to conflate what we call “the soul” with intelligence.

This is entering the world of philosophy, metaphysics and religion and leaving the world of science.

The modern instinct is to simply call bullshit on anything which cannot be materially verified, which is in many ways a very wise thing to do. But it’s worth leaving a door open for weirdness, because apart from very limited kinds of mathematical truth (maybe), I think everything we’ve ever thought to be true has had deeper layers revealed to us than we could have previously imagined.

Consider the reported experience of people who’ve had strokes and lost their ability to speak, and then later regained that ability through therapy. They report experiencing their own thoughts and wanting to speak, but something goes wrong/they can’t translate that into a physical manifestation of their inner state.

Emotional regulation, personality, memory, processing speed, etc… are those really that different from speech? Are they really the essence of who we are, or are they a bridge to the physical world manifest within our bodies?

We can’t reverse most brain damage, so it’s usually not possible to ask a person what their experience of a damaged state is like in comparison to an improved state. We do have a rough, strange kind of comparison in thinking about our younger selves, though. We were all previously pre memory, drooling, poorly regulated babies (and before that, fetuses with no real perception at all). Is it right to say you didn’t have a soul when you were 3 weeks? A year? Two years? When exactly does “you” begin? I can’t remember who I was when I was when I was 2 months with any clarity at all, and you could certainly put different babies in doctored videos and I wouldn’t be able to tell what was me/make up stories and I’d probably just absorb them. But I’m still me and am that 2 month old, much later in time. Whatever I’m experiencing has a weird kind of continuity. Is that encoded in the brain, even though I can’t remember it? Almost definitely, yeah. Is that all of what that experience of continuity is, and where that sense is coming from? I’ve got no idea. I certainly feel deeper. Remember that we all are not living in the real world, we’re all living in our conscious perception. The notion that we can see all of it within a conscious mirror is a pretty bold claim. We can see a lot of it and damage the bridge/poke ourselves in the eyes with icepicks and whatnot, and that does stuff, but what exactly is it doing? Can we really know?

Intuitively most people would say they were still themselves when they were babies despite the lack of physical development of the brain. Whatever is constructing that continuous experience of self is not memory, because that’s not always there, not intelligence, because that’s not always there, not personality, because that’s not always there… it’s weird.

I think it’s important to remember that. Whenever people think they have human beings fully figured out down to the last mechanical detail and have sufficient understanding to declare who does and doesn’t have a soul and what that means in physical terms, bad things tend to happen. And that goes beyond a statement to be cautious about this kind of stuff purely out of moral hazard; the continual hazard is always as empirical as it is moral. We can never really know what we are. Our perceptual limitations may prove assumptions we make about what we are to be terribly, terribly wrong, despite what seems like certainty.

I’m here playing devil’s advocate - this test doesn’t work. Here are some related thought experiments.

Suppose a soul is an immaterial source of intelligence, but it controls the body via machine-like material hardware such as neurons.

Or an alternative, suppose there is a soul inside your body “watching” the sensory activations within your brain like a movie. The brain and body create the movie & have some intelligence, but other important properties of the consciousness are bound to this observer entity.

In both these cases, the test just shows that if you damage the hardware, you can no longer observe intelligence because you’ve broken the end-to-end flow of the machine.

Its fine if you are playing or supposing in seriousness but with good humor, it doesn't really change how anyone else should interact with you :)

But yes, supposing that then you would expect to only see damages that correspond such as different forms of paralysis or other purely mechanical damages, not things that change the interior perspective.

Otherwise you start postulating the existence of a thing whose sole justification is your desire for the existence of that thing, which is natural when you start questioning beliefs and kick out all the supports without meaning to.

I think this is what Bertrand Russel's teapot was meant to ellucidate.

Why would you doubt neurons play a roll in intelligence when we've seen so much success in emulating human intelligence with artificial neural networks? It might have been an interesting argument 20 years ago. It's just silly now.

> It might have been an interesting argument 20 years ago. It’s just silly now.

Is it?

These networks are capable of copying something, yes. Do we have a good understanding of what that is?

Not really, no. At least I don’t. I’m sure lots of people have a much better understanding than I do, but I think its hard to know exactly whats going on.

People dismiss the stochastic parrot argument because of how impressive big neural nets are, but it doesn’t really invalidate that argument. Is a very, very, very good parrot that learns from everyone at once doing basically the same as what we do? I’d argue no, at least not fully. It’s absorbed aspects of us extremely well/is a very weird, sophisticated mirror, yes, and is copying something somehow, probably in a way reminiscent of how we copy. Is it basically the same as what we’re doing when we think? Partially? Fully? Not at all?

A typical engineer would say “good enough”. That type of response is valuable in a lot of contexts, but I think the willingness to apply it to these models is pretty reckless, even if it’s impossible to easily “prove” why.

To be clear on the exact statement you made, I think you’re right/it’s pretty clear neurons play some very important role/potentially capture a ton of what we consider intelligence, but I don’t think anyone really knows what exactly is being captured/what amount of thought and experience they’re responsible for.

That person's argument is borderline insane to me - a severe lack of knowing what is unknown, a reverence of current best-models (regards modern science, including neurology - yet, open minded investigations beyond are also a requisite here.) And the pompousness is what truly boggles my mind ("Its silly to believe this, now.) A look in the mirror would suffice to say the least...

Anyway, thank you for a great answer and conversation throughout this thread.

Regards neural networks, parroting and the emulation of intelligence (or the difference between an emulation and the "real thing"):

Well, somewhat like you say, we cannot propose a valid comparison from one to the other without an understanding of one (consciousness) or both. It's fascinating that there are some open, valid and pressing questions about what / how the output of this new wave of software is concretized (from foundational, semi-statistical algorithms in this case.)

Yes, I do agree neurons have something to do with the "final output". But this is a subsection of the problem - organic neurons is-an/are order(s) of magnitude in complexity beyond what the tricky "parrot" is up to. Moreso, these components perform very different functionally - the known functions of the neuron compared to ANN, backprop etc. The entire stack.)

P.S: One interesting theory I like to simulate and/or entertain is that every organic cell in the body has something to do with the final output of consciousness.

Thank you, appreciate the compliment.

And yeah, are definitely a lot of open questions related to all of this. Love how its brought so many deep questions into focus.

Your theory makes sense in an evolutionary context. It is possible that all cells and organisms have some general intelligence. Humans do not have the ability to recognize this because evolutionarily it was only helpful to recognize intelligence when it could pose a threat to us. And the biggest threat to us in general was other humans as we are tribal animals. So we don't see it, we only see specialized intelligence that historically posed a threat to us.

It would explain why most "experts" didn't see GTP-4's abilities coming. Many of them expected that it would take a major algorithm or technology improvement to do "real intelligent" things, because they fundamentally misunderstood intelligence.

Please read the comment I was responding to. I was addressing the suggestion that perhaps the brain is as relevant to intelligence as the eye. Cognitive Neuroscience has been a thriving field for nearly half a century now. I didn't say we have it all figured out, just that it's obvious neurons are a piece of the puzzle.

If anything the experience with artificial neural networks argues the opposite - biological neurons are quite a bit different than the "neurons" of ANNs, and backpropagation is not something that exists biologically.

The other comments have pretty much covered it. We can pretty clearly demonstrate that neurons in general are important to behavior (brain damage, etc) and we even have some understanding about specific neurons or populations/circuits of neurons and their relation to specific behaviors (Grid cells are a cool example). And this work is all ongoing, but we're also starting to relate the connectivity of networks of neurons to their function and role in information processing. Recently the first full connectome of a larval fruit fly was published - stay tuned for the first full adult connectome from our lab ;)

Again, IANA neuroscientist, but this is my understanding from the literature and conversations with the scientists I work with.

What else would intelligence run on?

Every cell in our body, and every bacterium living in a body (e.g. gut flora), contribute to our intelligence. It looks plausible (to me) that there's one "top cell" among them that represents the "person", others just contributing via layered signals, but whether this "top cell" is a neuron or another kind of cell is unknown.

Ah, so you're not studied in neuroscience with some revolutionary theory. A shame.

Maybe the neurons are the hardware layer. The software is represented by the electronic activity. There is a good video https://youtu.be/XheAMrS8Q1c about this topic.

> There is no evidence that intelligence runs on neurons.

1. Neurons connect all our senses and all our muscles.

2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end.

Don’t underestimate what a neuron can do. A single artificial neuron can fit a logistic regression model. A quarter of a million is on then scale of some our our largest AI, and biological neurons are far more connected than ANN. An ant quite likely has a more powerful brain than GPT-4.

> 2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end.

Our digestive systems appear to be important to our behaviour, though. Some recent work in mice showed that if colonised with bacteria from faeces of humans with autism, the mice would begin to show autistic behaviours.

So, not sure your argument here is especially strong.

The role that digestive systems potentially play for our behavior is via neurotransmitters, so in the end it is the neurons, that are responsible.

If the study you mention show something else, I would actually be very interested in a source.

There is no evidence that any of those are emergent properties. It’s no more or less logical than asserting they were placed there by a creator.

There is evidence that people believe that GTP-4 is intelligent since it can solve things like the SATs. But if you start taking away weights one by one at some point those same people will say it isn't intelligent. A NN with 3 weights cannot solve any problems that humans believe requires intelligence. So where did it come from? I don't know, but it clearly emerged as the NN got bigger.

It's possible that the comment you're responding to is referring to the new paper out of Stanford/Google on emergence as a 'mirage'. The gist of it is that the shape of the curve for 'metric' vs 'params' is dependent upon the type of metric. For classification it has a certain shape, for accuracy it has another, etc. The authors claim the observation that these curves are determined by the choice of metric means that it isn't true emergence, but rather is due to the way in which GPTs are typically trained to predict the next token (and then beam search or typical sampling is often applied afterwards, etc).

The paper is somewhat new so I haven't done a proper review to know if it's solid work yet, but this may offer some context for some of the comments in this thread.

I suspect that there's a sweet spot that combines a collection of several "neurons" and a human-readable explanation given a certain kind of prompt. However, this "three-body problem" will probably need some serious analytical capability to understand at scale

here's a clue, start your research

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).

My favorite line: “Our overall procedure is quite compute intensive.”

Should that be measured in number of nuclear power plants needed to run the computation? Or like, fractions of a small star’s output?

Yes but they are from openai so they can just write papers that say whatever they want to say without minding the metrics and then pretend like it is some kind of science.

> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to.

Exactly. Especially:

> ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles;

This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks.

It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

Like say, in court to detect if someone is lying? Or at an airport to detect drugs?

You don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc.

But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.

>Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc.

Evolution is still doing it's thing.

What’s the risk? Someone allergic to peanuts will eat peanuts because ChatGPT put it in their diet plan? That’s silly.

Yes, that's the risk, and people are literally doing that because "if it put them in the recipe, it knows that quantity is safe for me", or "I asked it if it's okay and it cited a study saying that it is".

Those cases sound like Darwin Awards mediated by high technology

I hear crushed glass has very great health benefits

> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

Doesn't this criticism also apply to people to some extent? We don't know what the purpose of individual brain neurons is.

People are better understood intuitively. We understand how people fail and why. We can build trust with people with some degree of success. But machine models are new and can fail in unpredictable ways. They also get deployed to billions of users in a way that humans do not, and deployed in applications that humans do not. So its certainly useful to try to explain neural networks in as great of detail as we can.

Or we can build trust using black box methods like we do with humans, e.g., extrapolating from past behavior, administering tests, and the like.

We can, but the nice thing about neural networks is the ability to do all kinds of computational and mathematical manipulations to them to basically pick them apart and really find out what’s going on. This is important not just for safe deployment but also for research on new methods that could be used to make them better. Plus we need this ability to help avoid neural networks with intentionally hidden features that appear to behave linearly in certain regimes but are designed with a strong nonlinear response when special inputs are applied. You could have all the tests you want for a self driving car based on real world conditions but some bad actor with access to the training system could create a special input that results in dangerous behavior.

The more fundamental problem is the sheer size of them, and this is only going to get worse as models grow larger to become more capable. Being able to look at the state of individual neurons during inference is very convenient, but that does not by itself make it possible to really find out what's going on.

As a person I can at least tell you what I do and don't understand about something, ask questions to improve/correct my understanding, and truthfully explain my perspective and reasoning.

The machine model is not only a black box, but one incapable of understanding anything about its input, "thought process", or output. It will blindly spit out a response based on its training data and weights, without knowing the difference whether it true or false, meaningful or complete gibberish.

As a person, you can tell what you think you do and don't understand, and you can explain what you think your reasoning is. In practice, people get both wrong all the time. People aren't always truthful about it, either, and there's no reliable way to tell if they are.

Is it really fair to say this brings us “no closer” to explainability?

This seems like a novel approach to try to tackle the scale of the problem. Just because the earliest results aren’t great doesn’t mean it’s not a fruitful path to travel.

> the explainability situation that has been missing for decades in neural networks.

Is this true? I thought explainability for things like DNNs for vision made pretty good progress in the last decade.

Reply on news.ycombinator.com