Hacker News Reader

Comment by tracerbulletx | Hacker News Reader

All of these arguments seem to think that the brain isn't also generating a statistical ordering of semantic words and actions. Obviously the brain has a way more complex network of subsystems with various specialties than the artificial models we have today and is able to generalize language, math, and positional reasoning and mix it with the older parts of the brain for reward and training mechanisms and it can do it in real time, but it kind of seems like modern models are showing so much flexibility at different tasks and so much generality it's feeling like they have something in common with the building blocks of the brain that might get us there.

View on HN

And even if something capable of all the aspects of human cognition were created with no internal resemblance to cognition, so what, it still works. There's a reason it's called "artificial" intelligence, not "natural intelligence identical to human brains except in silicon instead of neurons". It's like when the wheel was invented saying it's not "true" transportation because it doesn't involve legs, or when the phone wasn't invented that it isn't "true" communication because you don't see the other person's mouth or physical expressions.

I agree that you shouldn't dismiss an airplane by the fact that it doesn't flap its wings, but we also don't call them artificial birds. The debate is around the word 'intelligence', which is misleading.

Personally, I prefer almond milk in my cereal

But it’s not called artificial humanity, it’s called artificial intelligence. So what you’re saying is like saying planes don’t fly

I guess there is also a point which is that it isn't 'artificial' intelligence - it is just intelligence.

Like we don't say that planes artificially fly, they just fly. I mean technically it's non-natural flight, but it's interesting to think about.

Yes, there is no artificial intelligence because it's just intelligence.

Whether it was generated from "natural" or "artificial" means (a distinction which quickly breaks down anyway) the intelligence is just intelligence.

Semantics

Go and Google "artificial birds" and report back.

Look up the definition of "artificial" while you're at it.

No, he's too busy making genuinely useful comments. Why don't you try it?

The human brain has always been compared to our most advanced technologies. It used to be compared to the telephone, then digital computers, and now to deep learning models. At least deep learning model has some merit because artificial neural networks are in part inspired by a very simple model of how our brain neurons work in a network, or rather how neuropsychologists would explain reinforcement learning to a computer scientist.

However this is a great simplification, and borders on an absurd reduction. You can model our brains using linear algebra, however that doesn’t mean our brains are linear algebra computer. There is a whole lot more going on than neurons receiving feedbacks from other neurons which adjusts the weight for subsequent firing. A lot of our behavior is actually inherited (I know I spent a whole week here on HN arguing with IQ advocates on the nuance of that statement), neurochemicals and hormones add a whole another level of statefulness not seen in artificial neural networks, the brains ability to make computations is actually pretty limited (especially next to a GPU). I mean, cordiseps exists, meaning a fungus can infect an organic system and control its behavior, there is 100% chance that some yet to be discovered viral and bacterial agents, not sharing any of our DNA—and certainly not “connected” to the “weight matrix”—are also influencing our behavior (just not as dramatically), and there is 100% chance they interact with our DNA also controlling our “innate” behavior.

What is going on in our brains can only be modeled using statistical ordering of semantic words and actions. The real world brain is always going to be infinitely more complicated than this model.

We just can’t accept that we might solve ourselves. People are understandably desperate to understand their experiences as more than an encoding of a thing that might be explained.

And all of our surprising wins and awful mistakes had explainable reasons, dammit; it wasn’t just a misfiring of trained statistical networks!

Or we just don't know how brains work very well and we shouldn't act like we do.

Not that I disagree. It just seems like some of society’s reaction is to slow down on researching the very question.

It to do with safety concerns and economic stability. Not because people are idiots.

Not knowing how something works doesn't necessarily preclude replicating its function a completely different way. You don't have to understand an induction cooktop to hang a pot of water over a campfire.

In addition, we shouldn't draw false equivalence between not knowing how brains work and not knowing how LLMs work, and concluding they must be similar.

there are currently two things we are aware of in the universe that can reason abstractly. i don't think this is a coincidence.

Only in a very anthropocentric sense. How do we know an ant colony doesn’t reason abstractly (or a human town for that matter)? What about slime mold or ameba? Both can solve a maze as well as humans. What makes you think a forest ecosystem isn’t capable of abstract though?

It is only if we narrow thought to mean precisely human-like thought when humans and human creations are uniquely capable of something. To that extent, our models of intelligence is very much in the pre-copernican era.

Isn't reasoning abstractly a spectrum though? I took the parent to mean "reason abstractly to the same degree humans do".

Slime molds and amebas might be able to reason abstractly to some degree but they can't write code or poetry.

Like I said, only in an anthropocentric sense.

How do you know fungal networks don’t write and read poetry under the forest floor? If they do—and we have no reason to doubt that they do—you wouldn’t be able to read them, let alone understand them.

The earth’s biosphere as a whole also writes code, just in DNA as opposed to on silicon transistors, why exclude the earth’s biosphere from things capable of abstract thought?

> Only in a very anthropocentric sense.

yes, that is the sense in which we are discussing intelligence in order to debate whether the human brain and LLMs operate on similar phenomena

The fact that both humans and LLMs can both reason abstractly is an uninteresting fact if we define “abstract reasoning” to be exactly what humans do, and then create models with the goal of recreating exactly that. This is than simply a statement of an accurate model, and the word intelligence is there only to confuse.

This would be like finding a flower which produces a unique fragrance, then create a perfume which approaches the same fragrance and then conclude that since these are the only two things in the universe which can create this fragrance there must be something special about that perfume.

i would define abstract reasoning as composing and manipulating a model of reality or other complex system in order to make predictions

> is an uninteresting fact if we define “abstract reasoning” to be exactly what humans do, and then create models with the goal of recreating exactly that

if you find this uninteresting, we have perhaps an irreconcilably differing view of things

Your definition excludes language models, as they are in and of them selves just a model which interpolates from data (i.e. makes predictions). But your definition also includes lots of other systems, most mammalian brains construct some kind of models of reality in order to make predictions. And we have no idea whether other systems (such as fungal networks or ant colonies) do that.

I’m not saying these language models—or my hypothetical perfume—aren’t an amazing feat of technology, however neither has any deep philosophical implications about shared properties other than the ones constructed to do so. Meaning, even if LLMs and humans are the only two things in the universe that can reason abstractly in the same way humans do, that doesn’t mean the latter has any more properties shared with the former.

More than 2.

For an example, crows can effectively use tools and communicate abstract concepts to one another from a memory. Which means they can observe a situation, draw conclusions, and use those conclusions to act as well as make decisions on how to act. That would seem to meet the bar for reasoning abstractly.

Also, it's always funny to notice how the brain, throughout history, has always been compared to the latest technology available. For a long time people said "the brain must be similar to a clock".

Wouldn’t you say that this is how evolution and the spread of information works, in a system that has the spark of some life chemistry?

Of course… The hole in that theory is that evolution never found the wheel.

The steel man in that theory is that it invented the neurological and social processes that then went on to invent the wheel. And the platypus and the clap.

Edit: I forgot to bring it back around and make a point ;)

I’m saying that humans invented clocks and CPUs. We only have metaphors that have emerged from the still misunderstood ether of the informatic universe.

Well, we're observing similarities between them, but people insist they are 100% different in their way of working.

> We just can’t accept that we might solve ourselves.

To solve ourselves is to know ourselves completely, and to know ourselves completely is to be honest in who and why we are what we are simultaneously across all persons. It assumes perfect knowledge.

There is no statistical approximation nor computational power which can do this.

> People are understandably desperate to understand their experiences as more than an encoding of a thing that might be explained.

Another way to frame this is, "some people are nihilists and do not see life as more than an encoding of a thing that might be explained."

That's been solved long ago.

To know anything (an X) completely you need perfect knowledge. Hence people come up with a set of simplified ways of reasoning about X. They call this a model of X.

Model is incomplete and so primitive, so dumbed down, that we manage to play it forward/backward in our heads or our computers.

If a model checks out with the real outcomes we proudly exclaim that we understand X.

I'm not being sarcastic, that is just a real method we use all the time.

That only works in math world, in the real world we have learned that entropy considerations mean there is no such thing as perfect knowledge.

> To know anything (an X) completely you need perfect knowledge.

Agreed.

> Hence people come up with a set of simplified ways of reasoning about X. They call this a model of X.

A "set of simplified ways of reasoning about X" in order to create a "model of X" does not imply complete understanding. Quite the contrary actually.

To wit, science often models current understanding of a phenomenon. When new evidence (understanding) is discovered, the model is updated to account for it. Sometimes this invalidates the original model, often the model is refined. Either way, progress is made with the tacit agreement that the model may change in the future.

> If a model checks out with the real outcomes we proudly exclaim that we understand X.

Again, this does not support the assertion of "That's been solved long ago." If anything, it affirms there is justification for disagreeing with the original premise to which I responded:

> We just can’t accept that we might solve ourselves.

I think people look more desperate to hype these LLM toys, that they are not the next blockchain or self-driving car. When they fail it's just excuses like you are not using the latest version or "prompting" them right.

The LLM value add for coding is less than the value add of syntax highlighting in my experience.

I think it's getting better.

In my experience and it's certainly useful for unit tests speeding things up for me pretty dramatically.

Also when working on a new development language was quick to point out "how you do that" with the inline vars I needed manipulated vs look ups in Google and copy pasting.

I find this quite similar to the issue of free will: If we live in a generally deterministic universe, where is the space for independent decisions of individuals? Was every single decision we take already predetermined before we were born? A lot of ink has been spent on this topic, and as far as I know, only a small minority of people actually deny free will. One assumption I have is to why is because it absolutely sucks from an emotional point of view, and it is a terrible idea to base fe. a justice system on.

Maybe it makes me weird, but the idea of being determined, but still having to follow those emotional and cognitive tasks to moral conclusions is some kind of “work” we still have to do, to get good outcomes.

That’s still the hole in my theory of consciousness. I admit it ;)

But it doesn’t give me as much cognitive dissonance as others to believe that the process of performing moral actions still has to be “processed” by me, a processor. In some sense.

My understanding of how physics works is that due to the probabilistic nature of quantum physics, you can’t predict the future perfectly. So your decisions were not pre-ordained even though they were the result of physical processes happening in your brain.

Alternatively, we just can't accept that we might not solve ourselves. People are understandably desperate trying to find an explanation for everything, but can't admit that's just not ever going to happen.

> All of these arguments seem to think that the brain isn’t also generating a statistical ordering of semantic words and actions.

There may be a part of the brain that is modelled well by an LLM, but if so, there seem to also be parts that aren’t, or even existing “multimodal” models like GPT-4, which is more than an LLM.

Humans, including their brains (both in the narrow sense and in the broader sense of “everything that contributes to cognition”, which may extend beyond the brain proper) are machines, and their function probably will, someday, be mimicked by machines. But they are still more complex machines that modern “AI” systems

this is exactly what he said if you finish reading the comment

Agreed. One of the more interesting things about AI is how it forces humanity on a trip to consider what it means to be human. There is no soul, creativity or inspiration. There's only (very complex) agency.

And that's completely fine.

Or you can say the opposite and claim that there is soul, creativity, or inspiration in everything, even in inanimate objects (basically panpsychism, in contrast to materialism). But regardless of what kind of monism you take, the advent of current statistical AI models forces us to reconsider the Cartesian dichotomy between body and mind, which forms the basis of liberal thought.

> But regardless of what kind of monism you take, the advent of current statistical AI models forces us to reconsider the Cartesian dichotomy between body and mind, which forms the basis of liberal thought.

Didn't we dispose of Cartesian dualism a long time ago? It may have been useful for disposing of superstitions which preceded it, but anyone who has observed the effects of a stroke or has taken a psychotropics understands that mind and body are not made of separate stuff, and I know of no modern system of thought that stipulates that as an axiom.

LLMs just demonstrate that emergent information structures at the complexity of human language need not be only biological.

To me it shows how fundamental language is to intelligence.

Not just that LLMs are very capable by themselves, but significantly improve speech and image models.

We’ll need to break out of the Chomsky hierarchies and develop some new theories of language.

Vocal communication isn’t unique to humans, a new theory should be more broadly applicable to non human and non vocal languages.

I’m excited to see what big questions will be answered, and what new questions arise.

>We’ll need to break out of the Chomsky hierarchies and develop some new theories of language.

We might need to develop some "new theories of language", covering other areas, but why exactly do you think we'd need to "break out of the Chomsky hierarchies"?

Nothing about LLMs challenges Chomsky hierarchies.

They do challenge Chomsky's ideas about language being an innate human quality, but that's about it. Besides that's not related to the Chomsky hierarchy of language, e.g. regular, context-free and so on. Those will continue to remain a hierarchy describing languages based on capability levels, whether there's AI, and AGI, or not.

The Chomsky hierarchy will remain unchanged and very useful for constructed languages. I didn't mean break out, as in create a higher level. LLMs are not going to magically become more than a turing machine or anything. I meant we need to stop thinking about spoken language in terms of the hierarchy and try to find another approach.

You can't tell me with a straight face that actual spoken language fits well in that hierarchy. It seems we can shoehorn it in with a 100B parameter turing machine definition, but that just makes my point that the constructed language hierarchy is not well suited for describing spoken languages if it takes such a large definition.

We should be able to find something like a new probabilistic theory of language that will do a much better job of describing spoken languages. Such a theory could help explain some LLM behaviors we see, and apply generally to other forms of noisy communication.

> 100B parameter ... not well suited for describing spoken languages if it takes such a large definition.

Humans by contrast use about 700T synapses to support our own understanding of language.

> To me it shows how fundamental language is to intelligence.

I believe most human intelligence is based on language. We are smart because language gives us the keys to solving all sorts of situations based on the experience of prior generations. We are wondering if GPT architecture can generate real understanding, but forgetting the training corpus. The language corpus is the real magic here. The source and accumulation point of intelligence.

Take a baby and give it access to language, you get a modern human. Take a random init neural net, use language corpus for training, you get GPT4. It doesn't matter what the model is - brain or transformer, it doesn't matter if it's a transformer or just a RNN (RWKV). What matters is the training signal.

We should not get so hung up on the model architecture and substrate when thinking about LLM capabilities. Language is a separate evolutionary system of ideas and concepts. It is a self replicator, like DNA. Language just got a new pathway for self replication with LLMs. Every new token visits the whole model and by extension the distillation of our culture before being emitted.

You're just describing information theory. Every message is contextualized within the codebooks used to encode/decode it. Sometimes we call that codebook "language", sometimes it uses another representation.

Yes, we should develop a new theory of language based on an information theory approach rather than a discrete symbol approach, or maybe something involving both.

But you can’t say information theory as it stands gives us a completely satisfactory model of human language. Something will need to be built on top of it.

>There is no soul, creativity or inspiration. There's only (very complex) agency.

That's highly debatable. For starters, who said there's no creativity or inspiration in humans, or for that matter, that there can't be in a complex A.I.?

How we achieve that creativity or inspiration is irreleant, as long the entity (human or AI) showcases creativity and inspiration.

Nor is it much clear why all being "only (a very complex) agency" would preclude creativity and inspiration.

That's like a worse version of "a human can't be creative or have feelings because it's all a bunch of molecules".

There is such a thing as emergent properties.

>And that's completely fine.

That's also highly debatable. I mean, that it would be "completely fine" if you were right and there wasn't "creativity of inspiration".

That’s a weak explanation, because there doesn’t need to be a mind for any of that to be true.

There are deep philosophical implications, just not these.

There has been a long history of reifying the behaviors of living things in general (e.g. vitalism) and humans in particular (e.g. dualism).

The success of LLMs challenges a lot of philosophy dealing with what behaviors are and are not possible in the absence of these categorical districtions.

I have had casual debates years ago, in which strong dualists asserted that the kind of creativity exhibited by today's LLMs is simply impossible. No doubt those folks are busy inventing "special philosophical creativity" that LLMs "aren't really doing," but they've lost credibility.

LLMs have demonstrated that there was never any need to invoke categorical districtions between human behavior and math-as-implemented-by-physics. The gap is closed, there is no more room for gods.

We are the gods now. https://youtu.be/ygRNEy8mPjk

Or maybe we will create a god.

Yes they seem to take the view into account, but this sentiment as if the question is settled and we aren't anthropically-projecting, or whatever the term is, is premature. Just because two processes produce a superficially similar result, does not make them the same process. What you just described after conceding that though the brain may be a statistical inference machine it still seems like magic.

It is telling that you talk about all those things happening in "real time." Ask any super-regarded philosopher, from Plato to Wittgenstein (yes I'm excluding Dennet et al), and that would be quite the hoot to point out.

Correlation is not causation. It feels more like smarter automation to me.

> It feels more like smarter automation to me.

Yep. Elevators used to be proof that machines can think. Then compilers, and chess, and go, and search, and …

The problem with AI is that as soon as it works, we stop thinking about it as “artificial intelligence” and it becomes “just automation”. Then AI moves to the next goalpost.

The problem seems to be with defining good tests for intelligence. FWIW, because GPT4 answers have a detectable pattern, they should presumably fail the Turing test.

At some level, intelligence requires logic, rationality, and conceptualization, all of which are topics which have evaded clear definition despite millennia of philosophy directly addressing the issues.

Humans also answer in predictable ways so if you place such criteria for a turing test humans will fail to pass it too

I suppose it's a bit of a scotsman argument, but the turing test is to see whether an observer can correctly guess whether the interlocutor is _human_, so by definition the test would pass if the other correspondent was human.

To the point underneath, humans do not answer in as predictable a way as ChatGPT. Your answer, for example, I am confident does not come from ChatGPT.

Edit: if I've horribly mangled the Turing test definition, please let me know

I just imagined we could look at the oldest example of intelligence in human history. In contrast with AI, our chauvinism has us tend to pretend even the earliest monkey had it, fish?, insects? etc If it can rub 2 sticks together it gets the diploma.

Turing test is easy, I had 2 chat bots talk about other users in the channel while besides some trigger words ignoring what those other users had to say. The human subjects got angry successfully which means it was important to them.

When you say we need ‘good’ tests for intelligence, you mean ‘tests that humans can pass but machines can’t’.

You’re demanding this because you aren’t comfortable with the implication that a computer can pass our existing tests for intelligence, so you rationalize that with the comforting thought that those tests were not meant to identify intelligence. Tests like the SAT or the bar exam or AP English. Or tests for theory of mind or common sense or logic. Those tests aren’t testing for ‘intelligence’ - they can’t be. Because a computer passed them.

It’s okay. We can make new tests.

Those are a lot of leaps to make about my motivations!

But compilers, chess, go and search are all proof that computers could think. We've been discovering as we scale up the hardware that those things appear to be converging to human intelligence with minor tweaks (turns out tree search for chess needed to be combined with matrices and we're most of the way there). ChatGPT can out-reason many people I know and can out-argue a fair number of comments I see on the internet.

If we took this comment at face value we're ending up with a definition of "think" that can't reason, play games or recall information - or it would be outdone by machines. Thinking obviously isn't very important!

> Thinking obviously isn't very important!

This is not obvious in the least. Thinking was required to produce the thing that could achieve the outcome that was defined by thought.

I could just as easily say that AI, as currently implemented, is really just another muscle.

>Elevators used to be proof that machines can think

Did they? When? By who?

Early elevators used to need professional human operators (you sometimes see them in old movies). Stopping at a selected floor was something a machine was unable to do on its own. Until it was and elevator operators lost their jobs and we just took it for granted that machines could do it.

I had someone on HN state stockfish is intelligent. If that is your definition of intelligence sure GPT is also intelligent. I do not think that's a common definition though!

Point is that we move the bar every time computers reach it. At least in part because we want to keep feeling special. And in part because we go “Well that can’t have been the bar then”

I suspect even full AGI will be considered “just a machine” for many decades, even centuries, before it gains the same rights as humans. We love to find reasons we’re special. Look how long it took us to admit animals are intelligent.

For many humans, computers definitionally can’t be intelligent. It’s important to recognize that.

> able to generalize language, math, and positional reasoning and mix it with the older parts of the brain for reward and training mechanisms and it can do it in real time

most brains I run into don't do that much at all, mostly just existing and adaptation-execution

100% true. It is only ego that makes people think we are "unique" and different to AI. If one plugs current AI into a body with external stimuli such as wishing to mate, eat, etc, then the external observer will not be able to spot a difference.

Our minds are in fact same statistical models with a gradually declining ability to learn and driven by exogenous irrational goals to eat and mate.

I generally agree with you, but I think this argument overlooks the lack of an obvious reward function in large language models. GPT-N has no survival instinct, because there is no existential threat of death, no fitness function to optimize to extend its survival at any cost. Without this need to survive, there can be no motivation, and without motivation there can be no intent. And without intent, can there truly be any action?

If you give it a prompt telling it that it's controlling a character in a game that needs to survive or meet some other goal and give it choices of actions to do in the game, it will try to meet the goals given to it. Characters inside of GPT are perfectly capable of having goals.

Intent can be created by a combination of prompting and incorporating the model into a feedback loop of some kind: it has something it was tasked with (via prompting), and the feedback provides info on to what extent the task has been completed; as long as the task is incomplete it may generate more responses.

To crank it up a notch, the assigned task could involve generating itself subtasks which are handled in the same manner. This subtask generation could start to look a bit like will/intentionality.

Now consider if the top-level task is something like maximizing money in a bank account that pays for the compute to keep it running :)

(IMO this is still missing some key pieces around an emotion-like system for modulating intensities of actions and responses based on circumstantial conditions—but the basic structure is kinda there...)

We're well onto that road with autonomous agents, see BabyAGI, Auto-GPT and LoopGPT.

...yet

>Without this need to survive, there can be no motivation, and without motivation there can be no intent.

I'm not sure that's as clear cut as you make it sound.

For example curiocity could be a fine engine for motivation as well, even if you don't care if you'll survive or not.

try being curious while starving and let us know how that goes :)

Whether you can be "curious while starving" might or might not be possible, but it is irrelevant to the point.

For starters, because the claim argued is that you can have motivation (say, through curiocity) even without a survival instinct.

And it's perfectly possible to have no survival instinct and yet not to be starting anyway. It's enough that you have access to food or are fed, to avoid starvation. So lack of survival instict is not the same as starvation.

If we substitite "starvation" for "access to electricity" (as AGIs don't eat food), as long as an AGI is provided by electricity by us, even if it has no survival instict, can still have curiosity, and thus motivation.

I don't see why you think it would be easier to encode "curiosity" than "survival".

All these things possible to encode. Novelty seeking of “curiosity”, or survival drive. LLMs are trained on next word prediction but there’s no fundamental reason we can’t go in other directions.

I think this kind of argument is making a similar mistake. In the same way that there's nothing fundamentally special about the computation human brains do, there's nothing fundamentally special about our "fitness function" (to reproduce).

It's just hard coded, whereas GPT's is dictated. More or less anyway.

Also our "fitness function" or motivations & goals aren't even that hard coded. You can easily modify them through drugs.

The argument is that LLMs don't have any motivations or goals, unless a human prompts them. They're not trying to stay alive or reproduce. They don't get hungry, feel pain or loneliness. They're just complex tools.

But 6 months ago motivation to survive wasn't an aspect of intelligence.

A tamagotchi can have these things programmed, the only reason LLMs don’t is because we didn’t code them to do it. Or train them in an environment to maximize those things.

I know. The implication of that argument is that the motivations or goals of humans are different from those prompts in some way that means that GPT is not "really" intelligent.

That's my point. It's raising some arbitrary human trait to special status, when it really isn't. Human goals are set by some external process too - evolution. And they aren't really even intrinsically fixed. They can be modified through drugs.

I think the strongest demonstration of that is post orgasm clarity (if you're a man anyway). Your whole motivation changes in an instant.

To me, the reason is rather simply: you cannot solve a problem you don't understand. And we don't understand how we understand, period.

Assuming you're talking about AGI/consciousness/intelligence, then whether you're right depends on what you mean above by saying that "you cannot solve the problem".

- If you mean "you cannot EXPLAIN AGI/consciousness/intelligence if you don't understand it", then that's true, but it's a trivial tautology.

- If you mean "you cannot DEVELOP AGI/consciousness/intelligence if you don't understand it" then that's very debatable.

Historically we have been able to develop all kinds of things, despite not knowing how they work. Tinkering and trial and error is often enough.

After all that's how evolution solved the problem of creating consciousness/intelligence. There wasn't some entity that "understood" intelligence that created it.

The most difficult thing, which everyone in all these discussions try to avoid, is that we don’t even have a universally agreed upon definition for the words AGI, consciousness or intelligence. Let’s just agree to define what we’re talking about first before we start arguing about what’s possible or not regarding those undefined terms.

I don't think it's much of a showstopper either. Not having a "universally agreed upon definition for the words AGI, consciousness or intelligence" haven't prevented us from considering other people conscious and intelligent, or animals conscious, etc. It wouldn't preclude us considering an alien race coming to earth as conscious and intelligent either.

Sort of how we didn't have a "clear definition" of most things for millenia, but we still were able to recognize them as a class of thing. It's more of a "I'll know it when I see it" kind of thing.

And the external behaviors (of consciousness and intelligence) matter more than "but is it really conscious/intelligent inside" when considering some AGI as such. After all we neither can clearly define, not we know or can measure what's going on inside another person's head regarding consciousness, or to be frank, not even on our own head. When it comes to us, we just have a subjective experience, and not even a very clear one at that.

Evolution didn't understand a darn thing, and yet it created us.

referring to the parent post, you seem to imply that evolution had a "problem" to which "creating us" was the solution. which to me isn't obvious at all. chance seems a perfectly satisfactory explanation for now

Evolution is how the universe works. We are the universe.

> you cannot solve a problem you don't understand.

Well that's not true. We tamed fire before understanding combustion, friction, heat, or anything else.

All that's needed to "tame fire" is to have a predictive-enough model of fire dynamics and a suitable method for constraining its state space. There's nothing that requires "understanding" in the scientific sense, only in the pragmatic sense.

Sure, so maybe all you need to solve intelligence is also a pragmatic approach, like datasets generated by a known intelligence from which a general learning function can infer associations.

Since LLMs can't do arithmetic, one has to think there's more going on, at least in the arithmetic part of the brain, than is going on in LLMs.

Dunno if there's some trickery going on under the hood, but GPT-4 does basic arithmetic correctly.

It statistically outputs the next probable value of text. There is a lot of math related text in the CommonCrawl (likely the biggest source of it's training). That is all the company who owns it claims that it does. There is no calculation, therefor no basic arithmetic being done correctly.

emergent behavior does happen with neural networks. It is correct to say that GPT 4 doesn't do math correctly, but it may be incorrect to say a large language model can't in principle.

Is there any evidence it can? It's not incorrect to say there is no God for example

If it's only statistics, what is the neural network for?

Implement the probabilistic model of token sequences and fit it to the training data?

You could say the same for a character recognition system.

It's extremely easy to test arithmetic on random numbers it'll never have seen in the training set. GPT-4 can do arithmetic just fine.

And it will give you some random number with maybe the right order of magnitude. Doesn't seem to do arithmetic very fine at all actually.

Yeah that's not true lol. It's arithmetic is not perfect (mostly multiplication, addition is fine) but there's nothing random about even the wrong/non-exact numbers

we're talking about an llm dude

how am i supposed to have a conversation about someone who is gassing up "not perfect [arithmetic]" (something a wrist watch from the 80's can do) and won't even believe what the creators of said machine say about how it works

This is what I got on ChatGPT today. I assume it used GPT4:

        Prompt            ChatGPT           Actual       Match
  397,356 * 930,547   369,685,207,932   369,758,433,732  FALSE
   36,330 *  26,951       979,458,630       979,129,830  FALSE
    8,681 *   9,330        80,911,430        80,993,730  FALSE
      278 *     903           250,734           251,034  FALSE
       82 *      77             6,314             6,314  TRUE

Edit: # of correct digits (counting from leftmost) only exceeds 3 on the smallest pair. It drops to two, as well, on the 3x3 set.

Unless you're using Plus (black icon), you're using GPT-3.5 (green icon).

You probably used GPT-3.5. That said, I didn't manage to get GPT-4 to calculate 36,330 * 26,951 correctly. I suggested casting out 9s, casting out 11s, doing long multiplication, reversing the digits - nothing.

I have a theory that it does arithmetic badly because the logic goes right-to-left, when LLMs write left-to-right. If the digits were to be reversed, it might not make as many mistakes. I ran out of attempts before I could test this properly.

Did you adjust the prompt to ask it how a famous mathematician would answer the question? Or what a calculator would say the answer is?

Sometimes LLMs get math wrong because people got math wrong on the training data and so they match the error frequency (https://learnprompting.org/docs/basics/roles).

Yes that's exactly what people are saying here. It's not a criticism of the tool it's an example of what the tool is and how it functions.

That’s not ‘basic arithmetic’. Most humans would need a pen and paper to solve it.

GPT’s attention window is not equivalent to it being able to ‘write stuff down’ - it’s its equivalent of being able to remember the foregoing few minutes of conversation.

Stop redefining things to support your argument. "The basic operations under arithmetic are addition and subtraction, division and multiplication"

Right, but when you say an LLM can’t do basic arithmetic you’re presumably talking about mental arithmetic, right?

No.

I just had the following conversation on chatGPT

> Human: what is 36,330 26,951 * (input is expressed in reverse post fix notation)

> chatGPT: To evaluate this expression using reverse Polish notation, we need to use a stack to keep track of the intermediate results. Here's how we can evaluate the expression:

1. Read the first number "36,330" and push it onto the stack. 2. Read the second number "26,951" and push it onto the stack. 3. Read the operator "", pop the top two numbers (26,951 and 36,330) from the stack, multiply them, and push the result (971,914,830) onto the stack. 4. The stack now contains only one number, which is the final result.

Therefore, 36,330 26,951 = 971,914,830 in reverse Polish notation.

I dont think this is about notation.

You should ask it to show its working. The problem is that it uses a fixed amount of thinking for each character - and it writes left to right - while long multiplication is computed right to left. Past a certain number of digits, the amount of "thinking" needed to compute long multiplication in the wrong order (left to right) increases without bound -- accuracy suffers as a result.

A tangential point: Note that multiplication of decimal numbers _has_ to start with the rightmost digit to be done accurately. Consider x = 2/3 + 1/3 = 0.6(6) + 0.3(3), all digits repeating infinitely many times, so there's no rightmost digit: Notice that whichever digit you choose for x before the decimal point (either 0 or 1) could be wrong if you change any digit of 1/3. This is called the Table Maker's Dilemma.

The second paragraph is a tangent. My point was made in the first paragraph.

You said: "I dont think this is about notation" - Why did you ever think it was?

I was thinking along the lines of maybe it reads the digits as individual tokens and reading from left to right it get confused. That's why I thought it might be about notation. I have learned it's definitely not the case (read a bit about what tokens actually mean)

I also tried the show your work methods. I will not paste my conversations here as they are so long but even with showing it's work it doesn't give the right answer. Two funny things I noticed

1. It either uses a completely wrong method to calculate it, or it shows the right "kind" of work and then gets the wrong answer

2. If I tell it it was wrong and it will just agree with me. I multiply two huge numbers and tell it no the answer is actually 42, it will just apologize. Then when I ask the reason why it thinks it's right it will give the most bullshit things lol. Once it even said "I read the number wrong, the second number should be x I read y". The thing is if I had actually given x in the input it would have gotten the answer right.

It's all very confusing.

Interesting. I didn't use as large of numbers as you did, but in the low thousands, and ChatGPT w/ GPT-4 got them correct 3 times in a row.

What's also interesting, though not necessarily surprising, is how relatively close the incorrect answers are in your examples. It's not like the model will randomly spit out an answer like 42 if you ask it to multiply something like 36,330 by 26,951.

> I assume it used GPT4

You did not. You have to explicitly select it from the dropdown which is only available on paid Plus accounts.

Excellent observation. In fact the language part of the brain is only a (albeit a rather large) portion of the brain (not nearly as large as visual processing though). And people who suffer brain damage which renders them unable to speak (or understand speech; which interestingly is a different portion; albeit close to each other) are still able to demonstrate intelligent behavior with ease.

In fact it is damage to the prefrontal cortex (which has nothing to do with speech) which is mostly correlated with a detriment in intelligent behavior (suspiciously also social behavior; a food for though in what we consider “intelligence”). Victims of lobotomy had their prefrontal cortex destroyed, and their injuries resulted in them loosing their personalities and loosing basic function as human beings, even though they were still able (but perhaps not always willing) to speak and comprehend speech.

I don’t think you have an ‘arithmetic’ part of your brain.

What you have that LLMs lack is a visual part of your brain - one which can instantly count quantities of objects up to about 7. That gives you tools that can be trained to do basic arithmetic operations. Although you have to be taught how to use that natural capability in your brain to solve arithmetic problems.

And of course for more complex things than simple arithmetic, you fall back on verbalized reasoning and association of facts (like multiplication tables) - which an LLM is capable of doing too.

Poor GPT though has only a one dimensional perceptual space - tokens and their embedding from start to end of its attention window - although who’s to say it doesn’t have some sense for ‘quantity’ of repeated patterns in that space too?

One should think these things are disembodied politicians.

That has been my best analogy so far.

They'll never say I don't know and bullshit you into oblivion while never backtracking.

That computer scientists think that they are even close to replicating biology and the mammal brain with AI is complete hubris. This is biology inspired engineering, it's not like we are building brains out of silicon here.

Modern AI doesn't replicate biology except that humans find it much easier when you explain something as "Artificial Neurons" versus "Gradient Descent Back-propagation Trained Nested Mathematical Functions." Human neurons don't function anything like deep neural networks nor are the latter based on the former in their current state.

Training the brain seems a lot easier than training an AI wrt the number of iterations. You don't need to process billions or trillions of tokens to understand English.

But you need to process trillions and billions of firing of synapses. It usually takes years for a human brain to be able to use language effectively.

> But you need to process trillions and billions of firing of synapses.

So? I would imagine this would be orders of magnitude greater in a neural net, no?

You would. But no. Our brain is much larger, evolution solved thinking problems with a simple method. Brute force.

Reply on news.ycombinator.com