Hacker News Reader

Comment by semiquaver | Hacker News Reader

I think ”poorly understood” is accurate. Understanding has levels. How brains think is also poorly understood.

I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.

Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.

A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be recognizable but are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.

Notably, you could still print them all day.

Being able to put a PC together doesn't mean you understand computers. If I ask you to make me a computer that runs a wristwatch, you'll likely be lost unless you have very specialized understanding.

I think we actually all know what "poorly understood" means. There's no need to play tedious semantic games.

The semantic game occurred when someone was equivocating "LLMs are poorly understood" with "brains are poorly understood", which are worlds apart in the extent to which they are "not understood".

“equivocating” is not the word you meant to use here. Probably “equating” I suspect. But yes, I stand behind my claim and I deny that it’s a semantic game. The two areas are not identical in human understanding but they’re on the same order.

If I ask you what makes a good fantasy novel and you explain to me the English alphabet and grammar, you actually haven’t explained what I asked.

Your DNA is merely data, and humans can reliably make more of it too.

It’s likely that at some point we will also be able to represent a scan of the human brain digitally as code and data. Assuming we don’t separately make a huge number of leaps in neuroscience which are far from assured, it will be very likely that we can get the appearance of an operational human brain simulated digitally well before we gain much if any understanding of why it does what it does.

All that is to say, being able to build something is not not not the same thing as understanding it.

If you are serious, comparing how well we understand the brain vs how well we understand LLMs, .. it's not a stretch simplifying that to "we don't understand brains, we do understand LLMs".

Because the extent to which we don't understand the brain, is quite overpowering.

Some people forget that when they say "but it's not different from what a human does" ...

I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.

We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.

The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.

Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.

I'm not following you here, you seem to be conflating LLMs ability of language use with the brain's ability of thinking and cognition?

Are you saying that thinking and cognition requires language use? Cause I think not.

Or are you saying that language use is sufficient for cognition and thinking? Cause I'm also not convinced of that.

What I am convinced of, is that a machine capable of language use is capable of tricking people into believing there's a "there", there. In pretty much the same way as the famous supra-normal stimuli experiment made baby seagulls believe that a stick with a red dot was their parent. It's exploiting our instincts.

I re-read my comment and I’m not sure where you got that impression. LLMs perform a simulation of cognition. if you have used them for anything nontrivial you would be a fool to deny the fact that behind the language there is something approximating cognition. If you haven’t, solving a bunch of millennium prizes is pretty good evidence.

Reply on news.ycombinator.com