Hacker News Reader

> I cant see how this doesn't describe, abstractly, human cognition. now, maybe llms are not fully capable of the breadth of human cognition

But, I can fire back with: You're making the same fallacy you correctly assert the article as making. When I see how a CPU's ALU adds two numbers together, it looks strikingly similar to how I add two numbers together in my head. I can't see how the ALU's internal logic doesn't describe, abstractly, human cognition. Now, maybe the ALU isn't fully capable of the breadth of human cognition...

It turns out, the gaps expressed in the "fully capable of the breadth of human cognition" part really, really, really matter. Like, when it comes to ALUs, they overwhelm any impact that the parts which look similar cover. The question should be: How significant are the gaps in how LLMs mirror human cognition? I'm not sure we know, but I suspect they're significant enough to not write away as trivial.

do they matter in a practical sense? an LLM can write a structured essay better than most undergrads. and as for measuring "smart", we throw that word around a lot. a dog is smart in a human way for being able to fetch one of 30 objects based on name or to drive a car (yes, dogs can drive), the bar for "smart" is pretty low, claiming llms are not smart is just prejudice.

You are assuming that because we measure an undergrad's ability to critical think with undergrad essays, that is a valid test for the LLM's capacity to think -- it isnt. This measures only, extremely narrowly, the LLM's capacity to produce undergrad essays.

Society doesnt require undergrad essays. Nor does it require yet another webserver, iot script, or weekend hobby project. Society has all of those things already, hence the ability to train LLMs to produce them.

"Society", the economy, etc. are operating under competitive optimisation processes -- so that what is valuable, on the margin, is what isn't readily produced. What is readily produced, has been produced, is being produced, and so on. Solved problems are solved problems. Intelligence is the capacity of animals to operate "on the margin" -- that's why we have it:

Intelligence is a process of rapid adaption to novel circumstances, it is not, unlike puzzle-solvers like to claim, the solution to puzzles. Once a puzzle is solved so there are historical exemplars of its solution, it no longer requires intelligence to solve it -- hence using an LLM. (In this sense computer science is the art of removing intelligence from the solving of unsolved and unposed puzzles).

LLMs surface "solved problems" more readily than search engines. There's no evidence, and plenty against, that they provide the value of intelligence -- their ability to advance one's capabilities under compeititon from others, is literally zero -- since all players in the economic (, social, etc.) game have access to the LLM.

The LLM itself, in this sense, not only has no intelligence, but doesnt even show up in intelligent processes that we follow. It's washed out immediately -- it removes from our task lists, some "tasks that require intelligence", leaving the remainder for our actual intelligence to engage with.

I'd just like to say, with unfortunately little to add, that your comments on this article are terrific to read. You've captured perfectly how I have felt about LLMs roughly from the first time they came out to how I still feel now. They're utterly amazingly technology that truly do feel like magic, except that I have enough of a background in math modeling and ML to de-mystify them.

But the key difference between a model and a human is exactly what you just said. It's what animals can do on the margin. Nobody taught humans language. Each of us individually who are alive today, sure. But go back far enough and humanity invented language. We directly interact with the physical world, develop mental models of it, observe that we are able to make sounds and symbols and somehow come to a mutual agreement that they should mean something in rough analogy to these independent but sufficiently similar mental models. That is magic. Nobody, no programmer, no mathematician, no investor, has any idea how humanity did that, and has no idea how to get a machine to do it, either. Replicating the accomplishments of something else is a tremendous feat and it will get our software very, very far, maybe as far as we ever need to really get it. But it is not doing what animals did. It didn't just figure this shit out on its own.

Maybe somewhat ironically, I don't even know that this is a real limitation that current techniques for developing statistical models can't overcome. Put some "AIs" loose in robot bodies, let them freely move about the world trying to accomplish the simple goal of continuing to exist, with cooperation allowed, and they may very well develop ways to encode knowledge, share it with each other, and write it down somehow to pass on to the future so they don't need to continually re-learn everything, especially if they get millions of years to do it.

It's obvious, though, that we don't even want this. It might be interesting purely as an experiment, but it probably isn't going to lead to any useful tools. What we do now actually does lead to useful tools. To me, that should tell us something in these discussions. Trying to figure if X piece of software is or isn't cognitively equal to or better than a human in some respect is a tiring, pointless exercise. Who cares? Is it useful to us or not? What are its uses? What are its limitations? We're just trying to automate some toil here, aren't we? We're not trying to play God and create a separate form of life with its own purposes.

if this is how you feel you haven't really used llms enough or are deliberately ignoring sporadically appearing data. github copilot for me routinely solves microproblems in unexplored areas it has no business knowing. Not always, but it's also not zero.

...and i encourage you to be more realistic about the market and what society "needs". does society really need an army of consultants at accenture? i dont know. but they are getting paid a lot. does that mean the allocation of resources is wrong? or does that mean theres something cynical but real in their existence?

Bro you have as serious problem with reading the first few sentences of a comment, finding something you disagree with, then skipping over the entire rest of the comment, characterizing it as "well they must entirely disagree with my worldview and now I'm under attack". You gotta take a step back.

Well; many LLMs can compose a structured essay better than most undergrounds, yet most also struggle with basic addition. E.g. Gemma-3-4B:

add 1 and 1

google/gemma-3-4b 1 + 1 = 2

add four to that

google/gemma-3-4b 1 + 1 + 1 + 1 = 4

So, 1 + 1 + 1 + 1 + 4 = 8

Of course, smarter, billion dollar LLMs can do that. But, they aren't able to fetch one of 30 objects based on name, nor can they drive a car. They're often super-important components of much larger systems that are, at the very least, getting really close to being able to do these things if not able to already.

It should be worldview-changing to realize that writing a graduate-level research essay is, in some ways, easier than adding 4 to 2. Its just not easier for humans or ALUs. It turns out, intelligence is a multi-dimensional spectrum, and words like "smart" are kinda un-smart to use when describing entities who vie for a place on it.

the llm by default (without better prompting) is vibe solving math, and if you tell a human to vibe solve it they might give similar results. tell a teenager to add two three digit numbers without a scratchpad and they will mess up in similar ways, tell an llm to show and check their work and they will do much better.

> tell a teenager to add two three digit numbers without a scratchpad and they will mess up in similar ways

More likely they will say "lol, i don't know". And this is better than a lot of LLM output in the sense that it's aware of its limits, and doesn't hallucinate.

It feels like I'm talking with an LLM that's losing a coherent view of its context window right now. Because here's the conversation up to this point:

1. "LLMs are smart they have intelligence that is some significant portion of the breadth of human cognition."

2. Me: "ALUs are also smart, maybe that's not a good word to use."

3. "But LLMs can write essays."

4. Me: "But they can't do basic math, so clearly there's different definitions of the word 'smart'"

5. "Yeah that's because they're vibe-solving math. Teenagers also operate on vibes."

What are you even talking about?? Its like you're an AI programmed to instantly attack any suggestion that LLMs have limitations.

dude, llm context windows keep the latest material, not the oldest.

i only respond to the most interesting shit. most ppl jump in with something interesting and then gish gallop into nonsense because they at some point were taught that verbosity is good

On a finite planet, we probably ought to care about how many orders of magnitude more energy that the LLM must use to perform a task than our 20-watt chimp-brains.

The creators of llms don't fully understand how they work either.

So how would you explain how LLMs work to a layman?