Hacker News Reader

Comment by famouswaffles | Hacker News Reader

We do not write the code that makes it do what it does. We write the code that trains it to figure out how to do what it does. There's a big difference.

View on HN

The code that builds the models and performance inference from it is code we have written. The data in the model is obviously the big trick. But what I'm saying is that if you run inference, that alone does not give it super-powers over your computer. You can write some agentic framework where it WOULD have power over your computer, but that's not what I'm referring to.

It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model.

You cannot say, 'we know it's not thinking because we wrote the code' when the inference 'code' we wrote amounts to, 'Hey, just do whatever you figured out during training okay'.

'Power over your computer', all that is orthogonal to the point. A human brain without a functioning body would still be thinking.

Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage.

I permit you, that yes, the data in the model is a LOT more cool, but some team could by hand, given billions of years (well probably at least 1 Octillion years), reproduce that model and save it to a disk. Again, no different than data stored in JIRA at that point.

So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking".

>Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage.

Obviously, it is different or else we would just use JIRA and a database to replace GPT. Models very obviously do NOT store training data in the weights in the way you are imagining.

>So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking".

Thinking is by all appearances substrate independent. The moment we created computers, we created another substrate that could, in the future think.

But LLMs are effectively a very complex if/else if tree:

if the user types "hi" respond with "hi" or "bye" or "..." you get the point. It's basically storing the most probably following words (tokens) given the current point and its history.

That's not a brain and it's not thinking. It's similar to JIRA because it's stored information and there are if statements (admins can do this, users can do that).

Yes it is more complex, but it's nowhere near the complexity of the human or bird brain that does not use clocks, does not have "turing machines inside", or any of the other complete junk other people posted in this thread.

The information in Jira is just less complex, but it's in the same vein of the data in an LLM, just 10^100 times more complex. Just because something is complex does not mean it thinks.

This is a pretty tired argument that I don't think really goes anywhere useful or illuminates anything (if I'm following you correctly, it sounds like the good old Chinese Room, where "a few slips of paper" can't possibly be conscious).

The information in Jira is just less complex, but it's in the same vein of the data in an LLM, just 10^100 times more complex. Just because something is complex does not mean it thinks.

So, what is the missing element that would satisfy you? It's "nowhere near the complexity of the human or bird brain", so I guess it needs to be more complex, but at the same time "just because something is complex does not mean it thinks".

Does it need to be struck by lightning or something so it gets infused with the living essence?

Well, at the moment it needs to be born. Nothing else has agency on this planet. So yes, the bar is HIGH. Just because you have a computer that can count beans FAST, it does not mean because you counted a trillion beans that it was an important feat. When LLMs were created it made a lot of very useful software developments. But it is just a large data file that's read in a special way. It has no agency, it does not just start thinking on it's own unless it is programmatically fed data. It has to be triggered to do something.

If you want the best comparison, it's closer to a plant- it reacts ONLY to external stimulous, sunlight, water, etc... but it does not think. (And I'm not comparing it to a plant so you can say - SEE you said it's alive!) It's just a comparison.

You're getting to the heart of the problem here. At what point in evolutionary history does "thinking" exist in biological machines? Is a jumping spider "thinking"? What about consciousness?

> It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model.

Sure, and humans are just biochemical reactions moving muscles as their interface with the physical word.

I think the model of operation is not a good criticism, but please see my reply to the root comment in this thread where I detail my thoughts a bit.

This is a bad take. We didn't write the model, we wrote an algorithm that searches the space of models that conform to some high level constraints as specified by the stacked transformer architecture. But stacked transformers are a very general computational paradigm. The training aspect converges the parameters to a specific model that well reproduces the training data. But the computational circuits the model picks out are discovered, not programmed. The emergent structures realize new computational dynamics that we are mostly blind to. We are not the programmers of these models, rather we are their incubators.

As far as sentience is concerned, we can't say they aren't sentient because we don't know the computational structures these models realize, nor do we know the computational structures required for sentience.

However there is another big problem, this would require a blob of data in a file to be labelled as "alive" even if it's on a disk in a garbage dump with no cpu or gpu anywhere near it.

The inference software that would normally read from that file is also not alive, as it's literally very concise code that we wrote to traverse through that file.

So if the disk isn't alive, the file on it isn't alive, the inference software is not alive - then what are you saying is alive and thinking?

A similar point was made by Jaron Lanier in his paper, "You can't argue with a Zombie".

This is an overly reductive view of a fully trained LLM. You have identified the pieces, but you miss the whole. The inference code is like a circuit builder, it represents the high level matmuls and the potential paths for dataflow. The data blob as the fully converged model configures this circuit builder in the sense of specifying the exact pathways information flows through the system. But this isn't some inert formalism, this is an active, potent causal structure realized by the base computational substrate that is influencing and being influenced by the world. If anything is conscious here, it would be this structure. If the computational theory of mind is true, then there are some specific information dynamics that realize consciousness. Whether or not LLM training finds these structures is an open question.

> So if the disk isn't alive, the file on it isn't alive, the inference software is not alive - then what are you saying is alive and thinking?

“So if the severed head isn’t alive, the disembodied heart isn’t alive, the jar of blood we drained out isn’t alive - then what are you saying is alive and thinking?”

- Some silicon alien life forms somewhere debating whether the human life form they just disassembled could ever be alive and thinking

Just because you saw a "HA - He used an argument that I can compare to a dead human" does not make your argument strong - there are many differences from a file on a computer vs a murdered human that will never come back and think again.

I think the discrepancy is this:

1. We trained it on a fraction of the world's information (e.g. text and media that is explicitly online)

2. It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online (which may or may not be different to the experiences humans have in every day life)

> It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online

This is going to be a huge problem. Most people assume computers are unbiased and rational, and increasing use of AI will lead to more and larger decisions being made by AI.

I see this a lot in what LLMs know and promote in terms of software architecture.

All seem biased to recent buzzwords and approaches. Discussions will include the same hand-waving of DDD, event-sourcing and hexagonal services, i.e. the current fashion. Nothing of worth apparently preceded them.

I fear that we are condemned to a future where there is no new novel progress, but just a regurgitation of those current fashion and biases.

and then the code to give it context. AFAIU, there is a lot of post training "setup" in the context and variables to get the trained model to "behave as we instruct it to"

Am I wrong about this?

Reply on news.ycombinator.com