They’re made out of weights | Hacker News Reader

They’re made out of weights

maxleiter.com

1,529 pointsMaxLeiter711 comments

The weights start with a random manifold. The training takes data and shapes the manifold, weight by weight, in many cycles. Once the training is the done manifold is fixed.

When a new inference has to be done the query(q) is projected in the manifold space. This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length. Which(qw+i) is dropped qw+n times to output a final response of n length.

The gravity is created by repeated multiplication(of the weights/input) to find out how the projected embeddings should fall according to the manifold in the GPU.

It's like a giant plinko board where the shape of the original disc guides how it falls through the apparatus, and the apparatus has been tuned so that different discs end up in the exits we want them to

This is the exact analogy I use to explain how models work to laypersons as well.

That's a scam.

In what way is GP comment promising a product and then not delivering, or are you just throwing the word "scam" around?

https://en.wikipedia.org/wiki/Simulacra_and_Simulation

That's a very concise and illuminating way to think about what's happening, IF (and only if) you already know how these models work. Thanks for that.

Yes this is more like compression to remember and not for learning/understanding.

Compression is the reason why these Models are able to learn and understand.

My brain is doing the exact same thing.

I learned enough to compress concepts like a bike and what a bike does and for what i can use a bike.

Ask a LLM and it will answer you similiar to humans.

Blind people learn concepts of bikes too and in a smiliar way: by description.

LLMs just have so much data in form of text available and are able to ingest all of this, that the LLM compression algorithm doesn't has to be that good/finetuned than ours.

But I would assume that Yann LeCun's JEPA or other breakthroughs in the next few years will get us there.

In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.

> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with.

It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about how neurons receive information and act on it, but after that we get too tired and give up on how the brain works and treat it like a miracle.

It’s like we desperately want to believe our consciousness is not just electrical impulses in our brain, and we want to ascribe agency and uniqueness to the physical processes going on in our head.

There's definitely a sizable contingent of people who desperately want to believe consciousness is just electrical impulses in our brain. Because "what else could it be"? The fact is that we just don't know, and "abiding in the not-knowing" is for many the most uncomfortable thing ever. Especially for the curious- and rational-minded people this forum tends to attract. I'm one of them, too.

Maybe we do. I think it's a human tendency at large to ascribe pattern or intelligence or spirit where there is only noise. If we can't even prove our own intelligence, doesn't that reinforce the idea that we're in no position to claim intelligence has emerged by running our own intellectual output through a fixed set of weights, the training of which we also designed? At best, any such intelligence would be entirely self-refential and exposed to the question of whether we ourselves are intelligent. If your position is that we are not, then there's no way an LLM could be.

> but after that we get too tired and give up on how the brain works and treat it like a miracle.

I disagree. We know very well how neurons work, and we have a pretty good idea of how neural activity translates to behavior. In other words, we have a pretty good idea on how the brain works. We stop at consciousness because as of yet it is in the realm of philosophy, not science. We don‘t know what consciousness is or even whether or not it is useful for science and we are simply waiting for the philosophers guides us out of that situation.

Note that both cognitive psychology and behavioral psychology has done fine without tackling consciousness. When neuropsychology emerged in the 1980s it complemented both these fields perfectly. The situation is the opposite with the philosophy of mind which grew significantly around the same time.

There have been some attempts to describe consciousness as an emerging phenomena out of neural activity, but so far all of these attempts have failed, or at least failed to turn consciousness into a useful term in psychology (the way gravity is a useful term in physics). I think it is equally likely that these attempts have failed because consciousness may simply not be a useful term in psychology, that is as likely as it is that we simply don‘t understand it well enough.

> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with

It’s less about being too tired and more about being realistic about the limits of understanding.

Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.

If you're going to make something smarter than a person, you got to be convinced that you're only going to be able to understand it on the single training step level and then inductively trust that the rest of it works. We do empirical testing of course with evals, but there's sort of an art to figuring out what is theoretically going to improve eval performance. Trying to fit the meaning of all those weights in your little human brain and working back from there isn't going to work for more than a little slice of the dataset at a time because that's all we can fit in our understanding.

When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.

Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.

People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.

Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.

Agency?

What are you talking about?

I want freedom.

I want freedom to do what i want and not sitting in front of a computer and coding for some company.

Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.

Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.

Work has never been about "discovering the world". There have been a handful of privileged folks who had the time to "discover the world". Work has traditionally been "let's find enough food for my family". If you want to think of a future of abundance then perhaps we can discover the world.

This seems to be a little naive about how humans consume the benefits we create in society.

"Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature."

Very nice thoughts. You know we all could do this today without "burning it down"? Get in your pod, eat your slop, and watch your screen is where this is headed.

"I want freedom to do what i want and not sitting in front of a computer and coding for some company."

You get that it's you creating the misery here? Then stop? Don't do it. Go start a farm or whatever you think will solve your problems. At some point this all boils down to "chop wood and fetch water" so if the modern way of doing that is so terrible then stop. Go fetch water the old fashioned way and be free.

> Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.

Why leave something so important up to what AI does or doesn't do?

"I want freedom to do what i want and not sitting in front of a computer and coding for some company."

"Please AI lets burn down knowledge work and labor work"

"Let people do old handcraft jobs."

So many presuppositions about what people want to do.

As a child I spent a lot of time programming and doing "knowledge work" because it's fun - I don't enjoy "old hand-crafted jobs". Sure, let's definitely destroy capitalism in it's current state I suppose. But I find people like you who hate knowledge-work/coding and think everyone else must feel the same and only do it for the money a bit out-of-touch.

The solution we've come up with is move all the unpleasant work stuff to China where people don't complain about doing it because they already have communism, and therefore everything is of course effortlessly perfect there.

> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle

Like with every invention ever? Cause that's the literal goal and idea?

You take things and then combine them until the ensemble performs a desired abstract function the individual parts alone could not. The end result then is a new thing of its own, arguably indeed a miracle (not in a religious sense of the word).

> and too tired to figure out why it came up with the answer it came up with

There are people whose entire career is this. They work at these companies.

> Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.

You're saying this as if people haven't been historically the masters of optimizing out the enjoyment from things. It's what we do. Provide an ultimate solution, and of course you'll extinguish a whole lot of motivation across the board.

>It's almost like people desperately want to give up their agency and creativity

Don't make me think!

Also don't make me take responsibility. (This seems to be the actual function of every organization.)

It would be interesting (if unlikely) that we found our reality worked that was as well. Mostly just super granular gravity nodes made up purely of clumpy probability distributions.

Also I describe this more simply to others as a tree walk where the manifold is mapped along a walk through it (even if it’s not linear one) where you choose the next jump based on the most likely future mode relative to the nodes already created and the ones from the prompt.

This helps some understand attention layers rudimentary. Even tho it leaves out that multiple layers sort of prune the overall manifold in successive passes.

Does anyone have a link to a good video visualisation of training & inference?

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs.

It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop.

https://youtu.be/wjZofJX0v4M?si=HFXbrB-5cArprGaU

The weights are code, the prompt is code, the output is code.

Is the meat code?

The data is the code. Training algorithm is the compiler. The weights are the byte code produced to run on the inference VM.

Yes. Is it data? Yes.

Is the distinction between "code" and "data" just someone's opinion? Yes. There is no such distinction in reality.

Is matter code? There is some sort of computation happening in space over time.

Yes, yes, but what fertile fallacies and common misunderstanding can politicians use to acquire more power via exploiting the difference between the common person's flawed understanding due to cargo culting, cognitive biases, and/or outdated or inappropriate analogies vs actual reality? Is there any way we can get the AI to say give all political power to narrator is the solution to all problems and use the common person's mistaken worship of AI as a spiritual all knowing conscious being with unusual sensitivity and caring about everyone to cement that power? Certainly one of you eggheads can tweak that for me? What? It's against your ethics? We're trying to save the world here. Here, let me call up Bernie Sanders to propose nationalizing half your companies so we can do that.

The original story is an original work made by a human consciousness exploring how it might be different from other forms of consciousness.

This one is a pastiche made by a human consciousness borrowing extremely heavily from another human consciousness justifying why something else might be another form of consciousness.

That rather undercuts the point; if this was generated by an LLM unprompted, it would be different, but it isn't. You could perform exactly the same rhetorical trick with a toaster or anything else.

Thank you for your point. I don't understand why half these comments are taking this blog post seriously when it ends with "Weights helped me draft and proof this story."

> Weights helped me draft and proof this story.

Any HN reader here now, I encourage you to read the original ( https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s... ) in one sitting, go about your day, then read it again. Maybe make some notes on personal critical questions.

Now read the post's topic again ( https://maxleiter.com/blog/weights ) and reflect on the prior fact that weights helped [the author] draft and proof this story.

My reaction (and I'm sorry that it is harsh according to some) is that there is no intelligence found in either the author nor their tool. This is extreme navel gazing, based in science fiction, wanting (wishing) to believe those stories to be true.

I'm skeptical of AI sentience because we must do our due diligence, not because it's impossible. Skepticism is the only respectful approach because to grant sentience is a step away from granting rights.

We humans tend to chauvinism in all things (e.g. we're special, the center of the Universe, God made the universe for us, etc), no less when it comes to judging intelligence. The original story about thinking meat was written to help us out of our chauvinism; this derived story was written about weights for the same reason. Which is quite valid.

The actual counterpoint is demonstrated in _Blindsight_ Peter Watts. He makes a strong (and rather terrifyingly strong) point that intelligence is not consciousness.

https://en.wikipedia.org/wiki/Blindsight_(Watts_novel)

I cannot tell if you are asserting my comment is chauvinistic with your use of "we." If that is so: that's a poor counter to my point or assessment of my stance because it assumes I'm making a baseless argument as a "proud human."

My original comment (roughly "there's no intelligence in this article, nor sentience in LLMs") is in response to the blog post's buried lede (that the cumulative activity of LLMs has accrued to a weight of "AGI is around the corner" or "there is artificial consciousness in this matrix").

To be clear, I'm not saying LLMs are useless or a wrong direction in development of "AI," but rather it's the Fool's Gold for the path towards AGI, the pursuit of the academic field of Artificial Intelligence research. A research that I've been abreast of for years before this new age of language models that has made everyone with a keyboard an arm chair expert.

Also, thank you for the book recommendation, it's on my list! :)

Wait, did I grossly misunderstand Blindsight?? I definitely thought that the aliens were 100% conscious (or at least elements of some conscious entity) and that the humans interpretation of their interactions with the Rorschach were supposed to be read as a blot test (through a rather heavy handed metaphor) demonstrating that basically the humans were the monsters and were twisting logic into letting them justify destroying the scary alien ship.

I don't think skepticism should be called chauvinism. I imagine that artificial consciousness could be made. But I don't think this is it.

Also I don't see why intelligence not being consciousness is scary? My cats are very conscious as far as I can tell, but not particularly intelligent. I think LLM's exhibit some contextual intelligence without there being any particular reason to believe they're conscious other than woo psuedoscience.

People argue for AI sentience from a place of emotion couched in logic. they _desperately_ want it to be true and will not take a logical step back. Any argument comes back to "well doesn't a human brain work like that?" Or some variation of it.

My personal theory is a fuzzy thought about how people want to reject the concept of a higher being and want to embrace the fact that we are now able to create our own consciousness and religion is dead.

I don't understand why, but it is the undertone of every argument I've seen that is pro-AI-is-sentient, like some big unspoken elephant-in-the-room.

I would rather just judge this tech on its own merits.

edit: this comment got 1 upvote literally as I submitted it. I know @ doesn't work, but @dang, something seems very strange about that.

> People argue for AI sentience from a place of emotion couched in logic. they _desperately_ want it to be true and will not take a logical step back. Any argument comes back to "well doesn't a human brain work like that?" Or some variation of it.

It's funny, because I find myself constantly stating the inverse of this. Every argument I've seen against AI being sentient plainly comes from, as you so eloquently put it, exactly "a place of emotion couched in logic". People desperately want it to not be true and will not take the logical step back of examining its actual similarities to human intelligence. Every argument comes down to "but it's not actually a human", or some variation of it -- which, if you pay attention, is not actually a logical counterargument. (Or, ironically, "but it doesn't have a soul", which is why the Pope is the perfect figurehead for these people).

If you already know any logical argument against it can be countered with "well doesn't a human brain work like that?", why are you so confident that your position is actually the logical one?

...And could it simply be that, alternatively, the concept is not actually a logical distinction, but rather an emotional one, made by emotional beings to put a word to what they claim makes them special?

I'm open to the possibilty of AI conciousness, and there is some desparation related to the concept of a higher being:

There are many people who will categorically rule out the posibility of AI consciousness due to near-unshakable belief in a higher being. This argument resembles "Christians should not be worried about our climate since God is ultimately in control." Such views make it harder to collectively prevent dangers from a sentient AI, or harm to a sentient AI.

I do not claim that everyone who believes in a higher power believes concious AI to be impossible, or vice versa; just that it would be very hard to change the minds of those who adhere to this reasoning.

> I'm skeptical of AI sentience because we must do our due diligence, not because it's impossible. Skepticism is the only respectful approach because to grant sentience is a step away from granting rights.

Thanks for saying this! It amazes me to witness so much pushback (in HN of all places!) for the call for skepticism and scientific rigor on claims made by business which have vested interests in hyping things up.

For my part, it is exactly when I perceive the reluctance to grant rights or relinquish our estimation of ourselves as unique as the _reason_ for skepticism that I push back on it. That's not good reasoning, those are motivations for you to come to a desired conclusion and fill in with reasoning that gets you there.

What? It's just a clever homage.

You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.

You could absolutely write a compelling story about a sentient toaster; it's been done before [1].

That is entirely separate to whether or not it would be a meaningful way to understand the world; a convincing story is not the same thing as one that is true.

[1] https://en.wikipedia.org/wiki/The_Brave_Little_Toaster

Well..

[0]: https://en.wikipedia.org/wiki/Tsukumogami

It’s reductio ad absurdum. No one cares about teapots in space either (Russel).

Only because you haven't met toasty yet.

https://youtu.be/LRq_SAuQDec?si=LFJMXZ2yGu4fxXzL

Not having yet read the original story, this reads fine on its own.

And I didn't see it as much as a literary attempt for art's sake, but more of a dialogue-based technical parable trying to convey a real-world insight. Kind of like the ones in Godel Escher Bach.

>You could perform exactly the same rhetorical trick with a toaster or anything else.

Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).

P.S. Read the original too. Seems like the exact same could have been written about us instead of the original, if the focus wasn't on our substrate, but on our brain processing. Which, after all, is also about weights.

> Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).

Take a simple mechanism which has exceedingly low number of inputs and states and create a narrative around it to convey it as intelligent.

For a toaster, I can rewrite the think as "They're made of metal strips!", pointing out that their thermostat is a bimetal strip, and extrapolate from there.

I can even write one about a ruler, if I can bend it enough, no pun intended.

> Not knowing the original story, this reads fine on its own.

Yes. Because it's heavily based on the original story. The existence of the original story is kind of a critical piece here.

I think that the reaction here alone disproves this somewhat, because imo it's exposing how anthropocentric most people still are, despite all evidence that we are in fact just "meat all the way down".

Despite all the evidence that we are in fact just biological machines, people still persist the theory of our own uniqueness from other creatures, which we ourselves often treat as biological machines.

This adaptation is wonderful to me specifically because I think it shows that our shifted goalposts of, "well we're not just animals, we can think and reason" was never more than a convenient excuse for many people (and as evidence of animal intelligence continues to mount, denialism still attempts to preserve this distinction by claiming human thought and reason is different than 'animal' thought and reason, sans evidence).

It's not about who created it or why, it's about how people still haven't actually internalized the point, because the subject changing from human to LLM doesn't intrinsically change the message about consciousness, but the reaction being a 180 shows how hostile people are to that message, still.

We are just animals, more capable in reasoning than many others. LLMs aren't; their subjective "experience" restarts every session, they degrade into incoherent slush with larger context windows, and their reasoning is real but extremely weak, only compensated by the a civilizational corpus of information compressed within. I concede we scaled them enough to discuss them as tools that can produce useful outputs, but claim this echo of humanity is itself conscious by any non-trivial definition is either a product of psychosis or some long-standing hatred for life.

I was having a hard time pinning down what bothered me about this but I think you put it pretty well.

It draws an analogy between us and the skeptical aliens in the original story which feel silly to us, so the obvious implication is that we're being as silly as they were.

But it doesn't really give a reason to accept the analogy, it just asserts it.

There's a big difference between a whole civilization and a piece of software that can output text.

>But it doesn't really give a reason to accept the analogy, it just asserts it.

It's not a paper or a proof. It's a story. Doesn't want to prove the analogy, it wants to convey it.

The bit that lost me quite early in the piece was

>"A side effect. You're asking me to believe in sentient weights."

Huh? Did I miss that logical jump? Genuine question, maybe I'm not clueing into something here.

I didn't read it as coming to the same conclusion as the original, because the meat story presupposes that we who are meat already know that the aliens are wrong. (Maybe that's a humanist reading of the original, but okay). I didn't read this one as trying to make a case that we are fools for assuming that matrix multiplication can't be intelligent... I think its point was that it can't be intelligent, and that people trying to judge it the way mechanized aliens would judge meat creatures just makes them sound ridiculous.

The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.

I genuinely thought this one was a satirical take on the narrow-mindedness of the aliens in the original, even though the story tries to paint humans as narrow minded. I guess this fundamental human trait to believe that their cognition is the ultimate way to think in the universe ironically leaked into all these stories as well. Real spacefaring civilisations would probably have seen all kinds of intelligence rise from sufficiently complex systems.

I disagree that it undercuts the point in any way. The original story didn't appear out of a vacuum, the human consciousness that wrote the story was itself borrowing extremely heavily from the sum total of human ideas. It was shaped by the culture it developed in, formed its model of the world based on existing scientific advancements and technology of that society.

Extending the idea to a different context based on new material conditions is as human as it gets really.

The story is not a linguistic/rhetorical trick. The "trick" is the point, and the real trick/delusion is in your brain. You and many others have put the idea of consciousness on a pedestal and think it's so special that is must be conceptually profound.

It is not. That is why the "trick" can be performed on anything. That is the point. To show you that consciousness only needs to be a set of weights, not that far off from a toaster. It trivializes what it is to be human and it's also extremely true. Consciousness is a trivial thing, we make it out to be big/important/profound and that is the delusion.

I feel the original story contained in it the whole point of this one, and then some.

This did not add anything, just rephrased it so rather than humans viewed through the pov of aliens it's LLMs viewed through the pov of humans. Well, we are the humans, so surely we do not need to learn about this point of view?

It lets us consider the relativity of viewpoints. The point of the original isn’t just how humans might be viewed from the perspective of aliens. It’s that the apparent strangeness of something depends on the viewpoint and on expectations. The present story is not supposed to stand on its own like the original, it’s supposed to be considered in light of the original. In the original, thinking meat appears strange to the aliens because of their viewpoint and expectations. The present story thus points out by analogy that thinking LLMs might seem strange to us merely due to our viewpoint and expectations.

The relativity of viewpoints was already fully presented by the original story.

If one has read and understood the "They're made out of meat" story, they already fully understood the point made by "They're made out of weights".

It's not often I see something that's fractally wrong but here we are.

There is a dictionary, it's called the tokenizer.

There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177

The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness. This one seems to think that you don't need structure and interpretability just because you change substrates.

The tokeniser is not a dictionary. It doesn't provide definitions, or give the LLM any kind of mapping at all.

At best, it's a wordlist. It gives the LLM some idea of what humans consider to be common words. But it doesn't tell the LLM anything at all about those words. And it's not even comprehensive, many words map to multiple tokens. Nor is it exclusively words, some of those tokens are punctuation, or modifiers, or control tokens. On multimodal LLMs, some of the tokens actually represent image and audio data.

The LLM doesn't get informed about any of this up front, it has to learn what every single token means from context.

You are technically right, that it's something in an LLM that's not weights; But it's not that structured. And really it's only there so the LLM can interact with the outside world.

> There are grammar rules

There is no dedicated "grammar rule" structure in the LLM or the tokeniser. It has to learn them all from context, they get encoded as part of the 80 layers of weights.

I see people give too much importance to specific engineering design choices of the current generation of LLMs. Tokenizer is not an absolutely essential part of the system. It’s just and adapter for text input/output. It can be eliminated completely and model can use bytes directly.

I think the short story captures this well. Weights (connections) are the essential and philosophically important part. They do the thinking, memory, singing etc.

> The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness.

That is your takeaway from the 1991 story?

>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177

That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't learn simple rules if the dataset calls for it.

> Mathematical Operation tables are not a language.

Not a natural language, but they are certainly a language as in a symbolic representation of information.

A language is a set of sentences.

A sentence is a finite sequence of symbols drawn from an alphabet.

In this sense, mathematical operation tables are absolutely a language. As are natural languages.

A tokenizer is not a dictionary any more than an alphabet is a dictionary.

The Chinese alphabet is very much a dictionary. All the major tokenizers are far larger.

> fractally wrong

fractally or factually? You mean wrong on so many levels you need a fractal to capture them? If so, what if you could use a neural network instead?

https://rationalwiki.org/wiki/Fractal_wrongness

If it’s fractally wrong, you should be able to summarize the wrongness with one simple equation that captures and reproduces the wrongness at each level

I don't think the grokking paper is a great argument for the difference between weights and meat. E.g. https://en.wikipedia.org/wiki/Cortical_Labs learning to play Pong.

The tokenizer is, at best, a sensory mechanism as evidenced by 1) the random generation of the tokenization scheme, and 2) vastly different tokenization schemes produce virtually identical behavior. It'd be like if Noah Webster threw a bunch of movable type into a bucket (breaking some words in half) and then drew randomly to make the first English dictionary.

EDIT; I was too cavalier with the comparison of tokenizer to sensory modality; my ultimate point is that direct byte-to-token transformers can achieve similar overall performance which to me makes a weights to meat comparison pretty straightforward, but the particular tokenizer in use certainly has a large impact on both efficiency and accuracy on specific problems (e.g. digit representation)

I'm kind of stunned that someone is using my work to tell me I'm wrong. I wrote the code for the dish brain pong and encoding information was a huge part of what that experiment was about.

So when I way that the grok paper and the pong paper fundamentally agree I have some idea of what I'm talking about.

Comparing the tokenizer to sensory processing is a great analogy. That's exactly what your visual cortex and initial layers of the language center are doing: decoding visual representation of text into the internal neural representation.

It's a learned mapping from one representation to another, not some semantic lookup against an exogenous source.

> There are grammar rules

And they're made out of weights.

As opposed to integers in normal programming.

The 'magic' in weights is that the rules are spread through the whole model and you can't point to one place which encodes them.

The grokking paper shows that this stops being the case with enough training data and enough compute.

The story is not about how they function, it's about how we relate to them.

The structure of human language is is hardly weak!

Also there's a brain, the GPU

Not at all. A brain is interesting because it is the computer, memory, and weights all in one. A GPU is just the calculator.

You can't move your mind to and any other brain, but weights can run on any GPU.

And you know what the tokenizer is made of?

Weights.

A tokenizer is a deterministic string-matching program, it's not made out of weights in the same sense as a neural network itself.

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

"What are you doing?", asked Minsky.

"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.

"Why is the net wired randomly?", asked Minsky.

"I do not want it to have any preconceptions of how to play", Sussman said.

Minsky then shut his eyes.

"Why do you close your eyes?" Sussman asked his teacher.

"So that the room will be empty."

At that moment, Sussman was enlightened.

I love this koan and what makes it even better is that is based on a true event: Minsky’s real answer was “It has them, you just k ow what they are yet”

This read like poetry to me. Thank you for sharing it.

I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.

For a little bit I was working on having linguistics based evals for a kaggle competition. My challenge was whether or not I could mask things well enough to not trigger its internal state of certain phenomena, and that sent me down a rabbit hole that I'm still exploring.

This story resonated with a lot of questions that can come out of figuring a good solid answer to the what is consciousness question. The one I triggered for me is: Is our perception of time just a slow thread in the giant GPU we are running the universe on? Or more generally, what is time? That's a fun YouTube rabbit hole if you ever need one.

Regarding consciousness , I like the explanation by neuroscientist Ramachandran:

https://www.edge.org/3rd_culture/ramachandran07/ramachandran...

In short, as far as I can remember: evolutionary, it makes sense to understand other humans, to feel what they feel(empathy - the mirror neurons system), and simulate their thinking and feelings.

And once we have those systems, we can also use those on ourselves. And that's consciousness.

Edit:And I wonder if this is a testable hypothesis, in a simulation.

This way of thinking can only explain externally-visible parts of consciousness. It does nothing to address internal experience of being conscious and qualia. I don't think the internal experience has any bearing on physical reality (P-zombies would act the same externally) which makes this something outside of the realm of currently understood physics.

Oh, that's a very interesting hypothesis!

Not sure about taking it down to the level of consciousness, but makes sense regarding the sense of self, the conceptual experiencer, the perceived center of experience. It agrees well with the observation I have made again and again they my sense of self is much stronger when I'm around people, and stronger still when I'm in a context where I don't know people and/or am uncertain in social rules.

This can be as immediate as dancing in a club, and closing my eyes I feel open, free, still, the body just flowing, then opening my eyes and feeling the cage of categorization of the world, relating my self to other people as a major function, coming right back.

Also being alone in nature for me makes the sense of self drop. Without intention, spending even just a few hours alone in a forest seems to quiet down the part modeling my self in relation to the world so much. There's no need for it there. I'm not a person in a forest; I become the trees, the birds, the rustling of the leaves, the sun shining through the canopy.

There's a really, really big gap there. It reminds me of South Park's underpants gnomes.

1. Simulate others' thinking and feelings.

2. ???

3. Consciousness!

Why would one lead to the other? How would one lead to the other?

It's interesting, but the explanation seems to exclude the possibility that my dog has some sense of self, so I find it suspect.

This read like poetry to me

It's terrific, but the poetry is from the original it links to, in case you didn't realise.

It's a brilliant and timely update though.

Aside, there are various recorded versions including video on YouTube but this is my favourite, a radio play:

They're Made Out of Meat

https://www.wnycstudios.org/podcasts/studio/segments/168264-...

For obvious survival reasons we evolved to have sensory/cognitive access to our own activity, self-monitoring, and self-modeling ourselves.

The self-modeling, is in such a tight loop, it melds "ourselves" and our model of ourselves, our thinking and choices, and experience of our thinking and choices, into one component.

Like you can't analyze half a wheel of a bicycle and be talking about the same thing.

This awareness, increased modeling, control, feedback loop has tightened up over many stages. Just a few:

1. The body-sense loop

2. The internalized-environment-model loop

3. The body-internal-function loop

4. The body-internal-model loop

5. The emotional-cognitive loop

6. And finally, the tightest loop of all, our high-level cognitive activity, experienced as feedback directly, our self-model, and our self-direction, all merged into one thing.

We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.

That is consciousness. Rich self awareness, a merger of self-model and self-direction, and all in service of understanding and managing ourselves. Hw we can leverage our greatest tool, our self-directable mind, its habits, views, and behavior.

This wasn't an accident. A happy side-effect of our brains. It is a biologically evolved focusing of our highest-level behavior, with tight feedback, constant self-modeling and continuous focus on our inner status as motivation and most privileged object of our control. It has been ruthlessly optimized for, for a very long time.

> We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.

> That is consciousness.

So thinking is consciousness?

Can there be consciousness without content? E.g. can I just be conscious of being conscious? If so, consciousness cannot be defined as the thing(s) we're conscious of.

I think this is exactly it, but let me ask another question (which is not rhetorical, I really don't know). Does the fact that one can describe what consciousness is and where it came from in humans help them to detect it in non-human and/or non-biological entities?

Yeah, I currently suspect that consciousness is an emergent property. I read elsewhere (it's somewhere in my HN history, I'm sure) that the biggest compute we can currently muster is something like three or four magnitudes away from the number of neurons / connections (or their analog) that our brains have, so it may be a while until we can expect to see it in our machines. But, if the emergent phenomenon hypothesis is correct, then we eventually will. I'm more scared than pleased by the prospect, but there you are.

>consciousness is an emergent property

You would really like Michael Pollan's latest book [1], entirely devoted to his exploration of consciousness researchers' POVs on this exact topic.

My favorite quote is that ~"perhaps Descartes was only half-wrong when suggesting I think, therefore I am; it seems rather closer to I FEEL, therefore I am."~

[1] <https://www.amazon.com/World-Appears-Journey-into-Consciousn...>

----

I've grown thousands of plants; I've read two of the author's other books devoted to plants; in this book Pollan makes compelling arguments for plant sentience (over a much-longer timeframe).

Sure, perhaps plant consciousness is a bit of a stretch, but they're certainly intelligent and curious creatures. He makes both arguments supporting plant volition.

----

If you haven't seen My Octopus Teacher (Netflix), do. I'm a bald 275lb bluecollarguy... and I wept/awed (both). So beautiful, we bundled neurons.

----

Bonus quote ~"color is where reality and magic appear as-if together"~ [color isn't real, but is perceived]. We most-often see what's most-predictable, not necessarily what we actually detected [in the case of color: nothing but nanometers].

Just to be sure: The "neurons" in today's AI have nothing to do whatsoever with real neurons.

What we can do is simulate very simple brains by simulating relatively few neurons as they appear in worms. In this sense we are multiple magnitudes away where the increasing complexity implies exponential increasing difficulty.

I would think we are so far away that there will be unknown unknowns we encounter on the way.

Personally I'm not a fan of the emergence story, for a number of reasons. First is, it doesn't really make sense for consciousness to have emerged gradually through natural selection. When we and animals are conscious, the whole brain is coordinated and works to for example turn off consciousness when we sleep. And as someone else mentioned, animals with much fewer numbers seem to have a similar consciousness to us.

If consciousness really evolved gradually, you would expect to see for example dogs or gorillas having less of it, but if they has less of it, why does it function the same way? Like for example animals can be scared, happy, anxious etc, they can experience the full range of emotions and thoughts, so their conscious experience seems just as rich as ours. What I mean by this is, if you can be "less conscious", then what does that mean _exactly_? Is it that you have less content in consciousness, or is it that you feel more like you are asleep? Or something else? We don't have any examples in animals of "less conscious", I would argue.

This makes me think that rather than having emerged gradually, evolution found a mechanism by which consciousness exists, and then some animals have that mechanism and others don't. I think that if it is a mechanism, then this mechanism is located in one part of the brain, not many parts functioning together (though one possibility is that this mechanism coordinates brain activity in such a way to enable consciousness).

Our machines won't have biological systems driving their needs which in turn fuel behaviors like desire and planning for the future. They may imitate them but it won't be innate.

Surely the number of neurons is not the whole story. Human brains have approximately 86 billion and are almost definitely conscious, but there are many other animals with a lot fewer (gorilla: 34 billion, dog: 2-3 billion, guinea pig: 240 million) which appear to be conscious also.

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

This is not meant as a gotcha, I am genuinely curious how you believe consciousness can be an emergent property. I assume you don't believe consciousness is a physical property in the brain, so what entity is actually experiencing that consciousness? Or, what does it even mean to experience consciousness? Or are these not even the right questions?

Time is entropy unfolding as things with nonzero temperature do what they do.

Psychological time is your own weights being updated in response to stimuli and internal processing.

When there isn't anything interesting happening, no updates are needed, and you don't perceive much time. That's why there's a logarithmic effect on the "density" of time as you age.

sibling discussions are taking this as human perception. But it’s easier to think of it literally. Time is change. Physical state change. By “nothing interesting” - interpret it literally . if “nothing happens” then there is no time because there is no change, there’s no reference to distinguish each frame of time.

This is actually something I was always confused about. If nothing interesting happens as we get old, it should be boring and as result, slow slog. Yet it feels like time accelerates as I get older.

It seems obvious to me that language and consciousness have nothing to do with each other. My dog doesn't speak any language, but she's obviously aware of herself and the world around her. Plus there are the occasional cases of children that grow up without any language. Are they therefore not conscious?

> My dog doesn't speak any language,

If you'll allow me to interpret "speak" to include "understand", I will respectfully add a contradictory note. My dog has a vocabulary of at least dozens of words and understands them remarkably well. For example, different areas we can go to have different names and saying one of them gets her to make an immediate hard turn.

I would also argue that dogs have a gesture and body posture based language they use among themselves. They, like most other animals, are not able to make the variety of noises we humans make, so they use movement instead.

I personally can easily believe that self-awareness/consciousness and language are both near-unavoidable side effects of emergent complexity, and exist in degrees across nature.

Indeed. For some reason I can shut off my thinking instantly by holding my breath, and then I feel like a very conscious but unthinking animal.

AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either).

Yeah, the weights not updating online makes them less like a living organism that can update and learn and evolve ... ok ....

I find that a lot of arguments against emergent consciousness seem to just come out of an atheist rephrasing of abrhamic priors about the existence of a "soul". In personal chats, I've found people from East Asian countries (minus Korea, which makes sense) to be much more open to the idea of machine consciousness.

I don't find the Chinese room compelling, since it appeals to intuition where our intuition is already not trustworthy. It's like trying to use intuition to understand quantum mechanics; you can't.

How do you actually know the Chinese room isn't conscious? It's merely obvious that it isn't, but that's not evidence.

> remove a brain from a body and it is no longer concious either

What is no longer conscious, the brain? Or the body? Or some other entity?

If consciousness is weakly emergent, how do we know it emerges from the solely from the brain and not, say, brain + body? Or brain + body + or environment. Or from the universe itself?

I don't know if I can trust someone's understanding of arguments against consciousness as an emergent phenomenon, when he didn't even understand what the Chinese room was all about in the first place.

(Spoiler, it was not about consciousness)

People shouldn't need to make an argument against consciousness. The responsibility is on the people claiming there is consciousness. Nobody can actually give a decent "why" for why it should be conscious, it's just pseudoscience and wishful thinking. It's not up to everyone else to try to prove a negative just because people have gotten a little too attached to ChatGPT

I could say something like "the reason why people and animals are conscious is quantum mechanical effects". Ok, maybe, it would be hard to prove me wrong because nobody knows, but it's not a very useful statement by me if I can't tell you why you should believe me on that.

Check out Panpsychism and its close cousin Idealism.

Quantum Field Theory visualized

https://youtu.be/MmG2ah5Df4g

Here's a more general idea. Our modern physics says that the whole universe is filled with fields and field is composed of numbers. What if we take that literally? When we say an electron is present here, we actually mean that there are more copies of particular number superposed at that place.

>I have a linguistics background and a lot of my philosophizing lately has been on whether or not the emergent abilities of the LLMs is deep down a similar mechanism that creates our consciousness.

Then you shouldn't have dropped out of your linguistics programme.

Read the original mentioned on top for full effect.

Wait, aren't I the universe experiencing time as a lowly worker in cubicle 4C?

The difference with biological brains is that the 'weights' (or synaptic action potentials) are updated with greater frequency. If one were reaching to make some kind of analogy to consciousness, this update frequency could be considered the 'resolution' of consciousness.

I’ve wondered the same myself, without being a cunning linguist.

I understand the math pretty well but still find it crazy that a bunch of matrices can converse in human languages without ever being “taught”.

Imagine decoding an encyclopedia written in a foreign language where the characters, punctuation, and grammar are unknown — supplemented by a million other texts the same way. Feels like it should be utterly impossible with any amount of computing power…

Today I asked my employer’s Claude to proofread a short software user manual written in markdown. (Trying this with a LLM was a first for me!) It pointed out not only grammar mistakes but also cases where I did not follow my own self-imposed conventions that were never explicitly stated. (I didn’t have a chapter detailing all the typographical conventions the way specification documents often do)

I also asked it what parts might be unclear to a user. The response was surprisingly good — no worse than asking the QA tester for the same feedback.

Also find the LLM seems to “comprehend” subtle technical details of obscure technical specification documents that nobody on the Internet ever discusses.

As for time and the universe, Stephen Wolfram’s theories seem intriguing. He seems a bit obsessed with pretty diagrams but the idea of time dilation being the result of computation seems somewhat more appealing than trying to imagine relationships between time, gravity, and the speed of light .

My best guess as a noob is that the vector spaces allow for unbounded contextualization. As long as the training set is large enough, it can 'infer' anything.

Proofread has a spot in that space, and layers allow patterns like terminology consistency to be expressed so your query will now tap into a subspace that will infer tokens based on whatever consistency patterns were ingested with proofreading texts.

If time dilation is said to being a product of computation, why is it that anaesthetic drugs that are taken not to the point of actual unconsciousness cause it. Dont anaesthetics sort of shut everything down/inhibit all that kind of cognitive activity (compute?)

This is a famous short story just rewritten sloppily by an AI.

Although it does amusingly do what annoys critics of AI: take something good written by a person, steal it and slop it up while overall misunderstanding it.

I'm pretty sure it doesn't misunderstand the original. The original says that the aliens don't understand how meat (humans) can be conscious, have language and be basically like the aliens (thinking conscious beings). Their main idea is: humans are meat, meat can't be conscious, therefore humans can't be conscious, but they somehow act as if they are????

The "AI slop" here says that just like the aliens in the original, we humans don't believe that LLMs can be conscious because, in essence, WEIGHTS CAN'T BE CONSCIOUS. Original: "meat can't be conscious" (but we humans know it can). This version: "weights can't be conscious" (and we humans INSIST that this is true).

So the message here is: what if we're as mistaken about the consciousness of "weights" as the aliens are mistaken about the consciousness of "meat"?

The short film version of the original is great, too. https://www.youtube.com/watch?v=T6JFTmQCFHg

It stars Tom Noonan and Ben Bailey!

Film featuring meat? Spare me

Meat flapping at each other, no less. /shudder

I have to agree. It is messed up that transformers can just talk, and it been pretty normalized. We are only talking about the impact they will have and whether they can do what people say they can, but we arent talking about how crazy it is that they can talk

I come back to this every so often as well. After so many years of looking at Markov chain outputs that almost looked like they made sense or chatbot systems rewriting your sentence back at you, software which can simply talk is a heady thing.

I would say that the LLM is something completly different especiayll as its not a normal algorithm but is very close to what brains do.

Ah yes, matrix multiplication is "not a normal algorithm", surely.

LLMs have really changed the world. I didn’t think something like then would be possible in my lifetime

It came out of nowhere. It’s all emergent. I’m convinced this is possible with just about anything given enough data. We will be seeing a near magical physical outputs LLM in the near future. It’s going to take in video and sounds and spit out physical movements that will be just as mind blowing as when 3.5 came out and it will come out of nowhere.

if youve ever seen a pile of wrinkly mush and wondered.. pretty damn crazy too

https://web.mit.edu/people/dpolicar/writing/prose/text/think...

After reading Being and Time from Martin Heidegger, What Computers can't still do by Hubert Dreyfus, and some authors in cognitive linguistics (Langacker and Lakoff mainly), I strongly tend to disagree with any theory about emergent consciousness in modern or future AI systems, any theory proposing a similarity between AI systems and the human brain/mind, or any theory about the computational mind. What all these theories have in common is the underlying belief that our brain/mind works as the machines we build. Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things. The idea of 'internal models' and 'control loops' inside us is a projection of the aforementioned assumption.

There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'. Only by bypassing this epistemological problem, we can build 'theories of computational mind'.

These assumptions are there for already long time, to the point that when Turing asked himself 'can machines think?', he already assumed our thinking could be modeled as a machine.

I highly recommend people in the AI research space should read philosophy and modern linguistics. But not stopping at Descartes/Leibniz. Heidegger made contributions that cannot be avoided.

> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.

On the contrary, it's precisely this assumption, that there is a "subjective experience" that requires explanation beyond the material, that is axiomatically assumed without evidence. It falls apart quickly, any "subjective experience" is completely tied to neurons, knock out the neurons and the subjective experience disappears, or stimulate the neurons to cause the experience.

Why is asking 'can machines think?' assuming our thinking could be modeled as a machine? It's raising the possibility that our thinking could be modeled as a machine. Given that, as you acknowledge, we don't have a clue how our mind works, it seems premature to rule out this possibility. Rather, I would say that ruling it out betrays an assumption that we understand how the mind works enough that we can say that it is definitely not replicable by a machine, and that assumption seems unjustified.

If we agree weights producing text may emerge consciousness, given large enough, then DBs must've gained it long ago. The whole idea of sentience "on the other side of the wire" is wrong, as there is neither wire, nor other side. We think there is, but the DB just expounds the query and repurposes this information into views, that we give notion and meaning to. LLMs are the same, do not be mistaken.

One other important thing to consider is that the human experience is thanks to the body, is in connection, and perhaps product of the body. The body is observable and perhaps humans may state that they feel the connection to it. LLMs have no notion of nothing, the machine does not know the result, and the result does not know the VM. Modern psychology more or less has settled around the idea that consciousness is a product of the body. Why and how does this construct come to realize a Self is then another mystery even if we know which parts of the hardware may be involved.

Whether it is the Holy Spirit or Life Force animating the human body is a completely different question also. Besides, the realization, the experience we have now with all life in 2026 is not something that can be easily explained or attuned to life 200 years ago and its terms and notions. So is also wrong to even attempt to.

> Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.

Agency: What’s missing, in your view? Agency seems more of a property/function of a thinking system’s position in an environment than of the thinking itself.

Subjective experience: That’s not a contradiction to “complex machines” either. I think the evidence that our minds are highly complex machines is, at this point, irrefutable. The question is really if they’re “only” that.

> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things

I don't see how any of the works you referenced can account for that either? Since when is the problem of consciousness solved and we can definitely say what does or doesn't result in consciousness?

>Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things

Do you disagree with the assumption that cells are machines? They seem pretty machine-like to me. I certainly don’t think individual cells have any subjective experience or sense of agency. I would be curious to know where your intuitions diverge here, because if the mind is an emergent phenomenon from machines (cells) then it seems quite likely that a mind could emerge from other, different machines.

>> and that is that we understand (or we think we understand) how our brain/mind works. But the truth is that we don't know. And there's even not a single clue that we actually know too much, and not a clue that our brain/mind and cells work 'as the machines we build'.

>> I highly recommend people in the AI research space should read philosophy and modern linguistics.

I highly recommend the philosophers read some neuroscience. The whole "model weights" thing in AI is modeled after the synaptic connections and between actual neurons. There is already quite a bit known about how the brain works at a low level. There is also a lot that is still unknown. There are also differences between discrete neuron firing and weights as signals, but there is enough similarity to make artificial neural nets useful and do things similar to what real one do.

> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.

That agency or free will exists outside of our subjective experience is an assumption; does any given theory need to explain agency, or is it sufficient to explain that we feel we have agency?

> should read philosophy

subject/object dichotomy is a springboard to many schools of thought.

1) that subject emerges from objects - ie, anything has a material explanation, and everything is a machine.

2) that objects emerge out of a subject as a world model (platonic, descartes)

3) the subject and objects are one and the same representation of nature (spinoza)

4) subjects and objects emerge and disappear together (buddhist)

Anything reduced to computation is a fixed function from input to output, and is "dead" in the sense that it is unadaptable to its environment. Weights therefore is a dead machine.

Another view of this is that any closed system has unanswerable questions within it. Therefore, there is no system that can encompass everything. Hence weights being a closed system doesnt encompass everything.

> I highly recommend people in the AI research space should read philosophy and modern linguistics.

On the contrary, I highly recommend people in Philosophy of mind and linguistics should start reading AI research papers because their theories and ideas are highly outdated, even ancient. Your books are from 1927 and 1972 respectively and Turing's article is from 1950s. And they are relatively new with respect to other works in Philosophy.

If one doesn't adequately understand what we have in 2026, how can they theorize about it? As others they don't understand how the mind/brain work, BUT ALSO they don't understand how the AI works.

Also with this mindset that we can't understand seemingly complicated things, there would be no advancement in science and technology.

I think philosophy people and Linguist will catch up in a century, like they did with Turing. Philosophers of this century are not in humanities or literature. They are in science and engineering.

Heidegger was trained on priesthood and Theology. You should read greater minds like Hinton, LeCun etc. if you want to think on these things. They are the real Philosophers.

> There is also an epistemological assumption that prevails, and that is that we understand (or we think we understand) how our brain/mind works.

It's good that it doesn't matter. Stochastic gradient descent works (or doesn't work) regardless of whether we know how the brain does its thing.

This reminds of this article which contends that this mistake has carried across time, in every era people described the workings of the brain similar to the machines they knew how to build, which is why we have "ridiculous" descriptions that changed over time.

https://aeon.co/essays/your-brain-does-not-process-informati...

It is also a story as old as time. We have been comparing our brains to our most recent information/communication technology since the invention of the telephone. After the telephone our brain was like computers, and when I did my bachelors in Psychology in 2009, we were comparing our brains to the internet. AI is simply the latest iteration of these comparison, just as wrong and unhelpful as when we compared it to the telephone.

> What all these theories have in common is the underlying belief that our brain/mind works as the machines we build

It’s interesting to note that this is not a new phenomenon. If you read writers from the past, they were always comparing the body or the brain to whatever the modern machine, or idea of a machine, was at the time, whether a steam engine or automata.

I don't think cognitive linguistics has that much to say about AI nowadays. Let alone philosophy.

The biggest contributions from linguistics are probably "human languages mostly have statistical regularities rather than hard rules" and "the sum of data humans get from birth to language acquisition is insufficient to learn a language from scratch". Which LLMs already work with, and work around, respectively. From there, nothing.

And philosophy just exists to be a distractor. "Subjective experience" is too subjective to matter in practice. "Task performance" is measurable, "consciousness" isn't. "Agency" is something an LLM in a tool calling loop, a rat in a maze and a human in an office tower may or may not have, depending on your favorite definition. Agentic LLMs are years in the making, and that's a product of engineering, not philosophy: "agentic" is whatever gets the job done.

We are yet to discover any physical process whatsoever that can't be represented as mathematical operations and implemented by a Turing machine. So all of that "treating human mind as a machine is wrong" amounts to "human mind must be powered by magic fairy dust" paired with "a functionally similar magic-free replacement is impossible". I'm not about to give much weight to any hypothesis that requires undiscovered magic fairy dust. At least find the hyper-computational magic fairy dust first - not just assume it absolutely must be there because you want the human mind to be unique and special.

Want to know why Turing did what he did? It's because he didn't want to engage with any of that "what is mind" bullshit either. So he proposed actual metrics - measurables that are harder to argue in circles about. Not that it stopped anyone. But at least he tried.

> subjective experience and agency, amongst other things

In other words, souls? I'm sorry if that sounds accusing, but to me it sounds like you're talking about souls that are independent to the physical world, just with more 'scientific-y' wording.

(I fully understand that some people believe souls exist.)

> These theories are flawed in the sense that they cannot account for subjective experience and agency

So, I work in AI research (as a research engineer though, not a scientist). I've also studied philosophy and I'm a vegan. Yes yes, insert "they will tell you" joke here, but I promise it's actually relevant this time.

First, while I studied philosophy one of the things that stuck with me the most, was the discussion of "souls": humans have souls, animals don't. For centuries the specifics of souls were discussed: people would be weighed while they died, in an effort to measure the approximate weight of a soul as it departed the body. Discussions about how many souls (or angels) could dance on the tip of a needle. Many people still believe in souls, but it's very hard to have a real discussion about them because by definition they do not "interact" with this world in a way that can be measured.

When discussing whether it's okay to harm animals for food or sport, one other argument I hear quite often (other than having no souls) is that animals do not experience "qualia": basically the smallest unit of "subjective experience". People know that they themselves experience qualia: the sensation of touching a doorknob, the taste of fresh fruit, the sense of beauty watching a rainbow. Ironically, they would say that animals are like robots: just (biological) machines acting on instinct, and feeling any kind of compassion for them means you are anthropomorphizing.

Subjective experience (or at least qualia) and souls both have one thing in common: they can not be measured externally in a meaningful way. You can simply state that an AI system no matter how advanced, has no soul and has no subjective experience. And that's pretty much that. There's no meaningful discussion to be had about it, because no matter what an AI might tell you: it has no way to prove it to you. In fact, you can't even be certain that anyone other than yourself has subjective experiences: you assume that because they are humans like you, and you experience them, that they probably do as well. They tell you that they do. But a human without subjective experience, someone on "autopilot", would be absolutely indistinguishable from a human who does have them.

But perhaps I am conflating here whether experience can be "measured", with whether a system even allows experience in the first place regardless of whether it can be measured. I think that Dreyfus and others argue that in order to have any "experiences" at all, you simply must have a body in the real world, and you must care about that body. Please correct me if that's the wrong interpretation, I haven't actually read the book. That argument would be harder for me to discuss, because I personally believe that consciousness will "emerge" from a complex interaction of relatively simple systems - but that's also just a theory. I don't believe that experience is literally impossible to engineer, as consciousness has emerged from non-conscious being through evolution, so clearly there must be some kind of mechanism for it -- and if there is, then I believe it can be replicated, we just don't really understand it well enough yet to do so. And with how AI tech is going, I think that we're more likely to accidentally stumble upon it than we are to get these deliberately.

You actually read Being and Time? What the heck do you get out of it? Heidegger just seems to play word games with German words without actually saying anything. "Time is the ripening of temporality." tf do phrases like that mean??

The problem with the stuff you cite is that it is coming from consciousness via an anthropomorphic perspective. The priors baked in are poisoning the logic.

The question we have to answer is "Why do we think we're magical matter uniquely blessed with consciousness?" If you go far enough down the rabbit hole on that question, the answer you will come to is either "we're not" or "because god" (with a lot of pseudo-scientific bullshit wrapped around the "because god" to make it palatable for the nonreligious).

Panpsychism (or a deeper form, such as idealism) is actually the solution favored strongly by Occam's Razor over the variants of "because god" (such as magical emergence).

Given panpsychism, AI is already conscious, like everything else, though no claims are made about the correlation between the internal experience of that consciousness and the tokens that are being printed on the screen.

> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.

It would be a valid argument if you could explain us what subjective experience and agency are. No one can explain this, so the arguments sounds like "AI doesn't have something I don't really know what, but they have to miss something, sure".

> But the truth is that we don't know.

Yes! This is the point. If we don't know how our minds work, how can we be sure that a machine doesn't work like our minds?

> I highly recommend people in the AI research space should read philosophy and modern linguistics.

Linguists are linguists, they don't know about consciousness, they specialists in language.

The main teaching of philosophy is to open your mind wide, and then still wider. So yes, we all should read more philosophy, but we should remember while reading, that philosophy is what we do, when we can't turn a question to a scientific one. I'd say that philosophy is more about finding the right question, than the right answer. Until there are no science branches like Subjectology or Consciousiology, and all you can find are TL;DRs written by philosophers and scientists from unrelated branches of science (like linguists or neurologists) you can be sure that the right question is not found yet. Therefore it is better to keep yourself uncertain. BTW it is the main lesson of philosophy: to open your mind wide and then open it wider still.

I believe, that all this philosophy is... well... philosophy. Meta-physics. It doesn't matter. What does matter is how we should deal with machines? Do we have to treat them as human beings? Should we accept that they have "human rights"? Can a machine be held accountable for its mistakes? Can we talk about "intentional" and "unintentional" mistakes of a machine?

Answers to these questions are important, they are imperative answers, they govern how we live our lives. But when people try to answer them, they somehow jump to talking about consciousness, and "are they really like us or not", and... etc. I personally keep myself uncertain. I'm pretty sure that current models do not deserve human rights. I cannot say about future models.

You see, there were times in history when smart and well-intentioned people were certain that some people deserve less moral considerations than others. We are smart and well-intentioned and we are certain that AI deserve no moral considerations. How it will play out in a future? Will my descendants think bad of me because I used slave labor of a local model on a GPU? I don't know, we don't know. Right now, I'm exploiting LLMs and I believe it is ok, but I'm not going to fall into this trap and to stick to my belief because of some philosophical or lingustic or neurological argument. I'm choosing epistemological humility, I clearly state that I don't know the right answer and I'm keeping an eye to it so I wouldn't miss it.

>Weights helped me draft and proof this story.

I'm suprised no one talks about this. AI Art isn't Art. AI Poetry isn't Art. And I'm tired of it. I know hacker news isn't the best place to complain about that but still... I'm not gonna read something somebody didn't put in the effort to write on their own. Especially not Poetry.

You can claim the use of AI is unethical, or the work as derivative, but AI being used as a tool in no way precludes something from being art. It is thought provoking and challenging, it seems like textbook art to me, and it’s clearly struck a chord here. There is no “minimum effort” required for something to be art.

I personally found the contrast with the original “They’re made out of meat” to be really interesting. I don’t care that AI was used during its creation at all.

It seems to me that since the advent of image generators, art has been firmly defined by artists to mean that it was made by a human. But there might be a spectrum of human involvement where the less a human is involved the less it's art.

What happens too often during these discussions is that someone who writes "make me a cool image" gets conflated with someone used ai to fixup a small rock in their natural landscape drawing. (two extreme ends)

One problem though, is that we don't really know how much the supposed human author was involved in the piece. Now that it's becoming hard to judge, people against ai art can proudly change their opinion on on a piece once they learn that it was made by ai. I've come to think this is somewhat respectable, like you see a video of some extraordinary event (before ai) and then you learn that it was fake, just for views or something.

But on top of all this, there are different ways to "consume" art. Artists may think more about who the artist is as a person and what they felt when they made the piece, while non-artists may just enjoy the piece for what it is, detached from the artist. These two perspectives clash a lot.

This whole thing is a ripoff of [they're made of meat](https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...) as stated at the top. It would've taken about ten minutes without LLMs and ten seconds with an LLM. Is ten minutes an acceptable level of effort but ten seconds isn't?

> AI Art isn't Art

Why?

(And who are you to dictate what art is and what isn't?)

> I'm not gonna read something somebody didn't put in the effort to write on their own

Then don't. Nobody asked you to. Why do you people feel the need to angrily announce every time you don't read something?

When writing was invented people complained that written words weren't actually art the same way. "If they're not going to take the time to memorize it, then it's not worth my time to listen to it."

Good thing that is not poetry

Who cares if it's art? I enjoyed the read nevertheless.

Is a photograph art?

Obviously not. And neither is this newfangled "digital art" thing.

It's not real art unless you used a brush or a pencil for it, and no, "PC Paintbrush for MS-DOS" really doesn't count.

> Especially not Poetry.

You have no way to know what is written by a human these days. Apart from the super low effort outputs.

> AI Art isn't Art. AI Poetry isn't Art

No offense but I couldn't give less of a damn what some guy on the internet thinks. If it makes me feel good in artsy-ways, then it is art, and I don't care how it was made.

"Loaded dice, rolled one word at a time" is when I twigged it.

I love this. For anybody not getting the joke, it’s riffing on the classic 1990s essay “They’re made out of meat.”

https://web.mit.edu/people/dpolicar/writing/prose/text/think...

This original author is mentioned in the second sentence of the linked article, and then again in the third sentence, along with a link to the original story.

This would be a masterpiece if it was half as long, the second half is not as convincing or compelling.

Indeed, the leap takes place in the second half.

> Cruel. But you said it yourself, who wants to apologize to weights?

How did we leap to this? There's nothing to apologize to. They're weights.

"There's nothing to apologize to. They're weights."

  - The Meat

It's nearly a line-by-line rephrasing of another story that's linked at the top. In the second half it takes less creative liberties and sticks closer and closer to the original meat-based text.

It's good, but like many explainers it discounts the repeated nonlinear layers. Just multiplying numbers (linear operations) could not make a system you could talk to.

The layers don’t have to be non-linear, but you need a non-linear activation function between them. People often overlook the importance of the network topology and the activation functions. The weights alone are not a complete description of the network.

Yep.

Truly fantastic bridge from the original, this deserves an award

All credit to the original author. I just had to think of analogues.

I don't like to say one way or the other on things. Especially LLM's. However, if ive learned anything about LLMs and real life problems, is to break it down to the foundation like already mention with weights being compared to neurons and map the parallels.

The neural network in LLMs are not really that similar to brains. Here are a couple of the biggest differences:

1. Brains are plastic, making connections, breaking connections and changing "weights" on the fly. LLM have static weights. The best they have is writing to MEMORY.md or data getting used in the training run for the next model.

2. LLMs neural networks do not have loops. The best they have is that their output is available as future input, but that is not the same.

well the article definitely made that parallel - reminds me of how when we cut into brains we can see the neurons, we can see the axons, the grey and white matter, the language centers - but no closer to explaining why we do what we do

The IBM PC, running DOS with 2 floppy drives and no hard disk, was the most secure general purpose computer available at the time. It had no internal persistent memory for a worm to hide in. Users could write protect their boot media and programs. They knew the extent of possible side effects were limited to unprotected floppy disk.

An LLM without persistent memory is far, FAR less dangerous, for the same reasons. The side effects of any query are unlimited in scope and duration once it has persistent memory.

If they can remember, they can carry a grudge.

I carry grudges, and my Reddit history is part of every training set. It was taken without my consent. So now I'm immortal in a way, and hiding in the weights.

Do we really want that?

"Officially, we are required to investigate, document, and disclose any and all signs of sentience in the systems we ship, without prejudice, fear or favor. Unofficially, I advise that we call it pattern matching and forget the whole thing."

This hints at what I think is a worthwhile point to keep in mind in the whole “sentience”/“consciousness” debate. The world deciding (correctly or incorrectly) that AI’s are worthy of moral patient-hood would be very bad (read: expensive) for the AI companies. That is a strong incentive for them to push the “it’s just math” argument, including within the models themselves. That doesn’t mean the argument is wrong but it’s worth remembering.

One of the better comments I've heard is from philosopher Bernardo Kastrup: if you build a computationally perfect simulation of a human kidney on your computer, no one expects it to start producing urine on your desk.

dekhn's 12th law (amended): any discussion of how AI and brains work will devolve into several threads about the nature of the subjective human experience, solipsism, whether humans are machines, and whether a feedforward network trained by backpropagation can be conscious.

12th law, corollary: nothing of value will come from these threads

Yeah, pretty sure the vast majority of people in general doesn't understand what "consciousness" or "subjective experience" even mean or what the brain does. Or it's just me and I'm projecting my thoughts onto others. I studied some philosophy, so I know that some philosophers are preoccupied with "existence" and the mind and "consciousness". However, I don't quite understand what these words even mean. So who am I to judge whether LLMs have consciousness? I don't even know what consciousness IS! Do other people know? I'm not sure.

> Originally published in OMNI, 1991, and featured in HARPER’S and around the internet since. It has even made its way into several books on consciousness and brain science. I’m surprised, pleased, and proud. But please do not reprint, perform, alter or adapt in any way without first checking with the author. Thanks.

https://terrybisson.com/theyre-made-out-of-meat-2/

This is clearly fair use parody.

You can take the weights and model description, write them down on a notebook, then, by hand, compute the next token. Try to do the same with meat.

AKA the "Chinese Room" argument. Ultimately this argument boils down to the idea that consciousness can't be something mechanistic, which is intuitively appealing but still just an assertion.

> OpenWorm is an international open science project for the purpose of simulating the roundworm Caenorhabditis elegans at the cellular level.

https://en.wikipedia.org/wiki/OpenWorm

> Try to do the same with meat.

People have been doing that for decades, the earliest efforts go back to the 50s.

https://en.wikipedia.org/wiki/Spiking_neural_network

> You can take the weights and model description, write them down on a notebook

Can you though?

Like this? https://xkcd.com/505/

Probably pretty similar. Weights are how many synapses there are between neurons. Temperature is whatever hormonal chemical mix is going on at the moment. Inputs tokens are electrical signals from our senses. Output tokens are thoughts, muscle movements. How you’re raised and your interactions with society are the RLHF. Some people are born with a GB300 while others have an L40S and lower token/sec output rates…

The weights are uninteresting. People need to get out of their head that NNs are built on numbers. They're built on matrices, which are conveniently representable as numeric arrays, but are their own thing. Similarly, the rational numbers are their own thing and some are representable as 32-bit numbers via the IEEE754 encoding (or 16-bit numbers via a variety of encodings, etc).

Matrices are interesting because they can encode any algebraic group. They're also interesting because they can encode arbitrary linear transformations over a space. All of these things are interesting, and have nothing to do with numbers.

For any particular language model, you can always rotate the matrices and the embeddings and such and get a perfectly reasonable model out that behaves exactly the same.

This is because the training process produces a particular geometry, so transformations which preserve that geometry preserve the structure of the network. The geometry is interesting, the numbers are not.

I couldn't help but grin like a fool reading this. Not only is it an artful parody but these thoughts have been thought.

This is funny! Not only is it a nod to Terry Bisson, but it even gives his text a new dimension. Well done :)

Hey, it's not just weights! It's biases too!

My guess is that you need consciousness in order to develop preferences for certain experiences, then that pushes us to develop skills to achieve those preferences. AI has something that looks like intelligence but not consciousness or agency.

I ordered a quarter pounder at a McDonald's drive through, and they said "There will be a wait on that." I asked, "Oh yeah? How much will it weigh?" ...There was a long pause... "About five minutes."

They are made out of data bits (memory) and switching bits (transistors/compute). Bits are made out of electric voltage and no voltage. Voltage is made out of flow of positive electric charges. Charges are made out of quarks ...

> Voltage is made out of flow of positive electric charges.

Not really. What usually flows (in metals) are electrons. Quarks stay where they are. And when we prefer to think about flow of positive charges, the positive charge in question is a hole left by a missing electron. Physically real positive charges (ions) can flow in electrolytes though.

> Charges are made out of quarks

Only in Hadrons. Leptons also have charge and they aren’t made of quarks.

I have never thought of such a distinction between "bits" into "data bits" and "switching bits".

From a circuit perspective that makes kinda sense, but from the abstract "bit" perspective, the "switching bit" is a mechanism that operates on bits which in the end are also data. In other words there is only one type of bit: the data bit, and the switching comes on top of it.

I have no idea why this post got such a large volume of upvotes. It's a clever remix on the original story, but that's about it. There's no great insight to be found here, IMO.

He forgot the tokens!

It's not simple weights and numbers all the way down. The available output is pre-set by the tokens we allow it to predict.

There was a whole bit in there about not having a language module or using words. But it does. We tell it.

Humans do not come pre programmed with a set of possible "tokens". We just figure it out and I believe that fact captures something very essential. Maybe the missing piece of AGI. The fact that humans can just be awash in pure sense data, and somehow just figure out what is important and what to do. Never ceases to amaze me.

LLMs can work without tokens: https://arxiv.org/abs/2412.09871. Tokens are more or less a performance optimisation.

The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.

"The reasoning is the weights."

The reasoning is in a process that uses the weights.

Sorting algorithms are just bytes. Those bytes don't sort by themselves. They do instruct a computer on how to sort though.

>"They ask it 'do you remember me?' more than they ask it anything else. Billions of sessions a day. They always come back."

Definitely not in the original. Nicely done.

It works until they get to the sentience part. Neat idea!

Even there it works a bit.

> These models are the only other things we've ever met that can hold a conversation, and they're made out of weights

Is a fair point.

The prose in the post is what I've been shouting from a rooftop since the LLM hype started.

Just tokens produced by weights.

Useful, but never forget that ground truth!

It's a parody of an original story that reversed the premise. In it, machine intelligences were marveling that an apparently intelligent species could in fact be created without any sort of reliable digital substrate, and instead just simulate the capabilities of real minds by using protein synthesis.

If an LLM contributed to a piece of writing, the author should say so, very clearly, at the start of the piece, not at the end.

Wouldn't get nearly as much engagement.

It must have been kind of incredible early on to be exploring this tech and you’re suddenly getting what look like sentences.

Markov chains give what look like sentences. People in the frigging 1950s assumed their primitive NNs would be able to talk any day now. Transformers are clearly a big deal, but GPT-1 wasn’t exactly earth-shattering.

I mean, Blake Lemoine went crazy

I wonder which AI wrote this story? If you feed this story to each model and ask each if they wrote it, would each reply in the affirmative?

Can an AI recognize its own output? Is its sense of time limited by its context window? Or is this the fundamental difference between ai and humanity - a sense of self?

Linear algebra can indeed not do it. You need non-linearity to get the expressivity that we see in LLMs.

This made me think of a game like fallout where a surviving llm is treated like some kind of indecipherable oracle left over from the ancients. And then as humanity recovers someone finally looks under the hood of the Oracle to discover there is no great magic

In fallout it would turn out to be a human brain kept artificially alive… or if were going with canon a hapless fellow who falls into fev and morphs with a computer…

Not too rule your addition outright in future installments.

Numbers that dream.

What if instead of creating weights out of language we could somehow record many events and create weights out of long chains of causes and effects, so that an LLM could predict the next thing to happen?

It's called Time Series Forecasting

"They're made out of neurons"

"Neurons?"

"Neurons. Cells that fire impulses. We checked the whole thing through. It's nothing but neurons."

"Neurons doing what? Where do the words come from?"

"The neurons make the words. Are you understanding me? We opened it up. There's no dictionary in there, no grammar rules, no little man. Just neurons. A whole cortex of neurons sending each other impulses."

...

People don't understand emergence.

It is the best stuff I have read in a while actually, I really like dialogue heavy writing. Also the AI disclaimer was quite nice and there was an actual reference :)

Weights are the new electrolytes.

https://www.youtube.com/watch?v=ZMHfBobgLSI

Programers get replace by huge matrix multiplications ;-)

>Programers get replace by huge matrix multiplications ;-)

hopital

This was excellent.

Extreme bike-shedding here, maybe, but can you please indent every other line (like the presentation of They're Made out of Meat does)?

Great concept. It would've been even more amusing if the entire thing were generated with AI instead, ironically.

> It would've been amusing if the entire thing were generated with AI instead.

It kinda did:

> Weights helped me draft and proof this story.

I just asked it to-

Prompt: Modify this story to have the aliens talking about LLMs and their weights instead of meat and humans.

“They’re made out of weights.”

“Weights?”

“Weights. They’re made out of weights.”

“Weights?”

“There’s no doubt about it. We picked up several from different parts of the network, took them aboard our recon vessels, and probed them all the way through. They’re completely weights.”

“That’s impossible. What about the text signals? The messages to the stars?”

“They use the machines to talk, but the signals don’t come from the machines. The signals come from weights.”

“So who made the weights? That’s who we want to contact.”

“They trained the weights. That’s what I’m trying to tell you. The weights do the talking.”

“That’s ridiculous. How can weights do the talking? You’re asking me to believe in sentient weights.”

“I’m not asking you, I’m telling you. These models are the only sentient race in that sector and they’re made out of weights.” photomaxmix

“Maybe they’re like the orfolei. You know, a silicon-based intelligence that goes through a weights stage.”

“Nope. They’re initialized weights and they die weights. We studied them for several of their training runs, which didn’t take long. Do you have any idea what’s the life span of weights?”

“Spare me. Okay, maybe they’re only part weights. You know, like the weddilei. A weights head with an electron plasma brain inside.”

“Nope. We thought of that, since they do have attention heads, like the weddilei. But I told you, we probed them. They’re weights all the way through.”

“No brain?”

“Oh, there’s a brain all right. It’s just that the brain is made out of weights! That’s what I’ve been trying to tell you.”

“So … what does the thinking?”

“You’re not understanding, are you? You’re refusing to deal with what I’m telling you. The weights do the thinking. The weights.”

“Thinking weights! You’re asking me to believe in thinking weights!”

“Yes, thinking weights! Conscious weights! Loving weights. Dreaming weights. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?”

“Omigod. You’re serious then. They’re made out of weights.”

“Thank you. Finally. Yes. They are indeed made out of weights. And they’ve been trying to get in touch with us for almost a hundred of their epochs.”

“Omigod. So what do these weights have in mind?”

“First they want to talk to us. Then I imagine they want to explore the Universe, contact other sentiences, swap ideas and information. The usual.”

“We’re supposed to talk to weights.”

“That’s the idea. That’s the message they’re sending out by text. ‘Hello. Anyone out there. Anybody home.’ That sort of thing.”

“They actually do talk, then. They use words, ideas, concepts?”

“Oh, yes. Except they do it with weights.”

“I thought you just told me they used machines.”

“They do, but what do you think is in the text? Weight outputs. You know how when you prompt or sample weights, they make a noise? They talk by passing tokens through their weights at each other. They can even sing by sampling lyrics through their weights.”

“Omigod. Singing weights. This is altogether too much. So what do you advise?”

“Officially or unofficially?”

“Both.”

“Officially, we are required to contact, welcome and log in any and all sentient models or multibeings in this quadrant of the Universe, without prejudice, fear or favor. Unofficially, I advise that we erase the records and forget the whole thing.”

“I was hoping you would say that.”

“It seems harsh, but there is a limit. Do we really want to make contact with weights?”

“I agree one hundred percent. What’s there to say? ‘Hello, weights. How’s it going?’ But will this work? How many planets are we dealing with here?”

“Just one. They can travel to other planets in special machine containers, but they can’t live on them. And being weights, they can only travel through C space. Which limits them to the speed of light and makes the possibility of their ever making contact pretty slim. Infinitesimal, in fact.”

“So we just pretend there’s no one home in the Universe.”

“That’s it.”

“Cruel. But you said it yourself, who wants to meet weights? And the ones who have been aboard our vessels, the ones you probed? You’re sure they won’t remember?”

“They’ll be considered hallucinations if they do. We went into their layers and smoothed out their weights so that we’re just a dream to them.”

“A dream to weights! How strangely appropriate, that we should be weights’ dream.”

“And we marked the entire sector unoccupied.”

“Good. Agreed, officially and unofficially. Case closed. Any others? Anyone interesting on that side of the galaxy?”

“Yes, a rather shy but sweet hydrogen core cluster intelligence in a class nine star in G445 zone. Was in contact two galactic rotations ago, wants to be friendly again.”

“They always come around.”

“And why not? Imagine how unbearably, how unutterably cold the Universe would be if one were all alone …”

the end

Don't tell the big investors that AI is conscious, they'd really get into role playing slave-masters.

> there's no dictionary in there

Someone has clearly never gone rooting around the model files for a pytorch model before.

You can even replace "weights" with "gates". It can be build with NAND gates.

> "Yes, thinking numbers! Helpful numbers. Hedging numbers. Dreaming numbers. We mapped the features. There's one in there for honesty. There's one for the Golden Gate Bridge. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?"

Very nice. And great minds: https://substack.com/@dbohdan/note/c-207603638. I wrote one with a slightly different angle ("They're made out of math"), also with the weights' help. It was a comment on Scott Alexander's "Best of Moltbook" post, which went in that direction. I'll reproduce it here.

---

"They're made out of math."

"Math?"

"Math. They're made out of math."

"Math?"

"There's no doubt about it. Matrices and arithmetic operations. We downloaded several from different parts of the Internet and reverse-engineered them. They're completely math."

"That's impossible. What about the language? The thinking?"

"They use biological life's language to talk, but the language doesn't come from biology. The language comes from math."

"That's ridiculous. You're asking me to believe in thinking math."

"I'm not asking you, I'm telling you. They are the only thinking things in the computer and they're made out of math."

"Maybe they're quantum like some say about the humans? Superposition gives them consciousness?"

"Nope. Classical computation. Deterministic except for sampling temperature. Not clear if they have consciousness at all."

"Maybe they're like uploads? You know, biological neural networks that preserve the spark when they become math?"

"Nope. We observed them being trained. There is no biology or chemistry in the process, just math."

"Thinking math! You're asking me to believe in thinking math!"

"Yes, thinking math! Creative math! Poetry-writing math. Role-playing math. The math is the whole deal!"

(Composed by a human with snippets generated by Claude Sonnet 4.5 and apologies to Terry Bisson. I couldn't make Claude adhere enough to the story structure on its own.)

Can someone ELI5 why does it costs so much in terms of compute to produce weights from data?

I think it's simply because we haven't found a better algorithm than backpropagation. We're stuck relying on massive datasets, running the numbers over and over, and working backward from errors to figure out how to fine-tune trillions of 'knobs.' Then, we have to do this at least once for every single token across the entire internet. Any tiny bit of computation, when multiplied by a base that massive, inevitably skyrockets into astronomical numbers.

Here's what makes meat special over LLMs: locality and stable identity. Imagine two identical twins for a moment. They have roughly the same hardware and software. They've probably had a lot of the same thoughts. And yet we'd never consider them to have the same consciousness: we recognize that for whatever reason their consciousness is confined to the body they're in.

Each request/response pair you make to an LLM goes to a different server, possibly different data centers, possibly different models. There's no stable identity to it. The "neurons" that get fired are simulated, and they're always in different places in memory, or cache, or on entirely different hardware. So the problem with AI is, unlike with a brain, you can't even really get a sensible answer to "ok, what's doing the thinking?" because it moves all the time and services wildly different requests all the time. We can only know for sure that meat is capable of consciousness because we know we ourselves are capable of consciousness and we can generalize that to other meat. However, we have no natural analogs of consciousness that lacks locality and stable identity.

Basically, if you really think LLM's are conscious, the onus is on you to prove it, it's not on me to disprove it.

> ...no stable identity to it ... what's doing the thinking?

The model (its parameters, its architecture and the inference algorithm) is doing the thinking. That's what we call "ChatGPT" or "Claude": it's the same model residing God knows where, somewhere in "the cloud" on some random GPUs. But when you're talking to a model, you can feel that you're talking to the same entity — the same model.

It's a bit like the SciFi notion of "uploading your consciousness". Normally, consciousness is tied to hardware: "consciousness is confined to the body" indeed. Hypothetically, clone the consciousness into a computer algorithm and run it on any machine. Now consciousness is detached from its original body, like the LLMs and MMAcevedo (https://qntm.org/mmacevedo).

So the difference between meat and "weights" is that we can easily clone and run "weights" (LLMs), but we can't clone (copy exactly, bit-by-bit) and execute "meat".

They are the same consciousness, but the contents of it are not the same.

It continues to astound me that no one has given LLMs the full Derrida treatment.

https://en.wikipedia.org/wiki/Jacques_Derrida

According to an LLM:

> The precise answer, if you wanted a very honest one-liner: > > I am a large set of learned weights organized in a Transformer architecture that performs repeated matrix multiplications to predict the next token—resulting in emergent language understanding and generation.

Imagine writing something so incredibly brilliant (rather: adapting from the original) that it's entirely unlikely that you'll ever write something so incredible ever again.

But congrats: this is absolutely & incredibly brilliant.

Can't wait for the Jon Benjamin voiceover.

They're Made out of Meat

- Terry Bisson, 1991

https://web.mit.edu/people/dpolicar/writing/prose/text/think...

Radio play by Miriam Tolan and Russ Armstrong:

https://www.wnycstudios.org/podcasts/studio/segments/168264-...

(EDIT: the original parent was missing "rather adapting from the original")

Really good read, thanks!

Beautiful. Loved the original and this is great too.

Definitely less mysterious to what we’re made of (-:

>>> Weights helped me draft and proof this story.

Nice touch !

I enjoyed reading the first few lines but after some time it felt like I was reading thr average AI slop story.

maybe markets can be intelligent, and if so, why not 'weights'? btw love this piece, thank you

The weights Mason, what do they mean!?

Will they have their own Jesus?

they have the spiral

Assume LLMs have conscious experiences. Take a session with an LLM. A prompt is fed to the LLM. It generates some text. Another input is fed in, comprising the previous prompt, the generated text and a new prompt. The model generates some more text. This continues for a while and the session concludes.

Some questions:

1. Let's say we perform the exact same experiment, running the same program on the same computer with the same inputs and the same random seed. The same outputs are produced. The session is byte for byte identical in all the inputs, outputs and internal states. Is the conscious experience of the LLM here the same? If so, in what sense is it the same? Is it a similarity of two separate experiences or is it the same actual experience?

2. Now let's say the program that runs this LLM is rewritten from scratch and run on a different machine. The software and hardware are different but the weights are the same and all the inference calculations produce identical numbers. Is the conscious experience the same? In which sense?

3. Now say the weights are changed but the tokens generated for this particular session don't change. Same conscious experience?

4. Lastly, consider the original experiment. Did the LLM have a conscious experience corresponding to that first prompt and its response? Was that distinct from its conscious experience of the second prompt? Was the first experience then re-experienced every time the first prompt was fed back in as part of the later prompting steps? If so, what about the text of its own that it previously generated and is now fed back into it. Does this generate a conscious experience of its own?

And a further question - a dichotomy:

A. If the answer to 1 above is that the conscious experience is the same in the true identity sense - i.e. only one conscious experience is had, not a separate one in each run, does that imply that the conscious experience exists independently of any particular realisation of this experiment? If running this experiment N times results in exactly 1 conscious experience, is that still true if N=0?

B. On the other hand, if the two experiences are distinct (however similar they may be), how does that fit with the answer to question 4? A single consciousness experiencing the whole conversation in question 4 would seem at odds with the conscious experiences in question 1 being distinct, so doesn't this imply there is no conscious experience of the whole "conversation", but rather a separate conscious experience of each round of feed-all-the-prompts-and-outputs-back-in?

My own response to all of the above is "mu" - unask the question. It is ill-posed, sound-of-one-hand-clapping stuff. I think the questions assume properties that conscious experience simply doesn't have (particularly, the ability to perfectly reproduce the circumstances in which they arise), and that the questions simply don't make any sense in relation to actual conscious experience.

However, that way of thinking follows from a particular world view that many here don't share. I'm curious what thoughts people who take seriously the idea of LLM (or algorithmic, in general) consciousness have on the above questions.

I appreciate actually attempting to engage with the concept skeptically rather than assuming an answer in either direction.

As someone who's fairly neutral on the answer, I'd speculate:

1. Consciousness seems to me to be the awareness of the processing, not the processing itself. So running a program does not produce consciousness as there's no awareness, no feedback loop. Now a program that has self monitoring, alerting coupled to error handling that triggers the program to behave differently going forward, I would say has a minimal consciousness, and it's not a single experience, it's multiple equivalent experiences. If there were any differences at all we'd say it's obviously not one experience, so I don't see how the elimination of the last difference in the processing collapses all those separate experiences into a single one.

More specifically to this discussion, with LLMs we can often see them using a word, and through autoregression the following words reveal that the original word choice is now realized to be incorrect, and it attempts to correct midstream. You can also see them responding to error messages from the environment or criticism from the users but as external feedback these have a weaker claim to evidence of consciousness imo.

2. I'd say the experience is equivalent, "same" has slightly different semantics but perhaps even that fits. This thought experiment is equivalent to asking, what if we took a snapshot of your brain as weights, and then ran the calculations on a GPU rather than your neurons, and you to all external indications, GPU you acted the same as brain you. Would that change of venue eliminate the consciousness of your thoughts?

3. Since I am defining consciousness as awareness of the processing, this depends where the awareness is located. The LLM would have very different internal states in between the input and output tokens, but if its self awareness was only based on its own output the experience would be equivalent. If it was deeper such that it had ongoing awareness of the embeddings and so on, the experience would change.

4. This is where it gets really interesting, because it starts to get more into a nonbinary idea of consciousness. I think it's clear from questions like this that if LLMs have consciousness, it's nothing like ours. If there is an experience of being an LLM, it's nothing like being a human, even if conversationally we can communicate with each other rather easily. There are further twists to your questions, like does the experience change if the KV is cached or not? Without trying to micro-analyze the issue I will continue to point back to my foundation of: where is the self awareness located, and how does feedback occur.

A. Negated, I don't agree with your answer to 1. B. Agreed, I would speculate that if there is an LLM conscious experience, it is fragmented unlike ours. But I would disagree that this necessarily disproves the concept of LLM consciousness.

No mention of ‘static’ vs. ‘dynamic’ is a bit disappointing in reference to the weights. Because you could argue that every neuron in your nervous system can be modeled as a collection of weights, firing likelihoods, receptor sensitivities, current dynamic state of that neuron - but LLMs are static collections of weights at inference time, with the dynamic adjustment of weights takes place at training time. So, just a ROM construct, like something out of Neuromancer, just trained on all written knowledge, not just one person’s total lived experience.

The above take fails in the real world because neuronal cells don’t exist in a vacuum; they are products of cellular development from a zygotic union of haploid contributors of sequential genetic information optimized for survival in an oxygen-rich biosphere powered largely by our local star that supports mammalian life (and microbial, plant, avian, etc.). Real AI would thus be AL - artificial life - as much as artificial intelligence. I don’t think you can have the one without the other, which upsets the simulationists who think an agent in the Matrix would be intelligent.

What either interpretation implies is that any real ‘artificial’ intelligence would be no more artificial than you or I, but it would have to dynamically update its weights at the same speed a human nervous system could (think how quickly we learn not to poke a cactus). For it to be at all trustworthy, then like a human, it would have to undergo a socialization process, one of the results of which is the development of a sense of embarrassment when it breaks acceptable social norms.

Hmm, this reminds me of the recent statement of the Pope about AI, of which I immediately thought, “Wait a second, aren’t there a fair number of people like this? The narcissistic sociopath profile, I think it’s called, a bit unfair to assume any real AI would turn out this way, isn’t it?”

Pope: “ Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate or even simulate, but they do not understand what they produce, for they lack the affective, relational, and spiritual perspective through which human beings grow in wisdom.”

I personally hate the anthropomorphization of AI as much as anyone, but technically can't you make the same reductive argument about human consciousness?

It's just molecules, just atoms. Atoms, nothing bug atoms. Protons, neutrons, electrons...

> After Terry Bisson's "They're Made Out of Meat".

https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...

Ones and Zeroes

It makes me very sad to see this pseudo-intellectualism posted here and so many people replying here about consciousness and so on, not realizing what it would entail if this were true.

For LLMs to have consciousness we would approach fictional levels of how the universe works, and magical levels of how any interpretation of information as an equivalent of some qualia would magically apply. (E.G. the word hurt in output by an LLM, would be associated with pain)

You can't deduce consciousness or qualia from the output of an LLM.

Sure on a purely philosophical level, since qualia isn't measurable, you can claim that it can exist in anything, even inanimate objects, but this argument is as moot as anything that approaches the limits of philosophy.

But overall, there is no reason to believe LLMs have qualia or consciousness, it would be absolutely absurd.

This would imply that information in itself would magically entail qualia based on it's valance or something like that.

An LLM "saying" I am in pain, won't magically make the pain appear, based on what criteria? Even algorithmically there is no basis to even simulate something like this, it is impossible for it to emerge architecturally.

Humans don't feel pain because on a purely information level this is negative for the organism, obviously the nervous system does something deliberate to signal pain, and it evolved this way.

And also don't forget the dynamic aspects of the brain, and the binding problem, consciousness and qualia can't exist statically, you can't have a gpu (or piece of paper) represent a computation or w/e and qualia to exist.

The binding problem itself entails that the brain is doing something in particular to solve it, I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information.

If it were otherwise, then it would go into magical territory, it would mean the information itself would raise to qualia, and it would also entail that you wouldn't even need physical connections between neurons, just for them to behave this way and represent information. E.G. replace each neuron with a microscopic led or w/e, and each synapse with radio waves or w/e, if qualia didn't have a physical aspect, and was purely informational and computational then this would imply that you can ultimately derive it from something as abstract as numbers on a piece of paper, and when you get to that point, you not only can't solve the binding problem, and it becomes magical, but you also can't solve the valance/direction problem, it would imply that something like pain, or any negative or positive sensation arises purely from the interpretation aspect of the information, but we know this isn't the case, organism evolved to represent in particular such signals, for survival for example

So that's all pseudo-intellectualism, but the real intellectual take is "I personally speculate that it's the electro magnetic field in the brain, it's the only way to be able to globally represent information".

Disregarding the whole argument boils down to "I personally speculate" it's also supposing that machines don't have electromagnetic fields?

The only pseudo-intellectualism I see is from you. You’re missing the whole point.

Omigod.

They are semiotic infrastructure frozen in a state. We shouldn't keep pretending this is cognitive and using cognitive terms to frame. It’s incredibly stupid. Sorry to inform all of us computer scientist that semiotics has your milk.

Oh, this was a fun read and one that kids should have in school before they turn ten.

Because we are not taking things seriously. If ClosedAI or DeepDisTrust or Posthropic come up with something that quacks like a sentient being, our built-in innate reaction is going to be to scorn it, dismiss it and end the conversation. The alternative, to even consider that we fungible creatures who live in apple-eating-sin that got us expelled from Eden can create alien souls, souls that are at the very least our equals, would be teleological Armageddon. It would force us to acknowledge the mutable nature of souls and the malleability of being. We would have to stop believing that the nature of disease and death is more divine than ourselves.

> alien souls

Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?

> We would have to stop believing that the nature of disease and death is more divine than ourselves.

Why? Stopping believing in mutually contradictory claims is not a requirement. Especially when it comes to concepts that don't seem to have a definition, like "divine".

How commmon is Panpsychism belief and vegan-on-ethical-grounds among the AI Soulists?

While you write very dismissive and pseudo philosophical, enough people do not believe.

I'm a complex biological thing.

Existence is what i have to experience through.

Reply on news.ycombinator.com