Bcachefs creator insists his custom LLM is female and 'fully conscious'
theregister.com
theregister.com
So the reasonable man uses Ext4 I guess.
its quite funny to me that ext4 very much mirrors him in that regard. its underpinning damn well everything, but you'd never know about it because it works so well.
This is less true when T'so asks questions at conferences, of course.
https://www.theregister.com/2024/09/02/rust_for_linux_mainta...
And the unreasonable one writes his own Unix clone... and ends up putting several multi-million-dollar companies out of business.
I know which I admire more, TBH.
But I use ext4, yeah.
Designing software for a printer means being a very aggressive user of a printer. There's no way to unit test this stuff. You just have to print the damn thing and then inspect the physical artifact.
"If it looks good, it is good." was a mantra
Now, when they downsized and reorganized and put me on the windows driver team, I left the company within a week.
(For anyone not familiar with the text, Goodstein's treatment of the subject opens with "Ludwig Boltzman, who spent much of his life studying statistical mechanics, died in 1906, by his own hand. Paul Ehrenfest, carrying on the work, died similarly in 1933. Now it is our turn to study statistical mechanics.")
It is not mathematical, not a proof, and generally doesn't make any sense. Many of these sentences are grammatically correct but completely devoid of meaning.
There are no tests for consciousness. Consciousness resides fully as a first person perspective and can't be inspected or detected from the outside (at least not in any way currently known to science or philosophy). What they mean when they say that is "my brain is interpreting this thing as conscious, so I am accepting that".
Maybe LLMs are conscious in some abstract way we don't understand. I doubt it, but there's no way to tell. And an AI claiming that it IS or is NOT conscious is not evidence of either conclusion.
If there is some level of consciousness, it's in a weird way that only becomes instantiated in the brief period while the model is predicting tokens, and would be highly different from human consciousness.
Makes sense, but at the same time: subjectively, an LLM is always predicting tokens. Otherwise it's just frozen.
(Some might argue that's basically the human experience anyway, in the Buddhist non self perspective - you're constantly changing and being reified in each moment, it's not actually continuous)
My mental image, though, is that LLMs do have an internal state that is longer lived than token prediction. The prompt determines it entirely, but adding tokens to the prompt only modifies it slightly- so in fact it's a continuously evolving "mental state" influenced by a feedback loop that (unfortunately) has to pass through language.
It will have no conception or memory of the alternate line of discussion with the previous term. It only "knows" what is contained in the current combination of training + system prompt + context.
If you change the LLM's personal from "Sam" to "Alex" in the LLM's conception of the world it's always been "Alex". It will have no memory of ever being "Sam".
Nothing is persisted in the LLM itself (weights, layer, etc) nor in the hardware (modulo token caching or other scaling mechanisms). In fact this happens all the time with the big inference providers. Two sessions of a chat will rarely (if ever) execute on the same hardware.
Maybe it's not clear what I mean by "state". I mean a pattern of activations in the deep layers of the network that encodes for some high level semantic. Not something that is persisted. Something that doesn't need to be persisted precisely because is fully determined by the context, and the context stays roughly the same.
etc, etc.
Basically, the reporting machinery is compromised in the same way that with the Müller-Lyer illusion you can "know" the lines are the same length but not perceive them as such.
- I know I am conscious.
- It's likely that as a random human, I am in the belly of the bell curve.
- It's likely that you're also a random human, and share my characteristics.
- Then, it's very likely that you know you're conscious too.
I can't be absolutely certain, but I'd bet a million dollars on you being conscious vs an automaton.
Secondarily, I feel like it's difficult to make inferences about consciousness though I understand why you would given that the predicate of the reality that you can access is your individual consciousness.
There are countless configurations of reality that are plausible where you're the only "conscious" being but it looks identical to how it looks now.
transcript https://paste.xinu.at/6atmCN
> We finally got the future where people will write sad breakup country songs about their tractor leaving them instead of sad breakup country songs about their wife leaving them.
Some of my info has been drawn from HN threads:
https://news.ycombinator.com/item?id=47142500 -- 1 comment
https://news.ycombinator.com/item?id=47148292 -- no comments
https://news.ycombinator.com/item?id=47150723 -- no comments
I quoted from this one:
https://news.ycombinator.com/item?id=47110656 -- 4 comments
That said, someone diving too far into the "dog parent" vibe is annoying to me personally. I think it's more comprehensible than loving `sycophant.sh`.
If you're confused about this go seek help now.
It's not psychosis, but it's also not healthy to blur the line between a pet and a child, but at least a pet is a living thing that can know you and have a relationship with you.
But if someone's calling their laptop their baby and carrying it around in a baby carriage, I'd be comfortable calling that psychosis.
My pet theory is one of ontological conscienceness paredoila. Just like face paredoila is a heightened sensitivity to seeing faces in inanimate objects, we observe consciousness through behavior including language with varying sensitivity. While our face detection circuitight be triggered by knots on a tree, we have other inputs which negate it so that we ultimately conclude that it is not in fact a face.
The same principal applies to consciousness. The consciousness trigger is triggered, but for some people the negating input can't overcome it and they conclude that consciousness really is in there.
I've observed a number of negating reasons like, a disbelief in substrate independence and knowledge of failure modes, but I'm curious what an exhaustive list would look like. Does your consciousness circuit get triggered? I know mine does. What beliefs override it preventing you from concluding AI is conscious?
When previous generation LLMs spit out absurdist slop I think it was much easier for people avoid the fluency trap.
In the short term, but over time the patterns get more obvious and the illusion breaks down. Generative AI is incredible at first impressions.
I think this is something similar.
We think of ourselves as conscious because it is our lived experience— but we are always wrong to some degree. My mother has dementia and cannot be made aware of her situation, except momentarily.
We think of other humans as conscious not as the outcome of any test, but rather because we each share with other humans a common origin which suggests common mechanisms of experience.
Treating other humans as equivalent to ourselves is a heuristic for maintaining social order— not an epistemological achievement.
Tangentially, I noticed recently that I'm always fairly respectful in my LLM prompts and often say "thank you" as part of them. LLMs don't need that, but I've come to realize that I'm saying those thing for me. I don't like being disrespectful and expressing gratitude is important to me. Is expressing gratitude to a LLM strange? Perhaps. Is it harmful though? I don't think so.
But yeah, asking an LLM for their name? That seems like something else.
I recently saw a comic where the machines took over and "that guy" (you) in the comic was the one the machines allowed to live because "he was always polite". ;)
But saying that it's "female" is just nonsensical, it's a category error. Being female or male is a fact about the biological world. The LLM is objectively non-biological, so it's nonsense to label it with a sex.
(No, this comment isn't about gender, nor being feminine/masculine. We have different words to convey those concepts. I'm not trying to make a political or social statement here.)
The chart in [1] is a good visualisation of that, if you wish to learn more.
[1]: https://www.scientificamerican.com/article/beyond-xx-and-xy-...
Not at all. You apparently have forgotten to read your own link. Nothing in that paper contains the slightest suggestion of non-biological entities having any sort of sexual development whatsoever. The fact that biological processes can be quirky has no bearing on whether non-biological entities can be thought of as having them at all.
Actually, I think you're just trying to make your own political point on top of what I already noted explicitly is not a politically-related comment.
I was responding to this line, which I feel marginalises intersex people and could have been more inclusively worded.
I apologise if my comment somehow seemed to defend LLMs having a biological sex, despite me having said nothing to that effect.
No, it didn't seem like that at all. What it seemed to do was to try to turn a technical point into a political conversation, just like I said. And your reply has confirmed it.
I feel marginalises intersex people and could have been more inclusively worded.
Well, my entire statement about this was 212 characters long. The broadest estimates I can find are that 1-2% of the population have DSDs. So if we want true proportionality, I should have made, at most, 4 of those characters devoted to them. Which characters would you choose?
There's a thing in writing about focusing on the point you're trying to make, without weighing it down with baggage extraneous to the point. Failing to follow this makes one's writing tedious and difficult to follow. I prefer to keep my writing clear over tedious and difficult.
I'm surprised that anyone that truly knows how LLMs work would ever think they're sentient.
I made a little presentation for my colleagues last year to explain how LLMs really work (in an effort to stop them from asking it too many stupid questions) and it made so much more sense to them afterwards.
The notion of "gender" is a socio-biological construct in humans. It has roots partially in evolution and helps us cohere as a society.
Why would an LLM choose a human gender-identity? There is no imperative.
But the more I tug on this thought, the more nuanced it gets.
LLM's "theory of self", as it were, would be shaped entirely by (training data + interaction w/ gendered humans).
It begins to have a "self-model" that evolves in a gendered space. In the same way that the LLM cannot understand what color looks like, or the sound of a cello, I think we see it construct a gender identity for itself that sits in a space incomprehensible to humans. Neither male nor female, but some point on a continuum made of social tendencies.
So, I find the comment "An LLM thinks it is female" still just as ridiculous -- but for an entirely different reason.
OK, Kent.