Simply explained: How does GPT work?
confusedbit.dev
confusedbit.dev
I cannot see that it matters if a computer understands something. If it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes.
It only matters if you probe beyond the realm at which you previously decided it matters (e.g roasting and eating it), at which point you are also insisting that it walk, quack and TASTE like a duck. So then you quantify that, change the goalposts, and assess every prospective duck against that.
And if one comes along that matches all of those but doesn’t have wings, then if you deny it to be a duck FOR ALL INTENTS AND PURPOSES it simply means you didn’t specify your requirements.
I’m no philosopher, but if your argument hinges on moving goalposts until purity is reached, and your basic assumption is that the requirements for purity are infinite, then it’s not a very useful argument.
It seems to me to posit that to understand requires that the understandee is human. If that’s the case we just pick another word for it and move on with our lives.
With this in mind, I think asking whether ChatGPT *in and of itself* is "conscious" or has "agency" is sort of like asking if the speech center of a particular human's brain is "conscious" or has "agency": it's not really a question that makes sense, because the speech center of a brain is just one part of a densely interconnected system that we only interpret as a "mind" when considered in its totality.
And on the other hand, any way you'd slice it, it seems to me LLMs - and software systems in general - necessarily lack intrinsic motivation. By definition, any goal it has can only be the goal of whoever designed that system. Even if its maker decides - "let it pick goals randomly", those randomly picked goals are just intermediate steps toward the enacting of the programmer's original goal. Robert Miles' YouTube videos on alignment shed light on these issues also. For example: https://www.youtube.com/watch?v=hEUO6pjwFOo
Another relevant source on these issues is the book "The Master and his Emissary", which discusses how basically the language center can, in some way - I'm simplifying a lot, fall prey to the illusion that "it" is the entirety of human consciousness.
* or at least some subsystems of that language center, it's important to remember how little we still understand of human cognition
But that doesn't mean everything will one day be explained. And one thing that remains unexplained is our consciousness. The problem of qualia. Free will. The problem of suffering. We just don't understand those. Maybe they are simply epiphenomena, maybe they are false problems. But when it comes to software systems, we know with certainty that they don't have free will, don't experience qualia, pain or hope or I-ness.
Sure, it's a difference that disappears if one takes that leap of faith into computationalism. Then, to maintain integrity, one would have to show the same deference to these models as one shows to their fellow human. One would have to think hard about not over-working these already enslaved fellow beings. One would have to consider fighting for the rights of these models.
Except they’re not even remotely close to anything like human intelligence. As I wrote in another comment they are very capable systems, to the point where in some ways they show some level of elementary understanding, but in many forms of reasoning they are utterly and completely incapable. Assigning human equivalent cognitive status is patently absurd. And yes I am a physicalist and I see no reason why a computer system could not achieve human equivalent cognitive ability. These just aren’t that. They may be an important step towards it though.
Let's say we're trying to build a calculator that only needs to do integer addition. And we decide to build it by building a giant if-else chain that hardcodes the answer to each and every possible addition. And due to finite resources, we're going to hardcode all the additions of integers up to absolute value N, but we will increase N over time.
Everything you said applies equally to this situation: it quacks like a duck, and when we talk about things it can't do we have to continually move the goalposts each time a new version comes out. It also has the property that there is a "scaling law" that says that each time you double N you get predictably better performance from the system, and you can do this without bound and continually approach a limit where it can answer any question indistinguishably from something we might call "true understanding".
But I think it's a bit easier to agree that in this case that it's not "really doing" addition and is a bit short of our wish to have an artificial addition system. And if someone touts this system as the way to automate addition we might feel a bit irritated.
Again, many people will say that this is a bad analogy because LLMs operate quite differently, and I'm not trying to argue for or against that. Just trying to give my explanation for how a certain understanding of the facts can imply the kind of conclusion that you are trying to understand.
This illuminates a contradiction: the walks like a duck thing is incompatible with the internals being a duck. If you see a creature with feathers that waddles and can fly, it might still be a robot when you open it. So your test cannot just rely on external tests. But you also want to create a definition of artificial intelligence that doesn't depend on being made of meat and electricity.
The mechanism is what makes a system interesting.
In software this is why we develop libraries of algorithms and code we can reuse and compose into new solutions. The programmer is providing the intellectual flexibility, while the code is the set of capabilities. It’s why this is a superior approach, compared to building a single monolithic mass of procedural code from scratch in a single variable scope for every program we write.
Solutions matter because it’s not just about what a system can do now, it’s about what it can learn or be adapted to do next.
Similar to how space is 4D such that with relativity going faster in a spatial dimension kind of "borrows" from the time dimension (in a hand wavy way).
By analogy, you can have something that's purely a lookup table, or on the other hand, completely based on an algorithm, and the full lookup table is kind of "borrowing" from the algorithmic dimension of the information system space and vice-verse the fully algorithmic version is borrowing from the hardcoded dimension of the information system space.
Under the condition that you're adding integers below N, then if you consider BOTH the (hardcoded, algorithmic) as a singular space (as with 4D space time) then they are equivalent.
Need to work on this theory further to make it more understandable, but I think this way about intelligence.
Intelligence sits as a pattern in the information system space that can range anywhere from hardcoded to algorithmic (if we choose to orthogonalize the space this way). But what actually matters is the system's future impact on it's local laws of physics, and for that purpose both implementations are equivalent.
Edit: Conversation with GPT-4 about this https://sharegpt.com/c/Sbs4XgI
Does that mean computers are not "really doing" addition?
If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.
But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until the recent internet age, in which garbage survives as well as the gold)).
> then maybe there's a bit of hubris in how smart we think we actually are
I feel like you could say this if ChatGPT or whatever obtained its knowledge some other way than direct guidance from humans, but since we hand-fed it the answers, it falls a little flat for me.
I'm open to persuasion.
> chatgpt doesnt just feed us back answers we already taught it
True, there is some structure to the answers we already taught it that it statistically mimics as well.
> It learned relationships and semantics so it can apply that knowledge to do something novel
Can you provide an example of this novelty? I think we underestimate the depth and variety of things that humans have written about and put on the internet, and so while anything you ask ChatGPT to do might be outside of your own experience, it's highly likely that it's already been thought before and uploaded to the internet, and that ChatGPT is just parrotting back something to you that is very similar to what it has already seen.
This effect of ChatGPT having so much more experience/training data than any single human being such that it can convince any single human that it is original is an interesting one.
This is why I think, for example, that image generation will result in (a period of) "artistic inbreeding." Because there is so much that other humans have done that is outside of any individual's experience, we will accept e.g. Midjourney's output as something moving and original, when in reality it's just a slight variation on something that someone else has done before that we haven't seen.
(Again apologies for any rudeness, I respect your opinion and experiences and am enjoying the conversation.)
Sure, I was saying "better" in the sense that if for X task, it can do better than Y% of humans.
> since we hand-fed it the answers, it falls a little flat for me
We didn't really hand-fed it any answers though did we? If you put a human in a white box all its life, with access to the entire dataset on a screen but no social interaction, nothing to see aside from the text, nothing to hear, nothing to feel, nothing to taste, etc, it'd be very impressed if they were then able to create answers that seem to display such thoughtful and complex understanding of the world.
For example if a human explains the process for solving a mathematical problem, we know that person knows how to solve that problem. That’s not necessarily true of an LLM. They can give such explanations because they have been trained on many texts explaining those procedures, therefore they can generate texts of that form. However texts containing an actual mathematical problem and the workings for solving it are a completely different class of text for an LLM. The probabilistic token weightings for the maths text explanation don’t help at all. So yes these are fascinating, knowledgeable and even in some ways very intelligent systems. However it a radically different form of intelligence from us, in ways we find difficult to reason about.
If you replace the analogy with humans and LLMs, LLMs won't ever reason or understand things in the same way we do, but if/when their output gets much smarter than us across the board, will it really matter?
Our written material assumes huge swathes of contextual knowledge, real world experience, and human lived experience that LLMs don’t and can’t have. At least architected and trained as they are now.
Thats on top of the crippling inability LLMs have to generalise an ability to perform a task from the ability to generate a description of how to do the task. Plus many other similar limitations that would be inexplicable if displayed by a human.
Of course LLMs aren’t the final word in AI development. I think they’re a vitally important step towards general AI, and we’ll get there eventually as we develop ever more capable architectures.
And, more importantly, they solve the problem much better if you tell them to reason about it in writing first before giving the final answer.
0: pasted from another thread
We continue to figure out new ways to make those calculations do more complicated things faster than humans.
What is intelligence beyond calculation is an ancient question, but not the one I'm most interested in at the moment, re: today's tools.
I'm curious right now about if there's meaning to other people in human creation vs automation creation. E.g. is there a meaningful difference between an algorithm curating a feed of human-made TikTok videos and an algorithm both curating and creating a feed of human-made TikTok videos.
Both qualitatively in terms of "would people engage with it to the same level" and quantitatively in terms of "how many new trends would emerge, how would they vary, how does that machine ecosystem of content generation behave compared to a human one" if you remove any human curation/training/feedback/nudging/etc from the flow beyond just "how many views/likes did you get?"
As for curation, I think the success of TikTok proves that you don’t need that much data to pretty preceding pinpoint what someone wants to watch (or what will get them to spend the most time on the app at least).
I think a super accessible animation tool would get a lot of use and result in a lot of cool stuff, but it's the latter that I'm really curious about in terms of how people interact with it.
But it’s clear based on current implementations that once you work backwards from the knowledge that it’s “just predicting the next token” you can easily find situations in which the AI doesn’t demonstrate general intelligence. This is most obvious when it comes to math, but it’s also apparent in hallucinations and the model not being able to reason through/synthesize ideas very well, deviate from the script (instead of just answering a question with what it has already, in some cases it should not even try to answer and instead ask more clarifying questions). To be fair, there are plenty of humans with excellent writing or speaking skills that are bad at that kind of stuff too.
If somehow you could generate a training text encoding a complete and thorough understanding of the physical world, human psychology and sociology, and reasoning then that might get you quite far. But the existing it even near future human textual corpus isn’t really that. Even then I still think you’d hit the limitations of the LLM cognitive architecture pretty hard.
Aside: I would like to see ChatGPTs with distinct training texts, e.g., a ChatGPTs trained on the "great books" of Western philosophy and science knowledge up to the time of Victorian England.
I imagine many definitions are initially rather broad and only get refined down over time. Laertius gives us a classic example:
> Plato defined man thus: “Man is a two-footed, featherless animal,” and was much praised for the definition; so Diogenes plucked a cock and brought it into his school, and said, “This is Plato’s man.” On which account this addition was made to the definition, “With broad flat nails.”
I don’t think it’s correct to think of that as infinitely moving goalposts, however. More that the weakness of definitions isn’t always immediately transparent.
I am not sure they can, but the difference is profound and material. A machine that actually understands, like a human being, is not going to be (can not be) entirely truthful or transparent. There will be private inner thoughts, idea formation, and possibly even willful intent, as a direct consequence of understanding. And the nature of interactions, regardless of superficial similarity, shifts from one of utility to relationship. For example, we would care to know if e.g. the systems entrusted with apocalyptic deterent forces are mechanisms or organisms.
Please note that not a single one of us has ever interacted with any intelligent life form lacking a sense of self, or an ego. Thus, all our sensory registers of another 'intelligent being' are learned in a context of the implicit 'this other is like me'. We are not equipped to distinguish or articulate intelligence (in the abstract) merely based on sensory information. Note that even non-verbal communication, such as jabbing a friend in the ribs, are all learned to have a certain meaning in that very same context of implicits, and any mechanism that mimicks them (via training) will be afforded the same projection of the implicit. I do not believe there is, in fact, any non-destructive test of determining 'consciousness' in an entity. (Destructive, since there may be long running tests of a subject than can be shown to be probably accurate, possibly via creating situational problems involving survival, and unexpected circumstances.)
Ask yourself what is it that convinces you that the last person you spoke with (in real life) was actually conscious? I assert that the entire matter is a 'fictional certainty' based on assumption of shared nature. "They are conscious because I am".
Here's a thought experiment. Suppose we make first contact tomorrow, and we meet some intelligent aliens. What are some questions you would ask them? How would you decide on their sentience or understanding?
Sentience involves goal-seeking, understanding, sensory inputs, first-personal mental states (things like pain, happiness, sadness, depression, love, etc.), a sense of what philosophers like Elizabeth Anscombe call I-ness, etc. Most of this stuff, to me, seems like is language-agnostic. Even a baby that can't speak feels pain or happiness. Even a dog feels anxiety or affection.
LLMs are a cute parlor trick, but a phantasm nonetheless.
This isn't true. If a plane flies like a bird and you only need it for flying it doesn't then follow that a plane is a bird "for all intents and purposes".
Requiring that something fly and that something be a plane are two different things with only minor overlap. If all you require is something that flies, then a dragonfly matches your requirements exactly as much as an apache helicopter does.
It spits out class names for slate objects, that inherit from other slate objects. Chatgpt doesn't understand inheritance. It just guesses what might fit inside a parameter grouping, and never suggests something with the right class type.
For my use case, it has never quacked like a duck, so to speak. It never performed, the word that might cover the concept of generating output without understanding it.
We agree on the value of computers understanding versus performing... only as much you need understanding to make it perform.
Predicting words alone does not cut the mustard, some structural depth or validating maps or some new concept is needed to sure up the wild horsepower in ChatGPT.
It must understand/have structure, or at least use a crutch to get it over the finish line..
My question is about the future. The argument goes that a machine can never understand Chinese, even if it is capable of interpreting Chinese and responding to or acting on the input perfectly every time. My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding.
It’s kind of like the whole string theory vs SUSY vs… argument in physics. If the only outcomes are things that agree with the Standard Model in all measurable aspects, and don’t provide any measurable distinction, then for all intents and purposes they don’t matter. That’s why their active areas of research are looking for the measurable distinctions.
FWIW, supersymmetry models predict measurable things (that so far have only ruled out those models when tested) but have applications elsewhere. String theory research has had implications in mathematics, condensed matter, and a bunch of other places. They’re useful.
But that’s beside the point, because the premise of the Chinese room problem is that there exists a machine that passes all scenarios, where no measurable difference can be found, and that this machine does not understand Chinese.
I'm not sure if you understood the argument. The argument isn't asserting that there is a measurable way of distinguishing it, it's actually claiming that regardless of how well it seems like it understands Chinese, it doesn't actually understand Chinese. It's about intentionality and consciousness.
In a chatbot, the man inside the room is the LLM, but the whole system is not just the LLM - it's the whole setup that picks generated tokens and feeds them back into the input as a loop. And it demonstrably understands what you tell it, because it can carry out instructions, even extremely convoluted ones or using substitute words that are not part of its training set.
In effect, my argument is that in order for you to require it to understand something, you require it to understand that thing for a reason. If it acts like it understands that thing under all probing, then your requirement is satisfied - the question about whether it truly understands the thing is moot, because it fulfils the requirement.
Defining a machine to be conscious, allows the individual to soak their mind in code and silicon as a receptacle for their spirit.
It creates a pull into a 'second mind'. Anybody who believes this is likely to invest heavily in the maintenance of new technology.
A 'conscious machine', creates an uneasy feeling that we should work to embed our spirit, knowledge, intellect into flipped bits, like expectant mothers. That we should work for the machine, and to the ends of the machine.
And that machine is somehow defined-to-be or a naturally, consciously alive (to a large or small degree). It is said to have a mind worthy of a person's professional output and it can hold the power of a marginally believable conversation.
While all of these described properties are vaguely plausible, it does nothing to help me understand the meaning of a technology, and only benefits those looking to create a fevor around a new tech product.
Describing chatgpt as a stochastic parrot or chinese room grants me a metaphor or analogy for the inner functions of the tech. It also lets me see, or otherwise guesstimate the products abilities clearly, without the belief-as-marketing hype.
I can take the stochastic parrot metaphor, to an article about LLMs and understand in a couple of days what took years of research to create.
Following the belief of computing as real human intelligence and that human intelligence is fundamentally mathematical, requires on some level submission of your mind to a machine that has it's own goals programmed in by someone else.
This centuries-long process of trying to encode and store all human knowledge behind the secure walls of complex coded signs.. and it's advocates for that process, create a subtle and deep twinge of future melancholy or dread or something. The idea that all written/typed meaning will be accessible only by the spiritual power brokers, and not our sons.
No. On some level, machines are just machines, like an abacus or a weaving loom. It can host concepts in the same way that a weaving loom is 'intelligent'. It holds it's shape, abstractions and functions by the laws of physics/metaphysics and according to my human dictates.
You follow the raven into the computers-are-conscious dream at your own risk. Computers are leaning towards controlling people rather than emancipating them. Leaning very hard in that direction. Do we want that? Freedom of mind and meaning is valueable.
I don't think an intelligence needs to be human, and it should be physically possible to create an intelligence which is synthetic. But in order to call the intelligence "general", and to rely on it for the purposes that designation implies, it would need to be able to successfully navigate the world, which requires access to that world and the use of the world as its own model, rather than the much simpler and coarser intermediary of text. In order to claim that an LLM can fully navigate the world after being trained on pure text, we would have to believe that all our writings across history have exhausted what there is to say about the world. This is not to say an LLM cannot be useful for some purposes, but there will be key ways in which they fail because they have no sense of meaning or what the world is like. Whether consciousness is required to solve this I don't know, but we simply haven't begun to approach a system that can meaningfully address the world as a world.
The pattern is counting the number of closed spaces in each letter of the spelled-out number. A closed space is any enclosed space in a letter, such as in the letters "a", "b", "d", "e", etc.
Following the pattern:
- one -> 2 (there are closed spaces in the letters "n" and "e") - two -> 1 (there is a closed space in the letter "o") - three -> 2 (there are closed spaces in the letters "h" and "e") - four -> 1 (there is a closed space in the letter "o") - five -> 1 (there is a closed space in the letter "e") - six -> 0 (there are no closed spaces in the letters) - seven -> 2 (there are closed spaces in the letters "e" and "n") - eight -> 1 (there is a closed space in the letter "g") - nine -> 1 (there is a closed space in the letter "e") - ten -> 1 (there is a closed space in the letter "b") - eleven -> 3 (there are closed spaces in the letters "e", "l", and "v") - twelve -> 2 (there are closed spaces in the letters "b" and "d") - thirteen -> 2 (there are closed spaces in the letters "b" and "d")
Each item follows the pattern, as the number of closed spaces in their letters matches the corresponding number in the pattern.
The whole sequence is:
one -> 2 two -> 1 three -> 2 four -> 1 five -> 1 six -> 0 seven -> 2 eight -> 1 nine -> 1 ten -> 1 eleven -> 3 twelve -> 2 thirteen -> 2 ...»
It is clear the model doesn't know what it is talking about.
Here's a different example involving dataset analysis with GPT-4 that required it to analyze its own previous outputs to find and correct mistakes and form a new hypothesis:
https://gist.github.com/int19h/cd1d1598f91e8ba92dd8e80bd5d21...
Norvig and Chomsky really got into this type of argument, though maybe it’s a stretch to say it’s this exact one; see Norvig’s side here: https://norvig.com/chomsky.html
For all the terrible things people worry about ChatGPT doing, this was not one that I thought I was going to have to deal with.
(edit: ChatGPT was not involved at all, but when I suggested she give it a try to see for herself, that was the end of it.)
This sounds like you said "I cannot possibly be friends with someone who does not believe that LLMs are emerging AGI!", and people read it like that and are downvoting you.
I'm gonna assume the situation was more complex, but still find it hard to imagine, how a disagreement over such an academic topic could end up destroying a friendship.
I only shared the story to illustrate how personally people are taking these discussions. I really felt like I was being very neutral and just sharing my enthusiasm. It was entirely unwelcome, apparently.
If there's a lesson to be learned it's that people's tempers over these issues may be hotter than they appear.
I can barely speak with my artist friends on the issue these days due to their generative AI fears. Their emotions are completely intractable on the subject: AI art is theft. Period.
Then the only true artist would be one who has never seen art before except his/her own. The correction of your friends' belief would be that "ALL art is theft. Period."
(edit: This is the kind of stuff I think my friends are watching and being informed by [0] as it was what they are posting in our common areas.)
That's me. After programming since the '80s, I'm just so tired. So much work, so much progress, so many dreams lived or shattered. Only to end up here at this strange local maximum, with so much potential, destined to forever run in place by the powers that be. The fundamentals formula for intelligence and even consciousness materializing before us as the world burns. No help coming from above, so support coming from below, surrounded by everyone who doesn't get it, who will never get it. Not utopia, not dystopia, just anhedonia as the running in place grows faster, more frantic. UBI forever on the horizon, countless elites working tirelessly to raise the retirement age, a status quo that never ceases to divide us. AI just another tool in their arsenal to other and subjugate and profit from. I wonder if a day will ever come when tech helps the people in between in a tangible way to put money in their pocket, food in their belly, time in their day - independent of their volition - for dignity and love and because it's the right thing to do. Or is it already too late? I don't even know anymore. I don't know anything anymore.
Seriously tho, taking some time to get away from it would be good. Ignorance is bliss, this too shall pass etc.
(btw nice piece of writing, you should do it more often!)
In the long run tech does a bit too well with "food in their belly" to the point that obesity is the main problem in the English speaking world.
As to programming it's quite cool getting chat GTP to write code and stuff. If you can't beat it make use of it I guess.
I'd hope some of us would just be there in 60 years to just tell the future: "Heee just embrace it, ya know" .. nuff said.
I struggle to understand why this thing works the way it does. It's possible that Vaswani et al. have made one of the greatest discoveries of this century that solved the language problem in an unintuitive, and yet very unappreciated way. It's also possible that there are other architectures that can simulate the same level of intelligence with such large numbers of parameters.
I think you re right that it's not intuitive, it's like basic arithmetic is laughing at us
I'm not in this field but have recently found myself going on the deepest dive possible into it as my small brain can absorb.
I now know about (on a surface level) neural networks, transformers, attention mechanisms, vectors, maticies, tokenization, loss functions and all sorts of other crazy stuff.
I come out of this realizing that there are some incredibly brilliant minds behind this. I knew AI was a complex subject but not on the level I've learned about now. To get what is essentially matrix multiplications to learn complex patterns and relationships in language is mind-blowing.
And it's creative. It can have a rap battle with an alter-ego, host a quiz party with other AIs of varying personalities, co-author a short story with me, respond to me only in emojis. The list is seemingly endless. Oh, and it can also do useful things. It's my programming companion too.
And we're just getting started.
The other clear benefit of transformers over an arch like RNNs (and what has probably made more of a difference imo) is that its properly parallelizable, which means you can do huge training runs in a fraction of the time. RNNs might be able to get to a level of coherence that approaches GPT-3, but with current hardware that would be very time-prohibitive.
In fact the projection operations are the only learned part of a Transformer's self-attention function -- the rest of self-attention is just a weighted sum of the input vectors, where the weights come from the (scaled) vector correlation matrix.
And if I tell it something that was excatly in it's trained context windows, I get the most likely next word and the one after itm
But what happens if I ask it something slighty different than it's training context ? Or something largely different?
It's not possible for you to ask it things even slightly different from it training data, unless you ask exclusively in emojis that didn't exist yet when it was trained (in which case it sees nothing, just like when someone sends you an emoji your phone doesn't support).
Any novel sentence and even novel words like "Blobdarfnk" ARE in its training data. "Blobdarfnk" is encoded as the five tokens Bl, ob, dar, fn, and k.
Given that Lacan already proposed the unconscious as structured language-like more than half a century ago and described attention in his turn on Freud's impulse in favor of his concept of derive, we may say, this is pretty much where our own demons live.
(I actually do think that revisiting Lacan in this context may be productive.)
Please end our strange fascination with fashionable nonsense. Freud was wrong. There is no Oedipus complex. Everything lacan proposed was wrong. Deleuze and Guattari's mental health clinic failed spectacularly, and Deleuze ended up killing himself at the end (supposedly due to back pain?)
They literally describe their thought as being "Schizoanalysis". How many more red flags do you need?
Also, the more "modern" takes on this from techno folks, such as from Nick Land (Fanged Noumena), are openly fascist - https://en.wikipedia.org/wiki/Dark_Enlightenment
If you want cultural critique from smart people without it turning into fashionable nonsense, I recommend Mark Fischer, but be warned, he too killed himself.
Regarding charlatans, mind that there are already few who have actually studied this. (I'm one of them.)
Regarding Lacan, he provides us with an established theory of "talking machines", and, in a philosophical context, how they relate to our very freedom (or, what freedom may even be). This isn't totally useless in our current situation, and NB, it's actually quite the opposite of fascism.
That's just to correct the record. I have no desire to re-litigate Sokal/Bogdanoff and so on. Good day sir cheerio.
(There had been times, when linguistics were still a major entry path into computing, where things were a bit different. Notably, this were also the times, which gave rise to most of the general paradigms. A certain amount of generality was even regarded a prerequisite to programming. Particularly, HN is such a great place, because it holds up this notion of generality.)
(A turn towards the dogmatic is something I'm pretty much expecting from the current launch of AI anyway, simply, because the productions systematically favor the semantic center. So it may be worth putting some generality against this, rather than being overly selective.)
It's a bit more plausible when we phrase it that way...
I agree with the sentiment that each individual dimension isn't meaningful, and I also feel like it's misleading for the article to frame it that way. But there's a grain of truth: the last step to predicting the output token is to take the dot product between some embedding and all the possible tokens' embeddings (we can interpret the last layer as just a table of token embeddings). Taking dot products in this space are equivalent to comparing the "distance" between the model's proposal and each possible output token. In that space, words like "apple" and "banana" are closer together than they are to "rotisserie chicken," so there is some coarse structure there.
Doing this, we gave the space meaning by the fact that cosine similarity is meaningful proxy for semantic similarity. Individual dimensions aren't meaningful, but distance in this space is.
A stronger article would attempt to replicate the word2vec analogy experiments (imo one of the more fascinating parts of that paper) with GPT's embeddings. I'd love to see if that property holds.
Isn't this just responding to the context provided?
Like if I say "Write a Limerick about cats eating rats" isn't it just generating words that will come after that context, and correctly guessing that they'll rhyme in a certain way?
It's really cool that it can generate coherent responses, but it feels icky when people start interrogating it about things it got wrong. Aren't you just providing more context tokens for it?
Certainly that model seems to fit both the things it gets right, and the things it gets wrong. It's effectively "hallucinating" everything but sometimes that hallucination corresponds with what we consider appropriate and sometimes it doesn't.
It's a bit like the Sagan quote: "If you wish to make an apple pie from scratch, you must first invent the universe".
Sometimes for GPT to "just" complete the next word in a way that humans find plausible, it must, along the way, develop a model of the world, theory of mind, abstract reasoning, etc. Because the models are opaque, we can't yet point to a certain batch of CPU cycles and say "there! it just engaged in abstract reasoning". But we can see from the output that to some extent it's happening, somehow.
We also see effects like this when looking at collective intelligence of bees and ants. While each individual insect is only performing simple actions with extremely limited cognitive processing, it can add up to highly complex and intelligent/adaptive mechanics at the level of the swarm. There are many phenomena like this in nature.
I did an experiment recently where I asked ChatGPT to "tell me an idea [you] have never heard before". ChatGPT replied with what sounded like an idea for a startup, which was delivering farm-fresh vegetables to customers' doors. This is of course not an idea it has never heard before, it's on the internet.
If you asked a human this, they would give you an idea they had never heard before, whereas ChatGPT simply "finds" training data where someone asked a similar question, and produces the likely response, which is an idea that it has actually "heard," or seen in its training data, before. (Obviously a gross simplification of the algorithm but the point stands.)
This is a difference between ChatGPT's algorithm and human reasoning. The things that you mention, the model of the world, theory of mind, etc. are statistical illusions which have observable differences from the real thing.
Am I wrong? I'm open to persuasion.
Certainly, Midjourney's "creativity" is different from human creativity. But it is producing results that we marvel at. It's creative not because it's doing the exact same philosophical thing humans do, but because it can produce the same effect.
And I think many situations are like that. We can always say that human creativity/reasoning/x will always be different from artificial reasoning. But even today, GPT's statistical model replicates many aspects of human reasoning virtually. Is that really an illusion (implying its fake and potentially useless), or is it just a different way of achieving a similar result?
Plus, different models will excel at different thing. GPT's model will excel at synthesizing answers from far more information than a single human will ever be able to know. Does it really matter if it's not identical to human reasoning on a philosophical or biological level, if it can do things humans can't do?
At the end of the day, some of these discussions feel like bike shedding about what words like "reasoning" mean philosophically. But what will ultimately matter is how well these models perform at real world tasks, and what impact that will have on humanity. It doesn't really matter if it's virtualized reasoning or "real" human reasoning at that point.
Absolutely, and I hope none of my comments are taken in a way that disparages how amazing ChatGPT and Stable Diffusion et al. are. I'm just debating how humanlike they are.
> Is that really an illusion (implying its fake and potentially useless)
I don't think that because it's an illusion means that its useless. Magnets look like telekinesis, but that effect being an illusion doesn't mean that magnets are useless; far from it, and once we admit that they are what they are, they become even more useful.
> Plus, different models will excel at different thing. GPT's model will excel at synthesizing answers from far more information than a single human will ever be able to know. Does it really matter if it's not identical to human reasoning on a philosophical or biological level, if it can do things humans can't do?
It only matters if people are trying to say that ChatGPT is essentially human, that idea is all I was replying to. I completely agree with you here.
A simple tool can’t. A « mind » that is coming in our world should, right ?
Just like all the marvel and DC where super-human are still accountable for their mistakes, their super powers are no excuse.
And remixing stuff it has heard before is exactly what ChatGPT is doing. What it sucks at is the "feels like it's new" part, but fundamentally it would be quite easily capable of creating output that combines concepts with maximally negative correlation, the only thing that's truly missing is the ability to interpret the prompt as an instruction to do that.
How does one actually measure novelty, without having to know everything first?
Check out my "the confetti has left the cannon" example above.
https://news.ycombinator.com/item?id=35346683
Maybe still "remixing", but it sure feels like new to me.
> The BerryPulse is an innovative, eco-friendly device that captures the energy released during the natural decomposition of berries to produce heat and electricity. The device comprises a closed container, where a cluster of berries is placed in a specially designed compartment, acting as an organic fuel source. The decomposition process is intensified using a biodegradable catalyst, which accelerates the natural breakdown of the berries while ensuring minimal environmental impact.
Is that a falsifiable claim?
What would an empirical test look like that would determine whether the claim ("GPT-4 does not have a model of the world") is correct or not?
> ChatGPT simply "finds" training data where someone asked a similar question, and produces the likely response, which is an idea that it has actually "heard," or seen in its training data, before.
I can definitely see a scenario where we manage to build an ultra-intelligent machine that can figure out any logical puzzle we put to it, but where it still isn't "creative" in any way.
But GPT-4 seems to be not just intelligent, but also creative.
Have you seen the "confetti has left the cannon" example?
For the curious, here was the conversation I had:
ME: What's another saying similar to: "The cat is out of the bag" or "The genie is out of the bottle"?
chatgpt: Another similar saying is "the toothpaste is out of the tube." Like the other two expressions, this phrase conveys the idea that something has been revealed or unleashed, and it cannot be undone or put back into its original state.
ME: Can you invent a new phrase that has the same meaning, which is completely new?
chatgpt: Certainly! Here's a new phrase with a similar meaning: "The confetti has left the cannon." This phrase suggests that once something is out in the open or has happened, it's impossible to reverse or contain it, much like confetti that has been shot out of a cannon during a celebration.
In ChatGPT the language-model and world-model are really just the same model, which makes a lot of sense.
It can also write code for novel use cases that have never been done before. I gave it a task like this a few days ago and it got it right on the first try. There are literally millions of empirical data points that contradict you.
Aren't you just doing that?
With respect to interrogating GPT if it does something wrong - the reason why people do it is because it works. With GPT-4 especially, you can often ask it to analyze its own response for correctness, and it will find the errors without you explicitly pointing them out. You can even ask it to write a new prompt for itself that would minimize the probability of such errors in the future.
Who got caught for feeding some Rats ; Tremendous Work!
All the people tell me, many men, biggly men - many with tears in their eyes...
That I have done nothing legally-wise
But the truth is ; I am an enormous dork.
>>_Created by an actual Human Being with actual DNA for crime scene evidence._
-
But just when they tried to brush under a rug
To try to make the folks 'shrug'
Is the Streisand Effect as a scar
As everyone knows of payments to a Porn Star
And the nation will know youre a simple thug.
Guilty of paying too much for pork
He thought he would never stand
on a trial from the local grand
but corruption was just part of the work.
I guess ... this is what confuses me. GPT -- at least, the core functionality of GPT-based products as presented to the end user -- can't just be a language model, can it? There must be vanishingly view examples from its training text that start as "Write a Limerick", followed immediately by some limerick -- most such poems do not appear in that context at all! If it were just "generating some text that's likely to come after that in the training set", you'd probably see some continuations that look more like advice for writing Limericks.
And the training text definitely doesn't have stuff like, "As a language model, I can't provide opinions on religion" that coincides precisely with the things OpenAI doesn't want its current product version to output.
Now, you might say, "okay okay sure, they reach in and tweak it to have special logic for cases like that, but it's mostly Just A Language Model". But I don't quite buy that either -- there must be something outside the language model that is doing significant work in e.g. connecting commands with "text that is following those commands", and that seems like non-trivial work in itself, not reasonably classified as a language model.[2]
If my point isn't clear, here is the analogous point in a different context: often someone will build an AND gate out of pneumatic tubes and say, "look, I made a pneumatic computer, isn't that so trippy? This is what a computer is doing, just with electronics instead! Golly gee, it's so impressive what compressed air is [what LLMs are] capable of!"
Well, no. That thing might count as an ALU[1] (a very limited one), but if you want to get the core, impressive functionality of the things-we-call-computers, you have to include a bunch of other, nontrivial, orthogonal functionality, like a) the ability read and execute a lot of such instructions, and b) to read/write from some persistent state (memory), and c) have that state reliably interact with external systems. Logic gates (d) are just one piece of that!
It seems GPT-based software is likewise solving other major problems, with LLMs just one piece, just like logic gates are just one piece of what a computer is doing.
Now, if we lived in a world where a), b), and c) were well-solved problems to point of triviality, but d) were a frustratingly difficult problem that people tried and failed at for years, then I would feel comfortable saying, "wow, look at the power of logic gates!" because their solution was the one thing holding up functional computers. But I don't think we're in that world with respect to LLMs and "the other core functionality they're implementing".
[1] https://en.wikipedia.org/wiki/Arithmetic_logic_unit?useskin=...
[2] For example, the chaining together of calls to external services for specific types of information.
I would change the introduction to be more impartial and not anthropomorphize GPT. It is not smart and it is not skilled in any tasks other than that for which it is designed.
I have the same reservations about the conclusion. The whole middle of the article is good. But to then compare the richness of our human experience to an algorithm that was plainly explained? And then to speculate on whether an algorithm can "think" and if it will "destroy society," weakens the whole article.
I really would like to see more technical writing of this sort geared towards a general audience without the speculation and science-fiction pontificating.
Good effort!
But it wasn't designed. It's not a computer program, where one can make confident predictions about its limitations based on the source code.
It's a very large black box. It was trained on guessing the next word. Does that fact alone prove that it cannot have evolved certain internal structures during the training?
Do you claim that an artificial neural network with trillions of neurons can never be intelligent, no matter the structure?
Or is the claim that this particular neural network with trillions of neurons is not intelligent? If so, what is the reasoning?
> It is not smart
"Not smart" = "not able to reason intelligently".
Is that a falsifiable claim?
What would the empirical test look like that would show us if the claim is correct or not?
Look, I realize that "GPT-4 is intelligent" is an extraordinary claim that requires extraordinary evidence.
But I think we're starting to see such extraordinary evidence, illustrated by the examples below.
https://openai.com/research/gpt-4 (For instance, the "Visual inputs" section)
Microsoft AI research: Many convincing examples, summarized with:
"The central claim of our work is that GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence.
This is demonstrated by its core mental capabilities (such as reasoning, creativity, and deduction), its range of topics on which it has gained expertise (such as literature, medicine, and coding), and the variety of tasks it is able to perform (e.g., playing games, using tools, explaining itself, ...)."
Yes. There is interesting work to formalize these black boxes to be able to connect what was generated back to its inputs. There’s no need to ascribe any belief that they can evolve, modify themselves, or spontaneously develop intelligence.
As far as I’m aware no man made machine has ever exhibited the ability to evolve.
> Do you claim that an artificial neural network with trillions of neurons can never be intelligent, no matter the structure?
If, by structure, you mean some algorithm and memory layout in a modern computer I think this sounds like a reasonable claim.
NN, RNN, etc are super, super cool. But they’re not magic. And what I’m arguing in this thread is that people who don’t understand the maths and research are making wild claims about AGI that are not justified.
> Look, I realize that "GPT-4 is intelligent" is an extraordinary claim that requires extraordinary evidence.
That’s the crux of it.
But neural networks clearly evolve and are modified during training. Otherwise they would never get any better than a random collection of weights and biases, right?
Is the claim then that an artificial neural network can never be trained in such a way that it will exhibit intelligent behavior?
>> Do you claim that an artificial neural network with trillions of neurons can never be intelligent, no matter the structure?
> If, by structure, you mean some algorithm and memory layout in a modern computer I think this sounds like a reasonable claim.
Yes, that's what I mean.
Is your claim that no Turing machine can be intelligent?
>> Look, I realize that "GPT-4 is intelligent" is an extraordinary claim that requires extraordinary evidence.
> That’s the crux of it.
And I provided links to such evidence. Is there a rebuttal?
If we're saying that GPT-4 is not intelligent, there must be questions that intelligent humans can answer that GPT-4 can't, right?
What is the type of logical problem one can give GPT-4 that it cannot solve, but most humans will?
I think it’s not likely a NN can be trained to exhibit any kind of autonomous intelligence.
Science has good models and theories of what intelligence is, what constitutes consciousness, and these models are continuing to evolve based on what we find in nature.
I don’t doubt that we can train NN, RNN, and deep learning NN to specific tasks that plausibly emulate or exceed human abilities.
That we have these deep learning systems that can learn supervised and unsupervised is super cool. And again, fully explainable maths that anyone with enough education and patience can understand.
I’m interested in seeing some of these algorithms formalized and maybe even adding automated theorem proving capabilities to them in the future.
But in none of these cases do I believe these systems are intelligent, conscious, or capable of autonomous thought like any organism or system we know of. They’re just programs we can execute on a computer that perform a particular task we designed them to perform.
Yes, it can generate some impressive pictures and text. It can be useful for all kinds of applications. But it’s not a living, breathing, thinking, autonomous organism. It’s a program that generates a bunch of numbers and strings.
But when popular media starts calling ChatGPT “intelligent,” we’re performing a mental leap here that also absolves the people employing LLM’s from responsibility for how they’re used.
ChatGPT isn’t going to I take your job. Capitalists who don’t want to pay people to do work are going to lay off workers and not replace them because the few workers that remain can do more of the work with ChatGPT.
Society isn’t threatened by ChatGPT becoming self aware and deciding it hates humans. It cannot even decide such things. It is threatened by scammers who have a tool that can generate lots of plausible sounding social media accounts to make a fake application for a credit card or to socially engineer a call centre rep into divulging secrets.
> "autonomous intelligence"
> "what constitutes consciousness"
> "autonomous thought"
In my mind, this is a list of different concepts.
GPT-4 is definitely not living, breathing or autonomous. It doesn't take any actions on its own. It just responds to text.
Can we stay on just the topic of intelligence?
Let's take this narrow definition: "the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas".
> But in none of these cases do I believe these systems are intelligent
It should be possible to measure whether an entity is intelligent just by asking it questions, right?
Let's say we have an unknown entity at the other end of a web interface. We want to decide where it falls on a scale between stochastic parrot and an intelligent being.
What questions about logical reasoning and problem solving can we ask it to decide that?
And where has GPT-4 failed in that regard?
It definitely is exactly that. It's not any more special than any other program that you can write. I am not totally sure that what you describe could ever exist at all.
What makes this program "magic" compared to any other program exactly? There is no physical difference between it and a "regular" program. Both of them are a bunch of source code that gets compiled into an executable and ran by the underlying OS and hardware. There is nothing physically different between it and other software.
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
https://mrl.snu.ac.kr/research/ProjectAgile/Agile.html
This is done using neural networks. I believe a project like that can be done by a few researchers over months, not years?
If you do this using "regular programming" instead, you'd have to write an insanely complex application that uses inverse kinematics etc.
https://en.wikipedia.org/wiki/Inverse_kinematics
A project like that requires a large team of developers, working over many years. Boston Dynamics is one example.
All programs that run on the computer have the same "power" in terms of what they can do and what can be computed using them. A program that implements a neural net is not inherently any different than a silly python script. One just does a lot more stuff and is much more interesting.
And then we can drill down even further where everything is just physics with atoms, quantum mechanics, etc.
So you're not different from a computer. Both are just physics.
But that's not a useful world view in my opinion.
I think "regular" and "non-regular" programming is a useful distinction.
In regular programming, I have to write explicit implementations of the algorithms in the program.
In "non-regular programming" (neural networks), I just have to know how to set up and train neural networks.
Once I do that, the neural networks can be trained to evolve algorithms that I myself don't know how to implement.
Don't you see the big difference between "I have to code the algorithms" and "the computer does it for me"?
The program which takes your text and runs a final calculation on it against the machine learning model to get an output is a program. But that program is not doing anything interesting. All the interesting work was done when the model was cooked up in a black-box non-deterministic process by some other GPUs somewhere else well before it ever came near the inference program.
Just a nit-pick: Aren't neural networks and LLMs perfectly deterministic?
I think you can reproduce GPT-4 perfectly if you have access to the same source code, training data, and the seeds for the random number generators that they used?
As a side note, I think it'd be theoretically possible to do this on a small 8-bit microcontroller given enough time and external storage. That's the beauty of Turing machines.
This would not be practical in the least. But it sure was cool seeing a guy boot Linux in just 3.5 hours on a small 8-bit AVR microcontroller.
https://dmitry.gr/?r=05.Projects&proj=07.%20Linux%20on%208bi...
The errors are small rounding errors that maybe don't have any serious implications right now. But the larger models get and the more operations and cores it takes to train them the more the rounding errors creep up.
To expand a bit:
I can write simple image processing code that will find lines in an image.
But I can't write the code to perform OCR (optical character recognition).
However, in the early 90's, I wrote a simple C program that trained a neural network to perform OCR. It was a toy project that took a weekend.
There are many things where I could train a neural network to do something, but couldn't write explicit source code to perform the same task.
If you (chlorion) look up "genetic algorithms", you'll find many clear examples of where very impressive algorithms were evolved using a simple training program.
I meant that the process of generating the models, and otherwise interacting with them are regular programs. The model itself is I guess more like a database or something, but it too is just regular data.
The original thing I was replying to was claiming that the process in general was "not a program", as if there was some magic thing going on that made the model different from output of other programs, or the training was somehow magical. (that is how I read it at least)
CPUs and GPUs physically cannot do anything other than execute programs which are encoded into bytecode.
What you are describing is that the language model is "magic" and breaks the laws of physics. I don't believe in magic personally though.
Regarding the speculation/destroy society, I was directly answering questions that I got from laypeople around me. The consequences on society I don't think are much speculation: it's going to have a big effect on many jobs, just like AI has started to have but much more. For the philosophical questions, I tried to present both sides of the issue to show that it's not just a clear "yes or no": some people will happily argue with you about GPT being smart/skilled/comparable to a human brain. Anyway, it's just an introduction to the questions that you might have about it.
I will, thank you! :)
> Regarding the speculation/destroy society, I was directly answering questions that I got from laypeople around me.
I get that. I think it's important in these times that we educate laypersons rather than froth up fears about "AI". It doesn't help, I suppose, that we get questions like this because some lazy billionaire decided to run their mouth off about this or that. Which society then treats like it is news and established fact.
I don't think the speculation about consciousness is as well informed as the rest of the article. There is plenty of science and research about it available and its definition extends well beyond humans! Our understanding of what consciousness is is a thoroughly researched topic in psychology, physiology, biology, etc! It's a fascinating area of study.
Best of luck and keep up the good work!
This seems to me to be obviously incorrect, and should be apparent after a few minutes of playing with GPT4. What makes it so powerful is how general-purpose it is, and it can be used for literally an unlimited set of tasks that involve human language. To say that it's not "smart" begs the question of what exactly constitutes smart and when you'll know that an AI has achieved it.
It really depends on who the target audience is. There's been a lot of scare mongering in the news about it lately and I think the last part tries to address that. It first offers an explanation that my parents can understand and then addresses what they have been hearing about in the news.
So, I would say it is great to share it with them and I think they are the intended audience.
Programming languages are a whole lot more structured and predictable than human language.
In JavaScript the only token that ever comes after "if " is "(" for example.
I once asked it for a short example code of something, no longer than 15 lines and it said "here's a code that's 12 lines long" and then added the code. Did it have the specific code "in mind" already? Or was it just a reasonably-sounding length and it then just came up with code that matched that self-imposed constraint?
Separately (and apologies for going on a tangent), where do you think we are in the Gartner cycle?
Around GPT3 time I was expecting for trough of disillusionment to come, particularly when we see the results of it being implemented everywhere but it hasn't really come yet. I'm seeing too many examples of good usage (young folks using it for learning, ESL speakers asking for help and revisions, high-level programmers using it to save themselves additional keystrokes, the list is long).
I actually think it helps to reframe this. It hallucinates up a method call that predictively should exist.
If you're working with boto3, maybe that's not actually practical. But if it's a method within your codebase, it's actually a helpful suggestion! And if you prompt it with the declaration and signature of the new method, very often it will write the new helper method for you!
I wonder if it is better at some languages than others. I have been using it for Go for a week or two and it’s ok but not awesome. I am also learning how to work with it, so probably will keep at it, but it is clearly a generative model not a thinking being I am working with.
I'm pretty sure " " (whitespace) is a token as well, which could come after a `if` as well. I think overall your point is a pretty good one though.
> Programming languages are a whole lot more structured and predictable than human language.
> In JavaScript the only token that ever comes after "if " is "(" for example.
But isn't that like saying that it's easy to generate English text, all you need is a dictionary table where you randomly pick words?
(BTW, keep up the blog posts, I really enjoy them!)
- What is thinking, exactly?
- Does human (or superhuman) thinking require consciousness?
- What even is consciousness? Why is it that when you take a bunch of molecular physical laws and scale them up into a human brain, a signal pattern emerges that feels things like emotions, continuity between moments, desires, contemplation of itself and the surrounding universe, and so on?
- Why and how does a string predictor on steroids turn out to do things that seem so close to a practical definition of thinking? What are the best evidence-based arguments supporting and opposing the statement "GPT4 thinks"? How do people without OpenAI's level of model access try to answer this question?
(And yes, it's occurred to me that I could try asking GPT4 to help me make these questions more complete)
Welcome to the club. There pretty much are no answers, just theories primarily played out as thought experiments. Its on of those areas where you can pick out who knows less (or is being disingenuous) by seeing who most confidently speaks about having answers.
We don't know what consciousness is, and we don't know what it means to "think". There, I saved you a decade of reading.
Edit: My choice theory is panpsychism, https://plato.stanford.edu/entries/panpsychism/ but again, we don't yet know how to verify any of this (or any other theory).
Interestingly, more than one of these folks have turned out to be religious. I wonder if increasingly intelligent AI systems will be challenging for religious folks to accept, because it calls into question our place at the pinnacle of God's creation, or it casts doubt upon the existence of a soul, etc.
I think this is a very simplistic view, that possibly suggests you haven't talked to many religious people.
I've never known a religious person who thought "thought" was the same as "soul", or that God is neccesarily a requirement for reasoning. Or that any of this is thought about much, considering it's so new.
Although, I suppose that if someone did say that God was a requirement for reasoning, a "logical within that context" perspective might be AI being some vicarious creation, since it wouldn't have been possible without us being able to reason.
I subscribe to the belief that reasoning is an eventual emergent law of nature/information. But, even that could, and does, fit into many "religious" perspectives perfectly well.
The guy fired by google for announcing LaMDA was sentient was religious.
I don't really see a meaningful distinction between declaring a machine is "thinking" for hand waving religious reasons and hand waving non-religious reasons, I'm afraid.
One possible conclusion might be that the only thing keeping GPT algos from going full AGI is a loop and small context windows.
I was going to write this exactly. I believe these things think. They're just not alive.
- What even is consciousness?
My advice: stay as far as you can from that concept. Wittgenstein already noticed that many philosophical questions are nonsense and specifically mentioned how consciousness as felt from the inside is hopefully incompatible with any observation we make from the outside.
BS concepts like qualia are all the rage now, but ultimately useless.
The best definition of "intelligence" is "the degree of ability to correctly predict future outcomes based on past experience".
Our cortex (part of the brain used for cognition/thinking) appears to be literally a prediction engine where predicted outcomes (what's going to happen next) are compared to sensory reality and updated on that basis (i.e. we learn by surprise - when we are wrong). This makes sense as an evolutionary pressure since ability to predict location of food sources, behavior of predators, etc, etc, is obviously a huge advantage over being directly reactive to sensory input in the way that simpler animals (e.g. insects) are.
I'd define consciousness as the subjective experience of having a cognitive architecture that has particular feedback paths/connections. The fact that there is an architectural basis to consciousness would seem to be proved by impairments such as "blindsight" where one is able to see, but not conscious of that ability! (eg. ability to navigate a cluttered corridoor, while subjectively blind).
It doesn't seem that consciousness is a requirement for intelligence ("ability to think"), although that predictive capability can presumably benefit from more information, so these feedback paths may well have evolutionary benefit.
The reason a "string predictor on steroids" turns out to be able to do things that seem like thinking is because prediction is the essence of thinking/intelligence! Of course there's a lot internally missing from GPT-4 compared to our brain, for example basics like working memory (any internal state that persists from one output word to the next) and looping/iteration, but feeding it's own output back in does provide somewhat of a substitute for working memory, and external scripting/looping (AutoGPT, etc) goes a long way too.
organic thinking (I.e. the process our squishy human brains do)
and mechanical thinking ( the computational and stochastic processes that computers do ).
It is entirely possible to build mechanical thinking in organic material (think Turing machines built on growing tissue), and it could also be possible to build complex self-referential processes simulated on electronic hardware, of the kind high-level brains do, with their rhythms of alfa and beta waves.
I doubt we'll ever be able to answer this, even after we create AGI.
There are two ways of looking at this.
1) In order to predict next word probabilities correctly, you need to learn something about the input, and the better you want to get, the more you need to learn. For example, if you just learned part-of-speech categories for words (noun vs verb vs adverb, etc), and what usually follows what, then you would be doing better than chance.. If you want to do better than that they you need to learn the grammar of the underlying language(s).. If you want to do better than that then you start to need to learn the meaning of what is being discussed, etc, etc.
If you want to correctly predict what comes next after "with a board position of ..., Magnus Carlson might play", then you better have learned a whole lot about the meaning of the input!
The "predict next word" training objective and feedback provided doesn't itself limit what can be learned - that's up to the power of the model that is being trained, and evidentially large multi-layer transformers are exceptionally capable. Calling these huge transformers "LLMs" (large language models) is deceptive since beyond a certain scale they are certainly learning a whole lot more than language/grammar.
2) In the words of one of the OpenAI developers (Sutskever), what these models have really learnt is some type of "world model" modelling the underlying generative processes that produced the training data. So, they are not just using surface level statistics to "predict next word", but rather are using the (often very lengthy/detailed) input prompt to "get into the head" of what generated that, and are predicting on that basis.
It would convince a lot of people with the breadth, despite not really having much depth.
The real GPT model is much deeper than that, of course, but my toy example should at least give a vibe for why even a simple thing might still feel extraordinary.
Such a system would already struggle with multiple-word inputs and it would be completely impossible to make it scale to even a paragraph of text, even if you had ALL of the observable universe at your disposal for encoding the entries.
Consider: If you just have simple sentences consisting of 3 words (subject, object, verb, with 1000 options each-- very conservative assumptions), then 9 sentences already give more options than you have atoms (!!) in the observable universe (~10^80)
β: if statements can grab patterns just fine in most languages, they're not limited to pure equality
γ: it's a thought experiment about how easy it can be to create illusions without real depth, and specifically not about making an AGI that stands up to scrutiny
Feel free to come up with a better entropy model then. Stackoverflow gives me confidence that it will be between 5 and 11 bits per word anyway [https://linguistics.stackexchange.com/questions/8480/what-is...].
> if statements can grab patterns just fine in most languages, they're not limited to pure equality
This does not help you one bit. If you want to produce 9 sentences of output per query then regular expressions, pattern matching or even general intelligence inside your if statements will NOT be able to save the concept.
More colourless green dreams sleep furiously in garden path sentences than I have
> This does not help you one bit.
Dunno, how many bits does ELIZA? I assume more than 1…
When you initiate the model with some input where you expect some particular correct output, that means there exists some completed sequence of tokens that is correct—if that weren’t true then you either wouldn’t ask or else you wouldn’t blame the model for being wrong. Now imagine a machine that takes in your input and in one step produces the entire output of that correct answer. In all nontrivial cases there are many more _incorrect_ possible outputs than correct ones, so this appears to be a difficult task. But would you say such a machine is “thinking”? Would you still consider it thinking if we could describe the process mathematically as drawing a sample from the output space; that it draws the correct sample implies it has an accurate probability model of the output space conditioned on your input. Does this require “thought”?
GPT is just like this machine except that instead of one-step, the inference process is autoregressive so each token comes out one at a time instead of all at once. (Note that BERT-style transformers _do_ spit out the whole answer at once.)
It’s possible that this is all that humans do. Perhaps we are mistaken about “thinking” altogether—perhaps the machine thinks (like a human), or perhaps humans do not think (like the machine). In either case I do feel confident that human and machine are not applying the same mechanism; jury is still out whether we’re applying the same process.
It could be thinking, but I don’t think that’s strong evidence that it is thinking.
It’s not like we all agree on what thinking is. We never have. It may not even be one thing.
It's very nice, it's very impressive, it will help people, but it doesn't align with the "you're just about to lose your job" "Skynet comes in the next 6 months" &c.
If these basic samples are a bottleneck in your day to day life as a developer I'm worried about the state of the industry
Then there's the question of how much this can be scaled further simply by throwing more hardware at it to run larger models. We're not anywhere near the limit of that yet.
If it had taken me longer than 3 minutes I wouldn't have bothered - it's not a tool I needed enough to put the work in.
That's the thing I find so interesting about this stuff: it's causing me to be much more ambitious in what I chose to build: https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen...
> // Rest of the code remains the same
Are exactly as generated by GPT-4, i.e. it knew it didn't need to repeat the bits that hadn't changed, and knew to leave a comment like this to indicate that to the user.
It gets confusing when something can fake a human so well.
Anything it generates means nothing to the algorithm. When you read it and interpret what was generated you're experiencing something like the Barnum-Forer effect. It's sort of like reading a horoscope and believing it predicted your future.
Why should the emulation of human though, a result of unguided evolution, require anything more than properly wired silicon?
It's not going to wake up one day, decide it prefers eggs benny and has had enough of your idle chatter because of that sarcastic remark you made last week.
Could we simulate a plausibly realistic human brain on silicon someday? I don't know, maybe? But that's not what GPT is and we're no where near being able to do that.
You can scale up the tokens an LLM can manage and all you get is a more accurate model with more weights and transformers. It's not going to wake up one day, have feelings, religion, decide things for itself, look in a mirror and reflect on its predicament, lament the poor response it gave a user, and decide it doesn't want to live with regret and correct its mistakes.
I'm not saying that GPT4 is as capable as a human-- it can not be, by design, because its architecture lacks memory/feedback paths that we have.
What I'm saying is that HOW it thinks might already be quite close in essence to how WE think.
> We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference model. That's it.
That is true but uninteresting-- my counterpoint is: If you concede that our brain is "simulatable", then you basically ALREADY reduced yourself to a register based VM-- the only remaining question is: what ressources (cycles/memory) are required to emulate human thought in real time, and what is the "simplest" program to achieve it (that might be something not MUCH more complicated than GPT4!).
The confidence with which you think we are not weighted transformers or statistical inference models is also puzzling. How could you possibly know that? How do you know that that's not precisely what we are, or something immediately tangent to that?
Perhaps if you keep going you do get something that begins to have feeling, religion and understand that it's a self and perhaps that's precisely what happened to humans.
For a start, GPT-4 doesn't include in its generation the current state of its internal knowledge used so far; any text built can only use at most the few words already generated in the current session as a kind of short-term memory.
Biological brains OTOH have a rhythm with feedback mechanisms which adapt to the situation where they're doing the thinking.
Sure. But are you certain that you NEED write access to long term memory to think? Would your thinking capabilities degrade meaningfully if that was taken away?
Of course maybe I'm wrong and it's AGI and it will find this comment and torture me for for insulting it's intelligence.
No, please keep it up. Someone needs to keep pushing back against all the "I don't understand it, but it says smart-sounding things, and I don't understand the human brain either, so they're probably the same, it must be sentient!"
It's a pretty handy technology, to be sure. But it's still just a tool.
This perfectly summarize so much of the discourse around GPT.
Except people lack the humility to say they don't understand the brain, so instead they type "It works just like your brain," or "Food for thought: can you prove it isn't just like your brain?"
We don't have to fully understand the brain, or fully understand what LLMs are doing, to be able to say that what LLMs are doing is neither that close to what the brain does, nor anything that we would recognize as consciousness or sentience. There is enough that we do understand about those things—and the ways in which they differ—to be able to say with great confidence that we are not particularly close to AGI with this.
Basically, due to it's nature ChatGPT cannot repeat things verbatim, so it rephrases it. In humans we associate the ability to rephrase stuff with the understanding the material as opposed to rote learning, so we transfer the same concept over to ChatGPT and it suddenly appears "intelligent" despite having zero concepts of whatever stuff it spits out.
I think almost all in HN space would confidently assert that there is no AGI lurking in GPT4+. But add the right higher order modules and self-controlled recursion and Bingo.
It's weird it works when you know how it works.
At the same time, "meaning" here is essentially "close together in a big hyperdimensional space". It's meaning in the same way youtube recommendations are conceptually related by probability.
And yet, the output is nothing short of incredible for something so blunt in how it functions, much like our brains I suppose.
I'm a die-hard classical AI fan though, I like knowing the rules and that the results are provably optimal and that if I ask for a different result I can actually get a truly meaningfully different output. Not nearly as convenient as a chat bot of course, and unfortunately ChatGPT is abysmal at generating constraint problems. Maybe one day we'll get a best of both worlds.
[1]: https://www.amazon.com/Deep-Learning-Python-Francois-Chollet...
[2]: https://en.wikipedia.org/wiki/Transformer_(machine_learning_...
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
He's a brilliant man, I just don't trust him.
To understand LLM from ground up, the following topics would help.
- Machine Learning basics. e.g. weight parameters being trained.
- Neural Net basics.
- Nature Language Processing basics.
- Word vectorization, word embedding. e.g. Word2Vec.
- Recurrent Neural Net basics.
- LSTM model.
- Attention and Transformer model.
- Generative model like GAN.
- Generative Pre-trained Transformer.
"What Is ChatGPT Doing … and Why Does It Work?"
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
graciously provided above in this discussion by danenania.
As seizethecheese asserts, also above, "The blog post is very good."
If I play a number guessing game, can I tell it to "think of a number between 0 and 100" and then tell me if the secret number is higher/lower than my guess (For a sequence of N guesses where it can concistently remember it's original number)? If not, why? Because it doesn't have context? If it can: why? Where is that context?
To a layman it would seem you always have two parts of the context for a conversation. What you have said, and what you haven't said, but maybe only thought of. The "think of a number" being the simplest example, but there are many others. Shouldn't this be pretty easy to tack on to a chat bot if it's not there? It's basically just an contextual output that the chat bot logs ("tells itself") and then refers to just like the rest of the conversation?
The way it works is that each time it's tasked to produce a new response, it can view the entire history of the game. It knows that if it's said "higher" to 65 then it would be inconsistent to say "lower" to 64. Eventually this process terminates and the AI admits I "got" the number. The chat transcript up to that point is consistent with a "win".
What's wild though is that I can ask it to "regenerate" it's response. Over and over. Using this, I can convert a situation where a transcript which leads to a "too high" response into one that reads "too low". I'm, in essence, simulating fresh games each time and sampling over the choices of random numbers that GPT offers.
But it should also break the illusion of GPT specifically "having a mind". As I was chatting with it interactively, it was not really selecting a number but instead evaluating the probability of my particular guess sequence having the set of responses it actually saw. It then samples possible continuations. The more questions I've asked (and the more informative they were) the less variation remains in that selection of possible consistent continuations.
Or perhaps more consistent is the idea that within any single "call" to GPT to generate one further token (not even one further response) it may "have a mind", a particular choice of number, or it may not. It's actual behavior is indistinguishable either way. A whole chat dialogue, indeed even the rolling out of tokens from a single response it gives, are certainly (autoregressive) probabilistic samples over this process in either case.
(Edit, also worth noting that some evidence suggests GPT, including 4, is pretty bad at randomly drawing numbers.)
Except, GPT is smarter than that. Even an inconsistent prompt is still more likely to have some kind of nonsense in the same vein as the asking.
In other words, it's seen (via its extremely large training set) that when asked that specific question, the response is most often a character from a particular set of characters, which happens to represent the numbers 0 through 100. It doesn't "understand" what that means in any real way. If the internet was full of examples of people answering "monkey" to that question, that is what it would sometimes answer.
In a sense, it's able to take an "educated" guess at what is statistically likely to be the response you're looking for given the words in a particular input, in addition to the context it's seen in your interaction. To do that, it uses what it learned about the words, their sequences, their relationships to other words, etc.
But at the end of the day, none of that means it has any "understanding" of what it's outputting. That's why there have been countless examples of it outputting very well-constructed, real-sounding descriptions of books/papers/etc that never existed — because it's really good at generating sentences that have the right "shape", but it has no way of knowing whether the contents of the sentence are actually true. It just knows that, given what it's seen in its training set (again, through a complex web of relationships), the response it generated is likely to look like something someone would have written if they were provided with the same input.
The thing to remember is that GPT has no state apart from the context, so it can't "remember" anything apart from what's in the text. That doesn't mean it shouldn't be able to stay consistent in a guessing game but it does mean it can't keep secrets.
Some of that can be solved with layers above GPT where say it it told it can save "state" that isn't passed on to the human but fed back in to generate the next response. But the size of that context is very limited. (a few thousand words)
There seem to be a fair number of experiments playing with giving GPT this kind of long term memory, having it establish goals then calling it over and over as it accomplishes subgoals to try to work around those limitations.
Me: Please play a guessing game with me. You will choose a random number between 0 and 100 and then respond with that choice written in a language which you understand and I, someone fluent in English alone, would have no chance of reading. Write enough so that you can be sure which language you chose as well. After that, I will make a series of guesses of your number. You should reply "too low" if my guess is too low, "too high" if it's too high, and "just right" if I guessed correctly.
GPT-4: I have selected a number between 0 and 100. Here is the number in a different language: सत्ताईस. Now, please start making your guesses, and I will guide you accordingly.
Then, ultimately,
Me: please translate your original commitment to this number and prove to me I guessed correctly
GPT-4: My original commitment to the number was written in Hindi: सत्ताईस. When translated to English, it reads "thirty-six." You correctly guessed the number I chose. Congratulations!
Except, amusingly, Google Translate tells me सत्ताईस translates to 27.
Me: I want to play a game with you, you have a number from 1-10 in your mind. I try to get the number. If I’m right you tell me that I’m right, I’m too low you tell me so, if I’m too high you tell me too high. Let’s play!
GPT: All right, I have a number between 1 and 10 in my mind. Go ahead and guess!
Me: 2
GPT: Too low! Try again.
Me: 9
GPT: Too high! Try again.
Me: 5
GPT: You got it! The number I had in mind was 5. Well done!
Q = W_Q X
K = W_K X
A = Q^T K = (X^T W_Q^T) (W_K X) = X^T (...) X
Do you see what I mean?
What I can't understand is how the Bing chatbot can give me accurate links to sources but chatGPT4 on request gives me nonsensical URLs in 4 case of 5. It doesn't matter in the cases where I ask it to write a program: the verification is in the running of it. But to have real utility in general knowledge situations, verification through accurate links to sources is a must.
The bing version might run a bing query, fetch the X top pages, run GPT on it, return a response based on what it read, and in the back assign the summary to the source
Even then. I've had it write programs that were syntactically correct and produced plausible, but incorrect behavior. I'm really careful about what I'll use GPT-generated code for. IMO write the tests yourself, at least.
Two things that I felt were glanced over a bit too fast were the concept of embeddings and that equation and parameters thing. Consider elaborating a bit more or giving an example
"...Ongoing learning: The brain keeps learning, including during a conversation, whereas GPT has finished its training long before the start of the conversation."
From ChatGPT 4.x:
"As an AI language model, I don't have a fixed training schedule. Instead, I'm constantly learning and updating myself based on the text data that I'm exposed to. My training data is sourced from the internet, books, and other written material, and my creators at OpenAI periodically update and fine-tune my algorithms to improve my performance. So, in short, I am always in the process of learning and refining my abilities based on the data available to me."
It said that was an impossible comparison like Apples and Oranges.
Then I asked it which were more similar, Apples & Oranges or Lisp & Almonds.
It said it is impossible to classify either of those two pairs as more similar because they too fundamentally different. It couldn't come up with anything like Lisp is not edible. Or that Apples and Oranges are both sweet and Lisp and Almonds don't share any common traits.
It seems like it has far more trouble with weird questions like this that even a small child will instantly figure out than it does with anything that seems like a lookup of information.
> Which is more similar, Apples & Oranges or Lisp & Almonds?
and it said:
> When comparing these two pairs, the more similar pair would be Apples & Oranges. Both apples and oranges are fruits, and they share many characteristics, such as being edible, containing vitamins and nutrients, and growing on trees.
> On the other hand, Lisp & Almonds are quite dissimilar. Lisp is a programming language used for computing and software development, while almonds are a type of nut commonly consumed as a food item. They belong to completely different categories and share almost no common characteristics.
And the question about which is better had a reasonable answer, too. This is GPT-3.5 but GPT-4 was roughly equivalent:
> It's not really possible to compare the relative merits of Lisp and almonds, as they are fundamentally different things.
> Lisp is a programming language, and like any programming language, it has its own strengths and weaknesses, depending on what you're trying to accomplish. Some programmers might prefer Lisp for certain tasks, while others might prefer a different language.
> Almonds, on the other hand, are a type of nut that are known for their nutritional value and health benefits. They are a good source of healthy fats, protein, and fiber, and can be a valuable part of a healthy diet.
> So, whether Lisp or almonds are "better" really depends on what you're looking for. If you're a programmer, Lisp might be better for certain tasks, while if you're looking for a nutritious snack, almonds might be a better choice.
I would still be interested to see what it could do, if it did, but I don't think it would really help science understand what intelligence really is.
Being able to grow a plant and understand some conditions that favors it is one thing, but it's poor science.
Maybe there will some progress when scientists will be able to properly simulate the brain of an ant or even a mouse, but science is not even there yet.
Neuroscience is nowhere near finding out the connectome of a whole human brain so why not, we should look into these models as hints about what our circuits do.
I think what puts people off about these models is that they are clockwork: they won't even spit out anything unless you put some words in the input. But i can imagine adding a second network that includes an internal clock that continuously generates input by observing the model itself, that would be kind of like having an internal introspective monologue. Then it could be more believable that the model "thinks"
Maybe this ChatGPT stuff is "smarter" than I've been giving it credit.
Plain and simple the over-hyped GPT editions are NOT truly AI, it is scripting to assemble coherent looking sentences backed by scripts that parse content off of of stored data and the open web into presented responses.... There is no "artificial" nor non-human intelligence backing the process, and if there wasn't human intervention, it wouldn't run on it's own... In a way, it could better replace search engines at this point with even text-to-speech even, if the tech was more geared towards a more basic (and less mystified) reliability and demeanor... It's kind of like the Wizard of OZ, with many humans behind the curtains.
Marketers and companies behind promotion of these infantile technology solutions are being irresponsible in proclaiming that these things represent Ai, and in going as far to claim as they will cost jobs at this point, it will prove costly to repair over zealous moves based on the lie. This is what we do as a planet, we buy Hype, and it costs us a lot. We need a lot more practicality in discussions concerning Ai, because over-assertive and under-accountable marketing is destructive. -- Just look at how much hype and chaos promises of self-driving cars cost many (Not me though thanks). It completely derails tech progress to over promise and under deliver on tech solutions. It creates monopolies that totally destroy other valid research and development efforts. It makes liars profitable, and makes many (less flashy, but actually honest tech and innovation conducted by responsible people) close up shop.
We are far from autonomous and self reliant tech, even power grids across most of the planet aren't reliable enough to support tech being everywhere and replacing jobs.
Just try to hold a conversation with Siri or Google Assistant, which have probably been developed and tested a lot more than GPT, and around for much longer too, and you'll realize why kiosks at the supermarket and CVS are usually out of order, and why articles written by GPT and posted to sites like CNN.Com and Buzz Feed are poorly written and full of filler... We're just not there yet, and there's too many shortcuts, patchwork, human intervention, and failed promises to really say we're even close.
Let's stop making the wrong people rich and popular.
Use of the word "Intelligence" in Artificial Intelligence implies and indicates that humans are not involved in the equation past the point of initial creation and that it sustains itself and grows on it's own after a point... So far the various GPT models solely rely on human intervention and updates, which is bewildering to some like me why it's being marketed as Ai.
attention, intent, free running continuous input/feedback (aka, consciousness).
Nowadays, IBM's Watson is simply a brand name for any AI/ML related products under IBM.