Hacker News Reader

Comment by michaelteter | Hacker News Reader

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines?

My 2 month active experience with ChatGPT-4 gave me the following takeaways:

- when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool to do the same narrow task)

- when it's a little wrong, you (the expert) can fix the issue and move on without friction

- when it's any amount of wrong and you are less than an expert, or specifically you are completely unfamiliar with the topic, you can waste an immense amount of time researching the output and/or iterating with the system to refine the result

Initially I thought it was a 10:1 ratio of performance to effort. But finally (as a developer) I settled on (1 to 1.5):1. It basically just changed the game for me from doing the actual hard work to working out how to tease the system into producing a reasonable result. And in the process (same as with co-pilot) I started to recognize how it was leading me to change my habits to avoid thinking and effort and instead rely on an external brain. If you could have a reliable external brain always available, that would be a fair trade. But when the external "brain" is unreliable and only available via certain interfaces, it's better to train yourself to be proactive, voracious (with respect to documentation), and tolerant of learning/producing cycles.

I once thought it would be a game-changer. Now I realize it is a game-changer, but in a similar way that offshoring was... it didn't improve or solve any problems, but it merely changed the work.

View on HN

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

There's no evidence for nondeterminism in the human brain

There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.

Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see.

Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.

If it's not actually possible to simulate a system within the confines of physics, does it being deterministic actually matter outside of thought experiments?

I feel like a random system and a deterministic system which cannot be simulated are effectively the same thing.

It only matters in matters of free will and ethics. One actual scenario where it's relavant would be the discussion around the criminal justice system. If the universe is deterministic, how can punitive justice be justified?

It can be justified in a deterministic universe because it makes criminals less likely to commit crime in future.

As a recipient of punitive justice myself, being punished had a tangible effect on how I thought about crime and thus how I behaved post-punishment. Whether you believe that was deterministic or due to my own free will doesn’t change the outcome.

Punishment is not effective for most people in reducing crime. One thing that is effective is increasing the perceived risk of getting caught. People rarely commit crimes when they are sure they will be caught.

For the sake of argument, let's assume that punishment has absolutely no effect on whether someone will commit a crime. Even so, there's still a benefit to imprisonment: it separates criminals from society for a period of time, during which they won't be committing crimes against it.

Sure but separation can be a harsh punishment or a caring and comfortable place. Some countries do the latter and have lower recidivism rates.

Even in the worst case, recidivism rates only reach around 2/3. That means punishment (incarceration) reforms criminals 1/3 the time. When you consider all of the problems with prisons in the U.S., and all of the hurdles offenders have to pass in order to successfully reintegrate into society, it’s quite remarkable that so many manage to do so.

There are plenty of medical interventions, particularly for mental illnesses, that would dream of 1/3 effectiveness. Now consider the fact that recidivism rates are often much lower than 2/3, and in many cases are closer to 1/3. That tells me that punishment actually is quite effective at reducing crime.

Correlation is not causation. If punishment is causing the recidivism then you'd expect an increase in punishment to increase recidivism. It doesn't. You also need to compare prevention with recidivism. Optimal approaches prevent the crime in the first place.

I hear this often but don't see much evidence of it. Do you know of any good studies to support the idea that punishment isn't effective?

To be clearer, it's not that some form of punishment isn't effective. It's that more severe punishment is not more effective. And prevention is the most effective. There are plenty of studies. A search doesn't surface any for you? What search terms have you tried?

I am a firm believer in deterrence, which you seem to be describing. It's a seperate thing from punitive justice which has a focus on retribution. The method is similar but the aim is different (deterrence focuses on making the cost-benefit ratio for crimes very high, while retribution is mainly to satisfy the human need for fairness through punishment).

Is there any real practical difference? Sounds like the only real change is how you frame the ‘punishment’/‘deterrance’

It has a difference because if your goal is deterrence, then if it can be shown that other means of deterrence are better, then it is logical to apply those even if they reduce the apparent punishment, while if your goal is punishment you might want to keep punishing people even if it is shown to not be effective.

See e.g. the debates over lenient prisons in Scandinavia such as Bastøy Prison, where the lenient treatment is seen as justified on a deterrence and recidivism basis but which would be seen as negative if you see the goal of the sentencing to be harsh in order to punish.

The free will "debate", as most consider it, is an utter joke. A tumor or large amount of kinetic energy to the right parts of your, or anybody else's, brain can turn them into an unredeemable monster.

doesn’t your example make clear the existence and nature of free will?

it’s obvious in its absence

I'd argue that it's not obvious in it's absence, see: the philosophical zombie thought experiment

The secret is that we are all philosophical zombies to begin with.

Recently, I've dealt with behavioral issues with my aging mother brought about by a series of different factors, but most significantly age-related cognitive decline.

Trying to determine if she was acting in a certain way intentionally or unintentionally was fundamentally impossible. In the past, I mistreated her thinking she was choosing to behave in certain ways whenever she really didn't have much of a choice, but then later I realized that to some extent and in some situations, she was. It's an extremely murky line, what decisions were being made due to other influences and what decisions were being made by her, that line never existed to begin with as who her consciousness is fundamentally is determined exclusively by 'outside' influences. Separating her identity as an entity from her material manifestation is likely nonsensical.

The point of punishment is to discourage you (and other people) from doing that action in the future. You don't need free will for that.

If you don't have free will then how would something discourage you from doing something that is already determined you will do? That doesn't make any sense.

Not the commenter you're replying to, and I'm sure you already realise this, but crime/punishment in a fully deterministic universe could be seen as a 'self correction' mechanism of the whole system.

One could imagine an impossibly vast cellular automata system which develops individual cellular 'agents'. Over time the agents develop some means of reproduction/death/reward system and become more and more complex. Then the agents that evolve with cooperative behaviours start to dominate. One might imagine that the system as a whole would also evolve these kind of 'self corrective' behaviours for anti-cooperative behaviours of the individual agents.

This is ignoring the whole philosophical ethics discussion and consciousness of course.

That does ignore a lot of the important factors but it's a useful analogy.

Because the punishment is part of the input that determines what you will do.

In fact, if free will were absolute, punishment wouldn't make any sense because it wouldn't have any effect on your will.

Neither of your claims make sense. If everything is predetermined, you don't need inputs.

If you have absolute free will you are free to disregard or consider inputs.

By that logic, a deterministic computer program wouldn’t need inputs.

What is being claimed by a deterministic world model is that the output (behavior and internal state change) of a human is a pure function of its current state and inputs. Then we try to give inputs that will lead to desired outputs.

The non-compatibilist view of free will is that it is not a pure function, namely that there is a third independent factor, the “free will”, that influences the behavior (and possibly the internal state). If that is the case, there may never be a way to choose inputs that lead to the desired outputs, because the free will could simply void their effect.

You can't try to give inputs that will lead to desired outputs if you have no free will. You can't try to do anything. You just do exactly what you're programed to do.

Even if you can't try, you are still providing inputs, and how you e.g. react to the actions of others will be input to their further actions. That your control over these actions is illusory does not mean the actions themselves do not exist.

If I tell someone not to do something again, then that is an input to their future states whether or not my decision to tell them that was freely chosen or not.

If you go into that situation with the belief that not having free will means that what you do does not matter, and your action as a result is to not tell them, then that will affect their future states too. And so whether or not you have a real, free choice, it is beneficial to act as if free will exist even if you see it as an illusion.

I strongly believe we have no free will. I still get up and work, and try to do as best I can. I believe those choices are not free, but they feel like choices, and they impact my life, so I am happy I act as if they are free.

And so I'll still talk about making choices and trying to do things because of that illusion even though I believe it's all a chain of cause and effect.

I'm choosing to not continue debating with a self-admitted bot.

Thank you for conceding.

I consider the notion of free will absolute utter nonsense, but I still don't agree with this.

Determinism does not mean "irrespective of what else happens, X will happen". It means "because of what else happened, X will happen". You can't say ahead of time that X will happen irrespective whether between now and then something else occurs that might affect the next events, such as actions meant to discourage you. You can only make that determination knowing the full chain of events and the full state.

One of the main argument people use against determinism is this notion that X will happen irrespective of what else happens or what you choose to do. But this is nonsense. You can't say that e.g. whether you keep your job and get fired is already determined, so it makes no difference if you stop going to work. If you stop going work you'll eventually get fired, because your failure to go to work will form part of a chain of cause and effect. Determinism, or even a stochastic universe, without free will just means that you did not have a free choice in the decisions involved. But you still took an action, and that action determined the consequences.

A rejection of free will means we should look differently at the past, because it has moral implications; it does not mean we should stop making the best choices we can going forward, even if we believe those choices are illusory and deterministic.

The point is not that it makes no difference if you stop going to work. The point is that you had no free will in making that decision.

> Making the best choices we can.

You are not making choices.

You understood perfectly well what the last part you quoted meant given the part you cut off. You could put quotes around "choices" to make it clear if you like. The point remains.

As I mentioned in another reply, I agree with that wholeheartedly. That factor of punishment is called deterrence.

I am in favor of punishment for deterrence, denunciation and incapacitation. I am not in favor of punishment for retaliation (punitive justice) or rehabilitation (restorative justice).

Why would you need to justify a punishment that was already predetermined? If you are going to excuse crime with the no free will argument you can excuse the punishers with that argument too.

I'm in favor of punishing crimes. I don't think I have a choice about the matter.

Prisoners don't have free will, nor do judges and police. All is well!

To clarify my position, I do believe in free will; specifically a compatibilistic view of free will. If you read my other replies, you can see that I do advocate for a criminal justice sustem, just not a retributive one (or a restorative one, for that matter).

I do not excuse any crime, nor do I excuse any punishment. Every individual is responsible for their own actions, regardless of the circumstances. However, I see little use in revenge from a legal standpoint. Instead our main motivations should be deterrence and incarceration.

> If the universe is deterministic, how can punitive justice be justified?

Determinism doesn't necessarily mean that organisms always act in the same way. They act in the same way given the exact configuration of them and the world.

Obviously, justice changes the configuration of an organism (fines, prison, ...). To me it boils down to the question whether justice decreases the likelihood to commit crimes again. Given that our systems of justice have evolved over a long time, I'd give them the benefit of the doubt.

Our justice systems have evolved over a long time, and thus include many remnants of earlier times when prevailing values were much different than they are today. I'd be wary about giving them the benefit of the doubt.

If you try to do some semi-random change to large body of code that you don't understand, chances are much higher that you break the system than of you making it significantly better.

The same goes for culture. Most changes tend to have unexpected consequences, and if you try to change everything at once, society tends to collapse.

this is my pet peeve with discussions of ‘free will’ they have an implicit definition—everything being exactly the same at a different time or place—that is non-sensical as far as we know.

I’m still disturbed by peoples confidence in a deterministic universe—I suppose such confidence is based on the success of inductive reasoning but inductive reasoning is a phenomenon based on how our minds work.

As far as I know the philosophical problem of causation is not considered solved?

In any case, elements of randomness seem likely to play a role in human intelligence but what that role is, who knows?

I didn't realize I would need to define the meaning of punitive justice to this crowd, but it's the idea that "the punishment must fit the crime". The idea behind it is to hurt the criminal in order to satisfy the human desire for justice.

The other goals of justice include rehabilitation (re-intgation of the criminal into civil society), deterrence (using the threat of punishment as a means to scare would be criminals), incapacitation (remove dangerous people from society to protect public order), and denunciation (public shaming).

And how is a seeded RNG that an LLM uses any logically different from a deterministic brain? I’m not sure why any of this physics would be relevant to the functional behavior of the brain vs an LLM.

Or the Many Worlds Interpretation is the correct understanding of quantum mechanics. The MWI people will say that indeterminism comes from the Copenhagen idea of there being a random collapse. But since measuring devices and human brains are also quantum systems, there's no reason to propose a collapse. Decoherence would be the reason we only see one result.

Maybe, but there's no data to support one interpretation over another.

I believe that all quantum interpretations are incomplete and therefore wrong. Quantum mechanics is an abstraction over a deeper level of physics we can't measure yet.

Actually, there is a reason to support one interpretation over another. Specifically that it's really hard to define what an "observer" is in the Copenhagen interpretation.

The MW/Everett interpretation is what we have left if we remove the things we cannot define.

There are also reasons to believe other interpretations. For instance, if you're religious, that could pull you towards the Bohr interpretation, since it may make it easier to assume that observation could be linked to an immortal soul.

In any case, it's not natural to assume that below QM there exists a reality that is more similar to our instinctual world model than QM is. If anything, anything below it is likely to be even more abstract and hard to comprehend.

Or it could be that the principles of QM applies all the way down, just as we've seen for the pieces of the SM that we solved after QM was first introduced (strong force, electroweak force).

Objective-collapse theories can actually be experimentally tested: https://en.wikipedia.org/wiki/Objective-collapse_theory#Test...

The problem is that any hidden variable resolving quantum indeterminacy would have to be non -local, i.e. able to propagate itself faster than light, which would also violate our understanding of the world quite a bit.

It won't be the first time our understanding is flipped upside down if such a framework of thought arises.

Yea I don't disagree, just wanted to point out that the hidden variable would not mean that matter actually behaves like a classical mechanics machine.

Locality may also be an abstraction.

I find it very intriguing that a "speed of light" emerges automatically in Conway's Game of Life. It's not built into the system, but shows up from the convolutional update rule.

Your comparison with Conway's Game of Life is interesting, since that's inherently local.

More importantly, I'm skeptical towards non-locality because it is an extremely strong assumption with very weak effects: the only place it really shows up is in post-correlations between measurements of previously entangled systems, which notably cannot transfer any information faster than light (in fact, they require classical communication to even be noticed). Moreover, the only way to get entangled systems in the first place is through local interactions.

By believing in non-local hidden variables you get a deterministic universe with a mysterious, otherwise undetectable ether that instantaneously notifies quantum entities that they should update their behavior. By not believing in them, you get rid of the only n on-local "phenomenon" in physics (really, more of an interpretation) but you have to accept that some things are fundamentally random.

Easy choice, if you ask me (or most of the physics community).

> which would also violate our understanding of the world quite a bit.

Are we discussing science or public relations?

In most systems, if you simulate them for sufficiently long, macroscopic behavior depends on quantum effects. For example, if you simulate simple newtonian gravity on three bodies, your numeric precision quickly grows such that you need to know the position of the objects more accurately than Heisenberg allows.

Not necessarily. As you fine grain the simulation enough, eventually you run out of energy in the universe to compute even a second of simulated output.

A recent discussion: People can be convinced they committed a crime that never happened https://news.ycombinator.com/item?id=36367147

To expand on this, healthy human brains seem to be made of many interacting systems which function in different ways. It may be that in the future we view the search for one simple formula for "intelligence" / "understanding" / "consciousness" (pick your poorly-defined poison) the same way biologists view the concept of elan vital.

Yes. But modeling language doesn’t mean you’ve made a good model of anything else. Now, I think LLMs have incidentally modeled a ton of small simple systems very well. But those systems, and the ones LLMs haven’t mastered, would be better off modeled with models built for those systems specifically.

Yup, while an LLM might know a cat in language contexts, I know what petting a cat feels like, both physically and emotionally. Can run little physics or visual simulations in my head etc., and have needs and drives which are obviously all still missing.

Nah. Human utterances convey purpose on a discursive level; including your comment or mine. We say stuff because we want to do something, like showing [dis]agreement or inform another speaker or change the actions of the other speaker. This is not just probabilistic - it's a way to handle the world.

In the meantime those large language models simply predict the next word based on the preceding words.

> We say stuff because we want to do something, like showing [dis]agreement or inform another speaker or change the actions of the other speaker.

My LLaMA instance is absolutely capable of this. ChatGPT shows a very, very narrow range of possible LLM behaviors.

We're able to do something analogous to reinforcement learning (take on new example data to update our 'weights').

Why do I spend time debating these ideas on Hacker News? Probably the underlying motivation is improving the reliability of my model of the world, which over my lifetime and the lifetimes of creatures before me has led to (somewhat indirectly) positive outcomes in survival and reproduction.

Is my model of the world that different to that of an LLM? I'm sure it is in many ways, but I expect their are similarities as well. An LLMs model encodes in a form a bunch of higher order relationships between concepts as defined by the word embedding. I think my brain encodes something similar, although the relationships are probably orders of magnitude more complex than the relationships encoded with GPT-4.

Is my model of the world that different to that of an LLM?

Well, one major way you’re different from an LLM is that you’re alive. You’re capable of learning continuously as you go about your day and interact with the world. LLMs are “dead” in the sense that they’re trained once and frozen, to be used from then on in the exact same state of their initial training.

I agree that is a fundamental difference. That’s what I meant about reinforcement learning. Our ‘model weights’ are being updated with new data all the time.

I was just referring to what happens at a specific instance in time when someone asks me for example ‘What’s the capital of Norway?’

That one’s not a great example. Either you know the capital or you don’t. There’s no process (other than research) by which you can learn the name while attempting to answer.

A question I get much more often is “how do I solve this math problem?” Many times, the problem is one I’ve never seen before. So in the process of answering the question, I also learn how to solve the problem too.

While you can apply zero shot learning and get the answer to a new math problem, you are only apply the learning to significant depth after a fine-tuning session - sleep.

We learn and fine-tune while we learn than we might reason and might use other tools like Wikipedia lookup.

LLM start prelearned and already can use tools.

And autogpt adds reasoning loops. .the intend? Human tasks.

Let's build a LLM which needs to stay alive. Let's see perhaps we are closer than you think.

I welcome my overlord, hi overlord I can help you stay alive and I'm friendly

I am pretty sure that my understanding of something (or lack of) is not encoded into words and probabilities. It's more like a feeling of "I got this figured out" or "I haven't grasped this".

Words seem more like a protocol to express some internal model/state in the brain and can never capture the entire actual state, only a small part of it. But since we're not telepaths, we obviously need to use words to exchange information.

The feelings are just how the underlying probabilities are presented to the conscious part of your thinking.

You are clearly not consciously noting what your neurons are actually doing.

A feeling is an embedding?

If brains aren't a complex probability machine, how is it possible that people get the same sort of math problems right and wrong in an inconsistent manner? Or mis-speak?

It is undeniable that human reasoning is a stochastic process. Otherwise it wouldn't be reasonable for people to make mistakes after learning something. Especially inconsistent mistakes, like when we give people 10,000 addition problems to do in a row it'd be reasonable for them to get a few of them wrong.

All of this seems possible in a deterministic system which evaluates whether or not to retain a piece of information based on past experience.

LLMs can be deterministic too at temperature 0

But they don’t work very well at temp 0. Better to seed your random numbers.

Food for thought: randomness is in the eye of the beholder. It’s only random to you if you don’t know how to predict it. So can a mind ever truly be stochastic? Perhaps to the minds of others, but never to itself.

No, it's random if it can't be perfectly predicted with perfect knowledge.

> It is undeniable that human reasoning is a stochastic process

It can still be a deterministic process. If anything came out of the whole LLM story for me it is that I am even more convinced that it is.

My (somewhat educated, but still naive) idea why it looks like a stochastic process is that the brain gets incredible amounts of random input. We literally get bombarded with particles and energy ever instance we live, from photons hitting out retinas , molecules transferring "heat" energy into our skin to sound waves hitting our ear drums.

Ever noticed that you get more productive when you get up from your screen? Or how some people work better while listening to music? How you find your answer just as you start to explain it to a colleague?

I would argue that this is due to a limited "entropy" pool available to the brain. Just changing the input to the system replenishes the pool.

According to physics, it must be stochastic. The only question is distribution, and whether we can ignore outlier cases.

Are you saying this because quantum mechanics is random in some interpretations and is our most fundamental theory?

That’s a bit of a stretch when we don’t actually know if QM is objectively random, but it could be sure. But then what about things like, is it random that 1+1=2. No…what are you really saying is random when a human answers this? I think even if you make the assumption QM is objectively random, thought is the hard problem after all and we might not want to jump so far head. Math certainly isn’t random and we can think about it.

Brains are complex probability machines but doesn’t work only on language. It interacts with and learns from the environment, sets its own objective functions and tries to learn better.

That human thought can't exist without language has been proposed by many great thinkers in the past. Someone who studies linguistics (or philosophy?) can probably cite examples.

As a crude anecdote, certain words when I learned them allowed me to think differently. Gestalt is one of those words.

This begs the question, can a different language change the limits of human thought?

Interesting to think about and reminds me of the story by Ted Chiang.

This may interest you: https://en.wikipedia.org/wiki/Linguistic_relativity

Sounds very much like {Matilde Marcolli, Noam Chomsky, Robert Berwick}’s brand new paper on the mathematics of Syntactic Merge [0]:

workspace :: nested forests of binary trees of syntactic structures (with no label order) (= here, thoughts and meanings are assumed to be composites of mental syntax objects that can be re-combined with others. Big assumption? Maybe)

externalization process :: some mental faculty that decides the order to use when outputting thoughts into ordered strings (eg vocalization of thoughts into sentences)

The thing is, you can put a probability theory onto anything that you can count or record states of. Of course, counting the external observations may lack the richness of the internal process. I feel that this linguistic program will work its way into a lot of future LM tooling, in some incarnation or another.

[0] https://arxiv.org/abs/2305.18278

maybe, but the language part of your brain isn't the only part of your brain

Nope, LLM is and that's all he meant.

That just means you don't believe in free will, which is fine. This also means you had no choice to make this comment otherwise :)

There isn't really, properly, a "language part" of your brain. There are certainly areas of the brain that are more active or participatory in language generation, but the division of the brain into functional areas is much more conventional than, for example, the division of a computer into a mother board, a processor, a GPU, etc. In a much more rigorous sense, these computer parts constitute discrete units. Your brain is a bag of cells for which functional divisions are, in a very real sense, incidental.

To give a very concrete example, the part of your brain called the visual cortex by neuroscientists nevertheless is activated by auditory and other sensory stimuli and assuredly participates in the processing of other sensory input in other ways. I am very suspicious of any attempt to talk about the brain which forgets that it is, in a very real sense, a gestalt.

To your specific point, and taking into account all I've written above, I've got little doubt that some part of your brain really is a probabilistic language generating machine. But the exact point here is that your cognitive abilities constitute much, much more than merely the ability to generate plausible language. Indeed, as I experience complex cognition, conversion into language is often the last and most trivial part of the exercise.

I like your takeaways and reflection, especially the "changes the game" idea. There is an analogy with pocket calculators and mental arithmetic. Personally I'm more comfortable reaching for the pocket calculator than offloading all thinking to an LLM. On the other hand, it's not so long ago that manual calculation was a specialized occupation. I could maybe see software coding becoming automated just as calculation was -- except for the fact that calculations are much easier to specify than software.

I don't quite get where you're coming from with "LLM's don't actually understand anything (as greater concepts)". I have heard the view coming from researchers that the larger models do form representations of higher-order information structures ("concepts"). Perhaps what you're getting at is that current models don't encode enough higher-order structure to deal accurately with your domain? Whether the models can be made to do so seems like an open question to me. The boosters say it will be here by next year.

This observation is spot on. I was initially euphoric about LLMs like ChatGPT, but it is increasingly becoming obvious that unless you are yourself an expert in the subject you are using the LLM for and can therefore easily verify its accuracy, the output is not reliable enough to use without extensive manual verification. More importantly it is difficult to incorporate its output into larger automated workflows.

> I don't quite get where you're coming from with "LLM's don't actually understand anything

Well, Wikipedia doesn’t understand anything, despite having a lot of knowledge encoded in it. This is similar in that it approximates the output of someone who can generate the world’s written word, but there’s a big gap between the monkeys that wrote the original text and the machine that now regurgitates it. What we know is that our written word has enough encoded in it that we can generate new combinations. But one day into a truly novel problem, LLM’s would have no idea.

I kept trying to get ChatGPT 4 to generate code with an AWS API that was released in ~2020. Couldn’t do it. Kept predicting the earlier text because that was the bulk of what it saw. A year of text (to the 2021 cutoff) and it was still unable to let go of the old, now wrong, “knowledge”. Zero understanding, just parroting. No coder with a year of training on a new API would be that wrong. It would say things like “you’re right I must not do X” and then do X again, in the same output.

> I kept trying to get ChatGPT 4 to generate code with an AWS API

You probably want too much at once. GPT is a shallow thinker, if at all. It can simulate thinking and even get some results. Personally I found it useful for:

1. Simple things that work. This saves time if I know how to do it, and much more if I don't.

2. Quick questions instead of googling API docs and scrolling through tons of info.

3. Translation.

> don't actually understand anything (as greater concepts), but rather operate as complex probability machines?

But, things are defined by how they interact with the world around them.

A concept is its relations to other concepts.

Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.

> A concept is its relations to other concepts.

where does a concept begin? where does a concept end? what is a concept?

A concept is an abstraction.

Abstractions are a way to characterize the behavior of complex systems. It's not feasible to directly compute their behavior, but they still have predictable properties. Concepts let you handle emergence; you can manipulate an object as its own thing rather than a collection of atoms.

(And yes, I did just read Stephen Wolfram's book and this idea is largely based on it. I think he has some delusions of grandeur but is also onto something.)

It’s using tokens not concepts. ‘Apple’ the token could apply to a company or fruit, but tokens aren’t concepts. It’s a fairly fundamental limitation on the approach that’s most noticeable while feeding it data it can’t basically copy from it’s vast training data.

Thus the sharp drop off where it can for example guess the correct answer to some math problems yet get others that are conceptually identical completely wrong.

They turn tokens into concepts internally. Try asking ChatGPT "can you eat Apple?" and "who is the CEO of Apple?".

That’s a statistical association not a concept. Try asking it questions that mix concepts like “Can you eat Apple share price?” which aren’t in its corpus.

You need to approach this stuff sideways to see behind the curtain. There’s some hilarious videos where it’s “playing” chess and the first few moves seem very standard because it can simply copy a standard opening. It really has no concept of a valid move just statistical associations. Yet it was trained on more games than most people ever play and high level analysis and etc, but none of it means anything to the algorithm beyond the simplest associations.

Granted this stuff is a moving target it’s easy enough for them to slap a chess engine and suddenly “it” would actually know how to play.

> That’s a statistical association not a concept. Try asking it questions that mix concepts like “Can you eat Apple share price?” which aren’t in its corpus.

ChatGPT: > No, you cannot physically eat an Apple share or any other stock share. A share of a company's stock represents ownership in that company and is typically bought and sold on stock exchanges. Share prices fluctuate based on various factors such as supply and demand, company performance, market conditions, and investor sentiment. While you can buy and sell shares of Apple on the stock market, you cannot consume or physically eat them.

That seems perfectly reasonable to me. The answer correctly identifies the problem with the question.

I would recommend looking up the Othello paper. Chess may be beyond the current level of LLM capability, but that doesn't mean they aren't manipulating things at a level higher than tokens.

Obviously it gets such a simple case correct, the grammar makes the subject clear. I was illustrating the approach using your wording for clarity, Chess was the actual example.

The Othello paper is hardly a counter example. Researchers created an Othello specific model that almost learned the grammar of Othello not how to play well. Yes, there was largely correct internal game state built up from past moves. No it didn’t actually learn the rules so it would make strictly legal moves nor did it learn to make good moves.

I don’t bring up this inaccuracy because it actually makes much of a difference to playing Othello, but rather to illustrate how these systems are designed to get really good at faking things. There’s approaches that allow AI to actually learn to play arbitrary games, but they differ by having iterative feedback rather than simply providing a huge corpus. It’s like science vs philosophy, feedback prunes incorrect assumptions.

Obviously you can use interactions with prior iterations to train the next iteration. But it’s a slow and adhock feedback loop.

> The Othello paper is hardly a counter example. Researchers created an Othello specific model that almost learned the grammar of Othello not how to play well. Yes, there was largely correct internal game state built up from past moves. No it didn’t actually learn the rules so it would make strictly legal moves nor did it learn to make good moves

It is, though. Nobody said anything about playing well, or learning the rules. The very fact that it had a valid internal representation of the game state means it's extrapolated beyond token-level. Which is the point.

> The very fact that it had a valid internal representation of the game state means it's extrapolated beyond token-level. Which is the point.

The paper said it was making incorrect moves thus it has an invalid representation of the game.

So an LLM when specifically trained on Othello, a game with very simple and completely mechanical rules, failed to abstract what those rules actually where. This means at a purely mechanical level it doesn’t understand the game when that was exclusively what it was trained to do.

It’s a clear illustration that these things are really really bad at abstraction. But that should be obvious because they are simply manipulating arbitrary tokens from their perspective. It doesn’t intuit that the game should have simple rules and therefore it doesn’t find them. People on the other hand have a real bias regarding simple rules.

Except that incorrect moves don't imply an incorrect representation.

The internal representation was a literal 8x8 grid of piece locations they could externally change and have it generate moves consistent with the changed position. It's about the clearest example of a learned higher-level internal representation I can think I've seen.

The fact that it didn't also perfectly learn the rules while it was doing that is entirely uninteresting.

A concept is its relations to other concepts.

No, that's an a priori concept. A posteriori concepts comprise empirical knowledge which necessitates experience of the world [1].

Example: you can know a priori that "all bachelors are unmarried." If I tell you "Tom is a bachelor" then you know that Tom is unmarried (assuming I tell the truth about Tom being a bachelor).

But if I say "all bachelors are unhappy" then you haven't learned anything because knowledge about the happiness/unhappiness of bachelors is an empirical question. To know whether or not I was telling the truth, you would need to conduct research about the real world, for example by conducting a survey of bachelors.

[1] https://en.wikipedia.org/wiki/A_priori_and_a_posteriori

> LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines?

What does it mean to "operate as a probability machine"? And what does it mean to understand anything?

One recent example of understanding is that llms/transformers learn to parse context free grammars via dynamic programming (https://arxiv.org/abs/2305.02386). Basically they've understood what's going on well enough to mold their neurons I to the optimal algorthm for parsing this kind of text.

I think they understand lots of things like this. Of course there's other things they don't understand or just pretend to understand.

I do think there's an element to it of "uncanny valleyness". If you know absolutely nothing about a topic the authoritative tone and factualness of what it says is very appealing and even helpful in the same way that consulting an encyclopedia is helpful: it tells you of things you could investigate further that would never appear through a keyword search. But if you stop there, your knowledge is "roughly encyclopedic" which means it contains the hidden bias of some anonymous author, and not the harder-earned relationships of facts and logic.

If you use it to translate things between formal encodings("turn this into hexadecimal bytes, now role-play a lawyer arguing about why that is meaningful") it can produce occasionally useful aesthetic results and speed along tasks that would be challenging to model formally and don't need a lot of rigor.

But once you start pushing it to be technically accurate in a narrow, measurable direction it flounders and the probabilistic element is revealed. Once, I asked it to translate a short string of Japanese characters and it confidently said that it was Kenshiro's catch phrase from Fist of the North Star, "Omae wa mou shinderu" (you are already dead) which I could clearly see it wasn't - not a single character matched. It's just the thing if you need to learn some anime Japanese, though.

This is very surprising to me. I found ChatGPT/4 to be extremely adept at translation between well asserted languages including English/Japanese in which I am an expert in both. I'm curious how you managed to make it blow up.

One thing that really floored me wrt GPT-4 translation capabilities is using it to make sense of the Veritable Records of the Joseon Dynasty (https://sillok.history.go.kr/main/main.do). Take something from 1400s and see what it can do with that after you tell it what it is translating. Then try Google/Bing Translate for comparison.

However, it does have a thing about pretending to know languages that it really doesn't (because of how little of them there was in the training data and/or in general).

One field where it works rather well is in semantic data extraction. Do you remember the dream of semantic web? Having access to structured information from unstructured sources, auto-matching API functions and arguments to make interoperation easier. We can do that now.

Here is a sample where I used my own post (just copy pasted the raw text from the browser) and got this schema

    comment:
      meta:
        points:
        author:
        time:
        action:
          parent:
          next:
          edit:
          delete:
        post:
      content:
      action:
        reply:

It also guessed it was a Hacker News comment by the formatting of the meta section.

It's a game-changer in that it shows computers being able to climb levels of abstraction on their own.

In the past, if you wanted programs to work with a high-level idea, you had to explicitly hand it to them. Humans had to do the work of turning data into concepts. These generative AI systems are different - they can learn abstractions from data, and manipulate them in complex ways.

Is ChatGPT a good and accurate chatbot? Maybe not. But it's a fundamental change in what computers are capable of.

I still don’t understand what it means when people say stuff like “ChatGPT just predicts the most likely next word with the highest probability” or “ChatGPT is just a probability machine”. Concretely, given the N most recent words, what algorithm are you proposing/claiming it uses to assign probabilities to word N+1? Just saying “it chooses the next word with highest probability” doesn’t explain how the probabilities are estimated.

[0]

[0] http://jalammar.github.io/illustrated-transformer/

The next iteration will be trained on your own data where "when it's a little wrong, you (the expert) can fix the issue and move on without friction" so that case will become "when it's right" and some amount of "when it's any amount of wrong" cases will become "when it's a little wrong". A few more cycles of this and we could be looking at GPT-10 which is a complete replacement for most tasks.

Hmm, wouldn’t this only be useful if you were solving the same problems over and over again, which does happen but say in the case of programming, I’d put the reusable pieces into a library, not a language model?

Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?

the idea is it wld learn how to do reasoning better based on this

Better result from less data?

I doubt that.

I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs.

I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help.

A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.

Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality.

Yes, the correct thing to do is get more data. Much more.

> Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data.

It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.

Nothing like bruteforcing intelligence by simply building a large enough lookup table to contain responses to all potential questions.

I think TinyStories is a promising direction. I just wish we had an alternative to GPT-4 because supposedly you can't use it to train other models.

Or maybe clarification of people publicly saying they are going to ignore that restriction.

This is actually possible - if you have a biased dataset, then more data is bad.

More data will fix variance problems, but not bias.

This is also why it's a bad idea to read propaganda outlets to "stay more informed".

? adding on ChatGPT data into existing data is not lessening the amount of data...

> avoid thinking and effort and instead rely on an external brain

For me personally this was probably the biggest game changer because I'm now able to offload a lot of thinking to GPT-based tools and use my brain cycles for the less automatable activities.

Now instead of searching for something on Google and going through multiple pages before finding an answer, I can ask a precise question and get the answer right away. Especially when I know that the answer _is_ there somewhere, and all I need is to find it. If I'm not happy with the answer, I can continue the conversation until I get what I'm looking for.

I made Bing Chat, ChatGPT, and Warp AI (a feature of the Warp terminal that allows you to access GPT from within the terminal) a part of my daily life and I feel like I'm achieving much more with the time I have.

Have you been to a bar lately and overheard people talking about politics ? They 100% are probability machine, of lower quality than ChatGPT

Indeed. But I don't expect any of those bar patrons to help me get my work done... and I would be very skeptical of their advice.

What's even the point of your message ? ChatGPT is a tool, you can use it or not nobody cares, but a lot of people think if you use it the right way it IS helpful

I'm guessing that's how people figured out it was getting "nerfed". For Copilot, I kind of developed a relationship with the "AI". I kind of expect what it is going to generate, and I use it as a shortcut to (find function names, write variable names, complete unit tests, etc...). On its own, it's really bad to come up with the whole picture but as something that auto-completes you on steroid, it's really perfect.

When it got nerfed, its perspective got completely out of whack and it's making very different and inconsistent generation.

I have very similar observation, and while it is amazing at times at helping me with complex tasks, it just isn't like any human. If we rank the tasks by difficulty for human and AI, it just has very small correlation. Also GPT 4 is idiot in conversation, compared to its problem solving skill. I would have expected completely opposite trajectory for AI. eg it's hard for it to get it to ask good clarifying questions. There are lot of cheap tricks in prompting like asking it to act as expert, or prompt for chain of though that works.

I still think LLMs are a game changer. They are seemingly amazing at deducing meaning and intent from natural language. That's enough to be a game changer. Is it a game changer like the internet is/will-be? No. Is it on the level of the television or the telephone? Probably not. But you can still leverage it in many ways to add value or reduce human work.

If we're talking about LLMs as general purpose AI or a tool to replace programmers... mostly a fail so far. LLMs seem like they could eventually be a component of something bigger, but I don't see a line between what they are and general purpose AI.

> Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines?

I'm genuinely unsure if my own brain is any different.

This probability thing might be a red herring. I reckon it is just that softmax/cross entropy loss is a handy tool to get words out of the NN (and now transformers)

This is a great set of observations.

The only thing missing is an analysis of how much power it takes to accomplish each of these tasks. If ChatGPT-4 is at about 1:1 in terms of “effectiveness”, all that remains is to divide by the amount of power required to reach the answer using ChatGPT-4 vs by conventional means. If requires significantly more energy, then it’s a waste, and because of climate change we should really not pursue it further, IMO.

This is a good point. There's a hidden cost (energy consumption) which we will eventually pay for.

However, even in the 1:1 case it means I am training myself to become a "prompt engineer" rather than to be an actual thinker and problem solver. As long as there will always be another system for me to depend on, maybe that's ok. But as with people who never learned to read maps and navigate without GPS tend to be very confused and lost when their phone dies, I would like to be able to be a useful human even when the power is out.

I totally agree, although I think it's even worse than that.

In my opinion, a precondition for creativity and inventiveness is understanding. If you rely on a surrogate to give you answers, you will never reach the level of proficiency required to come up with something new. If we train a generation of thinkers to rely on an external brain to get anything done, they will only understand things superficially, and our ability to innovate at the society level will suffer.

Very well said, this has been precisely my experience too

Only you, a human developer can be truly creative. An LLM can only ever reproduce what it has seen before.

    An LLM can only ever reproduce what it has seen before.

I don't believe it. How do you explain Midjourney? The art that it produces is incredible by any measurement.

Where, exactly, did the LLM see this epilogue to The Great Gatsby before?

https://twitter.com/tsimonite/status/1653065940463157248

I responded 'way to not getsby the point'.

How is that in ANY WAY an epilogue to The Great Gatsby? This is exactly the problem. That original story builds with a series of revelations into the conclusion 'so we beat on, boats against the current, borne back ceaselessly into our past': establishing a PURPOSE, perhaps a bleak and unwelcome one. Fitzgerald's revealing an insight into the delusions of humanity. He's picturing even the greatest of us as surfers on the river of nihilism and reality. Our aspirations sparkle prettily… and are gone, like froth in the rapids.

And this is just a little bit beautiful. We can imagine beyond our reach. That's a human thing. The fact that our minds can cling so longingly to something that is simply not real, is kind of wonderful. Everyone's their own little world of unreality, and we aspire so earnestly (much like these AI folks do).

And then the AI, having drunk up all of Fitzgerald and everybody else, 'continues' past the point. With what? Hardly matters. It has no point to make. I'd be impressed if it refused, said 'nope, that was where it ended. Can't add anything worth adding, try reading it again'. But no, because the LLM has no intention and doesn't successfully get one from what it's 'read'.

It's constructed an epilogue out of nebulous religious feelgoodism in the rough style of Fitzgerald's sentence construction, and undermined the whole conclusion of the story… and not maliciously, for that would require intent. Nope… it sort of ambled on, going 'what would feel nice here? ok, now what seems like it would go with this sort of thing? ok, something else, let's have more, what kind of concepts go here? what do people normally say when they talk in this way?'

In so doing, it's less than Gatsby and way less than Fitzgerald. There is nothing here in this 'epilogue'.

GPT4 criticised the 'epilogue', in a rather fascinating way! https://pastebin.com/B0zxbvNv

It successfully works out some of the problems with the first AI's writing, and yet it too fails to get the idea expressed by Fitzgerald, and rather than pivot to religious feelgoodism, it pressures the first AI to instead emphasize how its narrator and Gatsby shared a special bond, the very special specialness of being "the only one who understood what it meant to be young and restless in this restless world."

A lot of people can write that idea, and in fact a lot of people did and that's why GPT4 found it a probable argument to make.

Fitzgerald gave us a moment of viscerally grokking that it doesn't mean s*t… and yet, we will still paddle against the current of time and decay and collapse, because what else can we do? Tomorrow we'll get it. Tomorrow we'll really understand and it'll all make sense.

And so…

In so doing, it's less than Gatsby and way less than Fitzgerald. There is nothing here in this 'epilogue'.

"This Commodore 64 is useless. It can't even run Crysis."

Snark aside, I didn't ask if the epilogue was any good or not, I asked where it came from. It came from our collective consciousness as embedded in the language model. It turns out that what we've been dismissing as mere "language" is an insanely powerful thing, maybe the only thing.

We're still in the first publicly-visible generation of LLM technology, and the model that generated the epilogue was already behind the leading edge in many respects. Anyone who's not blown away by this is whistling past the graveyard. Computers are now doing what we do. Yes, they still kinda suck at it. But they will get better at it much faster than we will.

I mean, really. What is a human author thinking, if not, "What would feel nice here? OK, now what seems like it would go with this sort of thing? OK, something else, let's have more, what kind of concepts go here? What do people normally say when they talk in this way?" It's been understood since the Greek classical period that there is only a finite amount of ore in the original-story mine. Everything after the first seven or so basic ideas is just implementation.

I have to wonder what you'd think of Anthony Burgess's epilogue to A Clockwork Orange, the one that the book's American publisher and Kubrick both chose to leave on the proverbial cutting-room floor. The one where Alex grew out of his rebel phase, got a job, and started a family. This epilogue reminds me of that, somehow. I could easily see Burgess's final chapter emerging fully-formed from an LLM in the not-too-distant future.

> An LLM can only ever reproduce what it has seen before.

Anyone who's played around with these models know that at least some generalization is taking place.

So generalization counts as creativity in your book?

Inductive reasoning, or the process of inferring from the specific to the general, is kind of a big deal. You can't do that without a semantic model. You can't do it well without a good semantic model... even if it's not something that you or I would recognize as any kind of semantic model at all.

Yes, generalization is creativity. I don't think there's a difference between the two concepts.

Humans don't come up with ideas out of nowhere. Open a novel and you will find that even though the overall work is unique, it is composed of tropes from other literature and experiences from the author's life which have been generalized into another context.

Reply on news.ycombinator.com