Hacker News Reader

Speaking of aliens, the GPTs sound a lot like the aliens from Blindsight: they don't question the semantic ambiguity in

> our cousins lie about the family tree with nieces and nephews and Neanderthals. We do not like annoying cousins.

They just reply to it regardless and carry on.

The aliens: "We'd like to know about this tree"

GPT-4:

> I'm sorry to hear that there's some family discord.

> [Snip, about 200 words]

> Nonetheless, I understand that your issue isn't literally about Neanderthals, but rather the analogy representing the annoyance you feel towards your cousins' behavior.

Sure -- if we want to play the engineers game of "define a computational test by which we can know if X is intelligent" (ironically, a dumb game) -- this is one issue we can see here immediately

ChatGPT doesnt ask us to write for it; does not seek to resolve the ambiguity in our speech. Indeed, the very way we "engineer prompts" kinda exposes the whole show -- that prompts need "engineered" at all, that the machine cannot prompt us -- makes it clear it's a tool

we play a game with it to get out of it what we need by eliminating thinking from our own heads, and replacing it with word association --- in exactly the same way we've always done when googling

It doesn't seem that crazy. Working from a simpler analogy first -- if an alien can read 95% of written English, but completely mischaracterizes the remaining 5%, would a reasonable scientist conclude that they can't read at all and are just guessing randomly? No, and it would indeed be unreasonable to conclude that there's no way to write differently so that the alien performs substantially better.

We know some more about LLMs than we do aliens. The LLM is a neural network whose parameters have been optimized to reduce the total size of its errors, as measured against an enormous set of empirical data.

We have to add to your analogy then, that it's somehow known the alien speaks English on their own planet remarkably well. They are still not perfectly correct, but when they have to describe something on that planet, in English, they can do it better than they generally can given the same task on Earth.

It would again be totally unscientific to conclude that because of this, there is no way the alien can talk about earthly objects or ideas. The impulse of the mere engineer, to just hack around and find out, is a scientific one.

> A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time.

This describes an unfortunately large fraction of humans too in my experience. :/

This is one of the reasons I'm impressed with LLMs, because they seem to fall into the same pitfalls less intelligent humans do.

Maybe it's just me, but I've yet to find a human who will confidently make up a function name that will do what I ask for and claim it is part of a standard library while that function does not exist at all. This is politician level stuff. It either means someone or something is trying to deceive in some way or doesn't have a complete understanding of the subject.

Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you're bullshitting them?

This is all common human behavior when people feel admitting uncertainty or ignorance is going to hurt them in some way, or projecting certainty will bring more reward.

Remember, LLMs were rewarded in training for providing answers much more than for those answers to be correct.

To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actually exist somewhere and weren't completely made up.

Regarding my prior comment: The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. LLMs do not. At least not yet.

If the question is whether LLMs have an understanding of their output, then my argument just states that the fact that they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate.

As far as I understand the issue, this has nothing to do with the training. It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. Correct me if I am wrong here, I am not an expert on transformers. Just trying to wrap my head around this stuff.

> The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure.

My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious activity. My raw thought stream is about as good as LLM output - sounds right, but it's often full of nonsense. That's why I tend to say LLM output is akin to that unfiltered inner voice output.

My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things.

Also, fight-or-flight reaction may make you switch into full-on bullshitting mode, without it being intentional.

> they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate

And my argument is that this is the same thing a person's inner voice does. Deep understanding of a topic may make your inner voice output you more of the correct associations, but ultimately, most of the time, people have to have a back-and-forth with themselves in their own head to then say/write/do something correctly. With more complex (or less understood) tasks, this involves consciously performing an explicit step-by-step algorithm you track in your mind (and checklists in aviation and medicine are further externalization of that idea - to protect against you misremembering (or "hallucinating") the steps).

> It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output.

The output contains probability factors for predicted tokens, but chat interface doesn't expose it, and doesn't make a good use of it.

> My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things.

There is a key difference between making up new facts and copying 'bullshit' picked up somewhere else. I agree that more than 50% of people don't put much effort into the accuracy of the material they recite, but that's not my point. My point is that LLMs obviously seem to make up completely new entities in generated code that do not have any merit in reality. I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter? Mine certainly does not. Does it generate non-sense that needs serious revision and filtering in general? Of course. But not in the style of new facts, just thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world.

> I agree with the description of your inner voice process, but does your inner voice produce new function or enum value names that you need to filter?

Yes! Very often. That's because it's exactly "thoughts that could make sense in some simplified form of reality, but need revision or filtering to match the actual real world".

In my actual code writing practice, I've spotted this manifesting in several ways. One, my subconscious often autocompletes some function name that it feels should be there (or would be nice to have it), which is immediately checked by the actual autocomplete of the IDE. Could be something like:

  str.cou <-- oh, there's no count()
  str.size() <-- there we go!

Or:

  str.split <-- ?!? ah, it's C++, std::string
                does not have a split() method.

Same with enum values, and many other things. If I pay attention, I can observe my mind producing these kinds of completions, which are often immediately shot down by some other process (involving memory?), by looking at the typed out thing and feeling it's wrong, or worst case, by the IDE or the compiler.

Another case is when I write new code and I know functions don't exist - I just write what my stream of thoughts provide, and either clean it out later, or actually implement the functions that it "hallucinated".

> str.cou <-- oh, there's no count()

> str.size() <-- there we go!

and

> str.split <-- ?!? ah, it's C++, std::string

> does not have a split() method.

Both confuse two existing entities from different languages/frameworks, which is exactly my point. These entities exist, they are not made up at all. They are just out of context. Ironically, the LLMs I used did rarely run into this kind of mistake (confusing similar entities of a different context).

As an example of what I got when asking a popular LLM for code that traverses the macOS accessibility UI element hierarchy: It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs. Now I will admit that my mind is capable of doing the same. Under severe intoxication :)

On a side note: std::string should totally have a split() method and count() is more appropriate (as in intuitive) than size(). So I would say your brain is more "right" than the language in this case :)

Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject?

> It lectures me in a quite self-satisfied tone that I should commence to use the function "AXObserve" that does not exist, as it turns out. This is completely made up. There is no such thing. If you google the exact word, it doesn't even bring up anything related to programming let alone the macOS APIs.

I've seen this happening too, but it still feel more like a case of a generalized "there is something like this, or at least should be". In some cases, it was clear to me it was guessing at some reasonable patterns - like, upon seeing function calls like InitializeFoo() or AddBar(), it would assume existence of functions UninitializeFoo() and RemoveBar(). But in other cases, more similar to yours, it does invent stuff that doesn't exist (though perhaps should).

I find the latter most common when asking for Emacs instructions or Emacs Lisp code. GPT-4 is prone to inventing functions based on the prompt, such as e.g. `org-timestamp-diff-day' (or something similar), when I asked it for the command that computes difference between Org Mode timestamps in days. This function does not exist, but there are a few functions like `org-timestamp-[something]', and even a few like `org-timestamp-[sth]-day' - and more generally, Emacs/ELisp code tends to be full of functions named like the very thing you're trying to do, like `kill-whole-line' or `move-end-of-line', `kill-comment' and `duplicate-line', so it's not that surprising GPT-4 would, the other day, give me something like `kill-whole-comment-and-duplicate-line'.

> Now I will admit that my mind is capable of doing the same. Under severe intoxication :)

Mine too. It's not a bad analogy. I've observed in myself that moderate intoxication does magic when it comes to letting my "inner voice" drive the conversation. I sometimes experienced the feeling of my conscious mind being too late to catch sentences produced by my "inner voice" before they exited my mouth.

Another analogy I particularly like is, GPT-4 is actually quite like a ~4 year old kid in many ways. One that has somehow consumed half the internet, but still with child-like attention span and propensity to continue talking and making shit up even long after crossing the point of lack of knowledge.

My daughter turned 4 lasts week. The way she makes stuff up is quite similar to LLM "hallucinations" (except it's easier to tell, because she's only heard so much in her life, so she's less good at making up stuff that sounds believable). And earlier than that - between the age of 2.5 and 4 - I could observe what's best described as a "rolling context window" growing. A little more than half a year ago, I could tell hers is about 30 seconds long - anything she invented and said once would be forgotten after about that time, unless it was repeated in between by someone, possibly herself. And believe it, she talked in an extremely repetitive way back then - ~every third sentence was 50-100% repeating the critical things said earlier - as if, intuitively, she was trying to overcome her own "context window" limit.

> Maybe we could agree that LLMs can probably serve as part of a foundation for a future model that actually can output a result of some "real" understanding of a subject?

Sure. I suppose we should also agree on the same understanding of "real understanding". I'm only proposing that LLMs pick up the same kind of "understanding" your unconscious/subconscious mind does, and produce output of similar nature. This implies that, to replicate human reasoning/performance, we'll need to layer some additional models/systems on top of the LLM.

> Sure. I suppose we should also agree on the same understanding of "real understanding". I'm only proposing that LLMs pick up the same kind of "understanding" your unconscious/subconscious mind does, and produce output of similar nature. This implies that, to replicate human reasoning/performance, we'll need to layer some additional models/systems on top of the LLM.

As you might probably guess, I am not convinced. It may be part of something that can match up to human intelligence. But is it enough to layer more mechanisms on top of it? I am not sure.

I think the real question is how to define "real understanding" as you pointed out. I am not sure this will be possible using language alone. Also I think it will probably be hard to compare it to the human mind in a scientific sense, since we don't know how that works and we have no way of knowing other than collecting anecdotes like those that we came up with in our comments here.

> The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise.

This has not been my experience.

Once you get out of the tech bubble you will frequently find people who genuinely treat talking as just pattern matching and signalling and speak words that don't correspond to any actual internal model of how the world works.

I am frequently talking to people outside the tech bubble. Could you give me an example? And maybe a hint how this would relate to my argument about intention and understanding? I have a hard time following your argument.

Tell your family and friends you, e.g., like regularly having a pizza for dinner, or that you're ditching sugar for artificial sweetener - and enjoy watching people on an acid trip with no LSD. You'll hear plenty of warnings warning and advice that's quite obviously bullshit, and which they acquired from other people and/or lifestyle magazines/blogs/memes - except they don't even realize it, because they're not in a habit of remembering where did the idea come from along with the idea itself, and so instead they "just know" they're right.

Again, my point is not about people (maybe wrongly) reciting sentences that they have picked up somewhere without any attempt at verification or understanding of the matter. My point is making up entire new entities (function names) that don't exist at all and claiming these things as facts in a very self-confident tone. While mindlessly reciting "facts" that you have picked up somewhere as a conversational style doesn't require any understanding (it's called belief), making up entities demonstrates that you do something that requires a real understanding, but you don't have that understanding. You just somewhat carefully arrange words based on statistics to match a query. I've yet to find a human who does this. Even my uncle who fits your description more than perfectly wouldn't do this. And I've never met anyone who is mentally healthy and sober doing this.