Hacker News Reader

Comment by SrslyJosh | Hacker News Reader

Reminder that these models have no concept of truth, no abstract reasoning abilities, and there's no guarantee that the plausible-looking text they spit out isn't complete bullshit.

Their output is impressive considering that they're incapable of reasoning, but it desperately needs to be sanity-checked by someone with relevant knowledge before using it for anything serious.

View on HN

I wonder how many people who think they have no reasoning abilities have actually seen GPT-4. There's clearly some form of reasoning with GPT-4.

There are situations where a human can reason better. But these are usually identified as bugs and will probably just be fixed later.

Yes, they're basically autocomplete machines but a human is a bundle of neurons optimized for survival anyway. But humans can become irrational to things that threaten survival. Classic medical example is the drama that emerged when one doctor suggested that washing hands would reduce mortality rates.

Humans and LLM-based AI evolve rationality based around different factors and form different "bugs" and I love that they're capable of fact checking each other now.

https://ekjsgrjelrbno.substack.com/p/language-models

Just because it produces sequences of tokens that we typically experience as evidence of reasoning does not mean that it reasoned to produce them. We know -- generally -- the process by which those tokens were produced. It is not deterministic, but its properties are understood. It's a language model! It is not a mind.

I would be very interested to hear how exactly a mind produces its reasoning.

Real time post shifting and mental gymnastics is truly a sight to behold. No the output isn't perfect and should be double checked (as with any professional domain - you think what an Engineer with decades of experience signs off hasn't been checked multiple times by multiple parties ?, you think a professional programmer writes code and..just deploys it ?) but LLMs reason just fine. If you have anything to say about that other than the nonsensical - "it's not true understanding just because", i'd love to hear it.

This line of reasoning sort of goes both ways. Advocates and claimants of the technology arguing that humans reason no differently and that LLMs just given the right data will be better than everyone at everything related is just as hokey.

I didn't say anything about reasoning no differently.

Suppose you have 2 equations. You don't know what these equations are. However, you know that for any input, the output is the same.

Any mathematician worth his salt will tell you that given said information, those 2 equations are equal or equivalent. It doesn't actually matter what those equations are.

Equation 1 could be say a+b and Equation 2 could be (a-5) + (b-5) + 10

Or more realistically, both equations could look vastly different.

The point I'm driving home here is that true distinction reveals itself in results.

The fallacy of the philosophical zombie is that there is this supposed important distinction between "true understanding" and "fake/mimicry/whatever understanding" and yet you can't actually test for it. You can't show this supposed huge difference. A distinction that can't be tested for is not a distinction.

This analogy does not make sense to me. We do not have equivalence between all of the infinite inputs and outputs here, we have equivalence in a finite number of cases and known cases where the two functions (human output and llm output) diverge drastically. Any mathematician worth their salt would tell you these functions are definitely not equal.

Now you could make the argument that these functions are close enough most of the time so it won't matter but unless you want to get really rigorous that's more of a stats / engineering perceptive not mathematics. Any more importantly that's very up for debate, especially in a high pressure situation like medicine.

Of course these models are wild and I'm quite impressed with them. I still can be worried about the damage someone who doesn't think things through could cause by assuming GPT-4 has human or super human level intelligence in a niche, high impact field.

The equation example was simply an illustration.

The point I'm making is that I can quite clearly show output that demonstrates reasoning and understanding by any actual metric. That's not the problem. The problem is that when I do, the argument Quickly shifts to "it's not real understanding!". That is what is nonsensical here. It's actually nonsensical whatever domain you want to think about it in.

Either your dog fetches what you throw at it or it doesn't. The idea of "pretend fetching" as any meaningful distinction is incredibly silly and makes no sense.

If you want to tell me there's a special distinction cool but when you can't show me what that distinction is, how to test for it, the qualitative or quantitative differences then I'm going to throw your argument away because it's not a valid one. It's just an arbitrary line drawn on sand.

> but LLMs reason just fine.

There is a lot of research that shows they have trouble reasoning. Folks working with LLMs and building them agree that they can't reason, yet pop-sci folks religiously insist they can reason.

This is not even a controversial topic.

https://arxiv.org/abs/2205.11502

Reminds me of this comic:

https://www.smbc-comics.com/comic/2010-01-31

I'm not the person you responded to, but that's an old paper. LLMs have come a long way since. Nothing conclusive but GPT4 does show some signs of something deeper. https://arxiv.org/pdf/2303.12712.pdf

Yes, I have read that paper and have forwarded it eagerly to colleagues. Respectfully, please read the paper you have linked, especially section 8.2.

Look at section 8.2 in the paper you linked:

"8.2 Lack of planning in arithmetic/reasoning problems"

> However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution

Nowhere in the paper is an evaluation on standard reasoning datasets (https://tptp.org) because they are much much much tougher than the simple examples that GPT-4 struggles with.

LLMs are a huge breakthrough but let us not muddy the science.

GPT 4 was evaluated on multiple reasoning benchmarks in the release paper.[1] The problems in TPTP are not really appropriate for a language model and exceed the capabilities of most humans working without tools. Clearly humans have some reasoning ability even if they cannot solve those problems.

[1] https://openai.com/research/gpt-4

Oh boy, if you want to fling papers/research, i can do that just fine.

https://arxiv.org/abs/2212.09196v1

https://arxiv.org/abs/2211.09066

https://thegradient.pub/othello/

Moreover, "have trouble reasoning" and "can't reason at all" are two very different statements. Not that i particularly agree with either but you should keep that in mind.

> Reminder that these models have no concept of truth, no abstract reasoning abilities

This has positive potential too - bias prevention. Give it an ultrasound and it'll do its work regardless of the patient.

None of this is guaranteed (there could be training pitfalls, i.e. maybe patient data is fed to AI, including race/sex, and if training data has bias toward non-white males then so will AI), but it's a potentially positive aspect to explore!

But bias has a negative impact too.

It’s not unusual for clinicians to interpret differently based on patient history and other risk factors.

The same CT scan is going to be interpreted very differently for a patient with no history of cancer versus one who went into remission 1 year ago.

That's not bias, it's use of relevant context. If you're going to have an LLM doing diagnoses, you're definitely going to feed it that sort of context.

It's a matter of degree though.

Do you just tell the AI "this person had cancer?", or "this person went into remission 1 year ago?" or "this person was treated with 4 lines of chemo, partial remission for 1 line, non-response for 2nd line, strong response for cycles 1-3 for 3rd line, but then discontinuation due to tolerability, then 4th line complete remission after 6 cycles plus two courses of radiotherapy"?

Theoretically you could do this, but capturing all this and correctly interpreting it is far harder for AI than for a human who has been directing treatment the entire time.

Meanwhile for late childhood to to mid adulthood, 99% of my visits have been a doctor listening to me for 2 minutes tops while applying what amounts to a flow chart of how to diagnose and treat something like strep. Yes a doctor is best, but remember in many places we're competing with "no access do doctor so just pray it doesn't go to Rheumatic fever."

Very often asking the _right_ questions is hardest part especially in a non-typical cases. Statistical ML models tend to do well on high-probability regions of data that are densely sampled in training and are less good dealing with outlier cases. I am curious to see how GPT-4 deals with hard atypical cases.

Did you mean to write bias can have a positive impact, too?

Any model or human will suffer from some degree of uncertainty. You can never say anything with 100% certainty.

If you know a patient had cancer before, the chance that they have cancer again is much higher usually than the general population. This is very valuable for diagnosis.

It's good to have a prior/bias to check more closely here and err on the side of caution.

It's good to check for the more likely explanations first when you have multiple competing hypotheses and limited capacity for testing.

> if training data has bias

I think most, if not all, training data has bias. Removing bias from training data is a challenge in and of itself that I don't think we've solved yet. I worry that there isn't enough incentive there for us to solve it and some minorities will be left behind.

Except we have very powerful reasoning systems at the moment. Attaching a fully trained LLM on a speciality outputting in a structured form into a semantic reasoning system and feeding the results back through the LLM in a feedback cycle should control for hallucinations. It’s pretty clear though to all but the most cynical these LLM are producing amazing results regardless of whether “reasoning” is happening. The degree to which they produce insightful, cogent, and complex synthesis of concepts in a rational seeming form is startling.

I’d challenge anyone to demonstrate humans don’t “reason” in a similar way. Are we really anything more than expectation engines powered by gradient descent expectation juice? How come our reasoning fails so often and we have to be taught logic through reinforcement learning? Why do we fall for fallacies so easily when we “know” about them and can “reason” so well? I will wager $5 that as we build ensemble models that can leverage the reasoning AI tech of the last 80 years in a feedback cycle with LLMs and other generative AI models we will be staring at intelligence that far surpasses our own, and perhaps be so alien we can’t recognize it because it actually is backed by reasoning - unlike ours.

> there's no guarantee that the plausible-looking text they spit out isn't complete bullshit.

Sounds just like Doctors.

Ironically this “convincing bullshit” comment is turning into something everyone parrots in every LLM discussion thread.

I think that someone that’s gone as far to built this with GPT-n is aware of its limitations.

The question isn't really whether someone that has built it is aware of the limitation, but whether the users of that thing that someone built is aware of them.

In one study (see reference) users were able to identify the AI about 2/3rd of the times (and identical for the human experts). So 1/3rd of the time the AI is not detected and may well be spouting nonsense.

https://www.medrxiv.org/content/10.1101/2023.01.23.23284735v...

I think it's "top-down reasoning"

It does in fact create some sort of model out of those billions of dimensions, and it's able to re-use it across many different fields, too

It's not "bottom-up reasoning" so it misses a bunch of little details.

Basically AI is a search through a vast space, and most of the approaches have focused on top-down search. Reasoning bottom-up is what AlphaBeta search did (rybka and all those other chess programs, that calculated all possible combinations 10 moves ahead). Those missed some of the "top-down positional" stuff, whereas AlphaGo missed some weird-ass edge cases

https://www.engadget.com/alphago-pushed-human-go-players-to-...

But it is a sort of intelligence. Just not one you can trust when details matter. You can use it for pretty pictures for now, and essays.

Generative AI is not deep but extremely broad. It knows many subjects. And can discuss more than any specific person.

But guess who produced all that content for it and made it publicly available. Humans!

(And in cases where it wasn't publicly available, I guess they can sue the company)

I experimented with asking the models to rewrite text using odd and specific rules - “all sentences must start with the letter b and end with the letter d.”

It was pretty good, but sometimes it wouldn’t even try. It would spit out a sentence that could have been easily rewritten to start with b and end with d - but it just didn’t.

These sorts of hallucinations are a big problem is high stakes practice.

LLMs like GPT4 operate on tokens, not characters, so they're operating with a severe handicap when playing those sorts of games -- they see a word like "robot" as a single token, not as a collection of letters, so they don't know that it starts with the letter R unless that fact appeared in its training material.

Interestingly, they do better at rhyming games, because those are based on associations between tokens which are easier to infer from usage in poetry, or by reading rhyming dictionaries.

Why was it able to still do a good job? The hallucinations were about 10%, but the rest often improved the original sentence, despite the silly rules?

Also, wouldn’t learning from a dictionary make it pretty easy to know what word a letter starts with? Everything in chapter “B” starts with b.

GPT-4 can do some reasoning, just not at a smart/trained human level. You cannot explain its results in USMLE, Uniform Bar Exam, and several other tests otherwise. This is a major improvement from earlier models.

This assumes there is not too much data contamination, but other people’s experiences with it suggest that the capability is real even if it might be a bit below what the test performance suggests. The latter is also explainable by a broader distribution of real world problems relative to tests.

See my other recent comments for more examples and elaboration, eg https://news.ycombinator.com/item?id=35301499

At the speed things go, give it a few months.

Reply on news.ycombinator.com