Hacker News Reader

Comment by loxias | Hacker News Reader

> Write a short note explaining your decision for every application

Is there any evidence or reason to suspect that this would result in the desired effect? (explanations that faithfully correspond to the specifics of the input data resulting in the generated output)

I suspect the above prompt would produce some explanations. I just don't see anything tethering the explanations to the inner workings of the LLM. It would make some very convincing text that would convince a human... that would only be connected to the decisions by coincidence. Just like when ChatGPT hallucinates facts, internet access, etc. They look extremely convincing, but are hallucinations.

In my unscientific experience, to the LLM, the "explanation" would be just more generation to fit a pattern.

View on HN

Not to refute what you said, but what you describe is quite similar to what we humans call rationalization, and it has been argued (e.g. by Robert Zajonc) that most of the time we make decisions intuitively and then seek a rationalization to explain them.

Also, good luck with human explanations in the presence of bias. No human is going to say that they refused a loan due to the race or sex of the applicant.

Well no smart humans, but it turns out there are plenty of dumb ones.

There's probably a lot of truth to this, but there's also a big difference. Humans are ultimately responsible and accountable for their actions. Should my actions turn out to carry a racist or sexist motive, even unbeknownst to me, i can be held to answer for that. That's a big difference.

The rationalization from a human is valuable because it's delivered by the accountable party. From a machine such rationalization is at best worthless, since you can't hold the machine accountable at all.

But what youre saying only layers over the fact that, to avoid getting caught and convicted, a sufficiently good explanation is all that's necessary.

Not quite. What I'm saying is that the explanation is valuable because it comes from the accountable party. That you have a good explanation does not absolve you of responsibility. The short and clear way to put it is this: You don't go to jail because YOU think you did something wrong, you go to jail because WE think you did something wrong.

If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process.

>If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process.

Well, a court/law just has to declare "AI" as allowed to be used in such decisions, and the whole recourse you describe vanishes though...

> my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that.

> How do they justify that AI system getting to make decisions that affect me and my life.

so you, a priori, make the assumption that your loan _should've_ been accepted?

If the decision wasn't an AI, but some actuarial that calculates and computes based on a set of criteria, and the result is a denial, you could still make the same argument of "why is _this_ the process, instead of something else (that makes my loan acceptable)?".

> I care what about how the accountable bankers justify putting that AI system into the process in the first place.

Computers are already deciding. As to why: it's because it's their money they're lending.

I work in a bank and you're laughably wrong. Computers are at best assisting in the lending process but more realistically they're completely ignored. There is NO AI (at least recognizably, you could argue that there's some sort of very simple expert model embedded in the 10 credit limits) involved in the process at all.

Out lending process is based on the judgment of a few specific individuals, with more involved clients requiring approval from more senior people. All steps of that process can be overturned by the overseeing person, and that person is accountable for their decision.

I didn't say there was AI.

Article 22 of the GDPR states “The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.”

Hidden in the definition of sufficient is "meeting the standards for absolution that an investigator would use". I honestly see no reason a system could not go so far as providing case law references or FICO + ECOA guidance as evidence for a compliant rejection, even if the underlying reason is some statistics based on sex.

And why would a perfectly reasonable bank tell its customers it's using AI? AI would provide the breadcrumbs and the loan officer would conduct a reasonable story using that - it's just parallel reconstruction at its finest. I imagine this is how credit scores work. A number comes out of the system and the officer has the messy job of explaining it.

I used to work in munitions export compliance and there intent really matters. It's the difference between a warning and going to federal prison. And intent is just a plausible story with evidence to back your decision, once you strip the emotion away.

More importantly, to the best of our knowledge, human thinking is not simply predicting the next word based on a past model.

but the fact that predicting the next words based on past models does such a good job of masquerading as human thinking indicates that much of human thinking isn't much more than that. It may even simplify the task of figuring human thinking by narrowing down the last details of of the smaller set of missing pieces.

an analagous result was obtained back when they mapped the small finite number of neurons in a snail brain, or the behavior of individuals in ant colonies. What looks like complex behavior turns out to be very simple under the hood.

for the vast ocean of the population who... not sure how to describe them... not good students when in school, would rather spend the bulk of their time with the TV blaring, eating cheetos and swiping on tik-tok, following the lives of celebrities and fighting about it, rather than do anything long term productive with their own lives... chat gpt may have already exceeded what they do with their cranial talents.

even a level up on the ladder, the types of office situations lampooned in The Office or Dilbert, are they doing much more as a percentage of time spent than chat GPT can do? "Mondays, amirite!?"

then the question becomes, are the intellectual elites among us doing that much more, or just doing much more of the same thing? I think a large portion of what we do is exactly what chap GPT does. The question is what is this other piece of our brains' that intervenes to say "hmm, need to think about this part a lot harder"

>such a good job of masquerading as human thinking indicates that much of human thinking isn't much more than that.

No, that doesn’t follow. It just means it roughly looks like human thinking.

Your comment is akin to saying a high resolution photo of a human has basically figured out a way to replicate humans. It looks like it in one aspect but it’s laughably wrong. Humans thought without language.

chatgpt is only laughably wrong to educated people. it's sadly better than most people can achieve on their own. Even for educated people, the laughable part is only a personal quirk that they focus on for psychological reasons, but taken seriously a paper written by chatgpt is a good place to start and would save an educated person a lot of time in just touching it up to make it presentable, or changing the query to pare down its search for a better starting point.

It's not doing nothing, it's doing a lot.

> taken seriously a paper written by chatgpt is a good place to start and would save an educated person a lot of time in just touching it up to make it presentable

If we ignore the minor requirement of the paper having any connection to reality, of course.

Judging from a lot of the commentary on this subject, it appears to me that a surprising number of people think that reality is little more than a quaint idea.

You’re still missing the forest for the trees. It is doing a lot and it’s better at producing text that is how a human would write about topics better than many humans.

That’s not at all related to being close to general human intelligence.

>but the fact that predicting the next words based on past models does such a good job of masquerading as human thinking indicates that much of human thinking isn't much more than that.

Going in a straight line does such a good job of predicting the next position of the car that it indicates driving isn't much more than going in a straight line.

Alonzo Church showed that lambda calculus can replace all of abstract numerical mathematics with manipulation of meaningless symbols laid out in straight lines, or curved around.

I would give this 10 points if I could! But that’s level 2 driving. It is not nothing.

I know that's meant to be snarky, but yeah. A good portion of driving is simply going where the car is already headed. A good portion of conversation is probably just stochastically stuffing filler words and noises into your ideas to aid in their reception.

Haven't you ever had a situation where you were speaking and you get distracted, but not interrupted, and your speech trails off or gets garbled after ten or so words? It feels sort of like you've got a few embeddings as a filter and you push words past them to speak, but if you lose focus on the filter the words get less meaningful.

I'm sure we're different than an LLM, but seeing how they generate words - not operate on meaning - rings true with how I feel when I don't apply continual feedback to my operating state.

I like the metaphore.

Politicians are exceptionally great at it, filling up conversations with nothing

Many would, I suspect, echo your thoughts. In terms of the people you're not sure how to describe (the 'vast ocean'), I'm just wondering how these ideas might find effect politically were they to be held by candidates. Clearly ascent to executive office would have to entail being economical with the truth (lying) to the electorate. Not that much of this does not happen already - some might think.

> but the fact that predicting the next words based on past models does such a good job of masquerading as human thinking indicates that much of human thinking isn't much more than that.

I'm not sure I agree with that logic. What it proves is that we as humans are bad at recognizing that text generation aren't thinking like we are... that doesn't necessarily mean thinking isn't much more than what it is doing though, it just means we are fooled. Given that nothing like this has existed before and our entire lives up until now have trained us to think something that looks like it is trying to communicate with us in this way is actually a human being I'd kind of expect us to be fooled.

Of course not, but it seems perfectly plausible that human brains are machines that take sensory input and memory and predict the next muscle stimulus.

Do we really know enough about human thinking to say that?

> human thinking is not simply predicting the next word based on a past model

Some evidence is emerging which indicates that the activations of a predictive system like GPT-2 can be mapped to human brain states (from fMRI) during language processing[1]. We seem to have at least _something_ in common with LLMs.

The same seems to be true for visual processing. Human brain states from fMRI can be mapped to latent space of systems like Stable Diffusion, effectively reading images from minds.[2]

[1] https://www.nature.com/articles/s41562-022-01516-2

[2] https://the-decoder.com/stable-diffusion-can-visualize-human...

One might argue that this comment is simply predicting the next word based on a past model.

We don't know that it's true that humans are "responsible" or "accountable" for their actions in any meaningful way. This smells like a "free will" argument.

IMHO, it's unlikely "free will" and "responsibility" are anything more than an illusion.

It really doesn’t matter for this discussion, because society and laws are structured as if people have free will. An AI has to live up to that as well.

Maybe I'm confused about the topic, then. What does "living up to that" mean here? Legal liability or something?

Yes, that was their point. Humans have accountability to one another - either legally or in less formal ways (such as being fired if you're making your co-workers uncomfortable).

Current machines simply don't have that kind of accountability. Even if we wanted to, we can't punish or ostracize ChatGPT when it lies to us, or makes us uncomfortable.

Their use & application can be made accountable.

The point of this thread is that the explanation a human gives carries some weight because the human know they may/will be held accountable for the truth of that explanation. ChatGPT's explanations carry no such weight, since ChatGPT itself has no concept of suffering consequences for its actions.

So, while both humans and ChatGPT can and do give bogus explanations for their actions, there are reasons to trust the humans' explanations more than ChatGPT's.

Whether or not we hold humans using ChatGPT accountable for their use of it is irrelevant to this thread.

Is the idea that we think the punishment factor makes the explanation somehow more reliable? Why would that be the case? It’s kind of a dark idea.

The only intent taking place here, is someone placing a process under automatic control. That's it. If I make a bot that posts random comments and it offends you, then I'm a bit responsible, but also you've gotten offended by a random comment.

Although humans are capable of rationalizing and they do it, they have other capabilities which is a key point that is missing in this argument.

Maybe no one will admit to refusing a loan based on applicant’s gender, but also no real world aircraft engineer will explain why they decided to design a plane’s wing in a certain shape, purely by rationalizing an intuition without backing it by math and physics. Also, there are a group of humans elsewhere that understand those math and using the “same” principles can follow the explanation and detect mistakes or baseless rationalized explanations.

Using airfoil design rationalization as an example may not be the best choice in this context since there is a fairly long history of incorrect rationalizations for wing lift.

"Incorrect Lift Theories": https://www.grc.nasa.gov/www/k-12/VirtualAero/BottleRocket/a... "No One Can Explain Why Planes Stay in the Air": https://www.scientificamerican.com/article/no-one-can-explai...

Ugh, not this clickbait again. The actual title of that article should be "explanation of flight defies a simple intuitive explanation that is both wholly correct, and understandable to the layman". Very different then "no one can explain it!"

I couldn't agree more! However, I think the content of the article is informative. If you strongly disagree I'm happy to remove the link to the article

No, I do think it's an informative, well written article. It's quotations come directly from some the most well-known, and respected names in the field of the aerospace. I've even referenced it before on this very site! I just bemoan it's terrible title, and am perhaps bitter from the times when relatives send me links with similar titles (usually of lesser quality) and I have to explain that yes, people in my field do know what their doing. The infamous "bees" quote being a particular bug-a-boo.

I would however critic it's use as an example to prove that we have a history of rationalizing explanations where none exist (and using that to draw a parallel with AI). While the title implies this conclusion, the article itself does not. We do indeed have a very good explanation of how aerodynamic lift works. That explanation just takes the form of a set of differential equations, and isn't something one can easily tell a group of 5th grader, without simplifying to the point of spreading errors.

I wouldn't use it as an example to prove that that we have a history of rationalizing explanations where none exist. Certainly, as you mentioned, the explanation takes the form of a set of differential equations. I would instead argue that there is a history of incorrect rationalizations from individuals who were akin to "real world aircraft engineers" in their time and that these rationalizations are still present in discourse around wing lift / design.

The fact that we have a history of rationalization was actually never questioned. Rather it was pointed out that it’s not the only thing that our minds do and that’s something you need to consider.

There are also humans who hallucinate. Studying this phenomenon is useful, yet, on its own, it’s says nothing about how human brain works in general.

I took issue with the example of aircraft airfoil design, probably too pedantically, due to the challenges technical individuals have had related to rationalizing the behavior, despite a mathematical explanation.

But I completely agree with your point that rationalization alone isn't sufficient. We struggle to describe the universe solely in words and rely on other tools to further describe phenomena.

How to provide AI models with these additional capacities isn't necessarily clear yet but there are some interesting ideas out there: https://writings.stephenwolfram.com/2023/01/wolframalpha-as-...

I assumed people get the point in my comment, which is the difference between rationalizing a gender based decision on loan and the process of thinking, designing and building something like an airplane, or a nuclear power plant or your phone, etc.

Edit: A sibling comment from SonicScrub, is more articulate wrt the example used.

Except for things we decide to be important enough that we should create and apply models.

Now, the decisions that went into these models might have been rationalised after the fact. Or biased. But these handmade models can been reviewed by others, the logic and decisions that went into them can be challenged, and rules can be changed or added based on experience.

Not so much with LLMs.

The difference is that I can ask it to explain it's decision as a racist and a anti-racist and it will happily hallucinate an answer for both.

Another term is “confabulation” — which has also been used to explain human linguistic reasoning

> Is there any evidence or reason to suspect that this would result in the desired effect?

Yes.

There's evidence that you can get these models to write chain-of-thought explanations that are consistent with the instructions in the given text.

For example, take a look at the ReAct paper: https://arxiv.org/abs/2210.03629

and some of the LangChain tutorials that use it:

https://langchain.readthedocs.io/en/latest/modules/agents/ge...

https://langchain.readthedocs.io/en/latest/modules/agents/im...

Thanks! I was genuinely asking, will need some time to read and digest. LangChain looks interesting.

Thank you. Also search the web for posts on "prompt engineering," to get a sense of how people are using LLMs today. It's pretty eye-opening.

But note that just because a model says it's using chain-of-thought or tools to come to a certain conclusion, doesn't necessarily mean that it is: https://vgel.me/posts/tools-not-needed/

Yes, I agree. But note that the same logic applies to human beings too. Just because people say they are using chain-of-thought or tools to come to a certain conclusion, doesn't necessarily mean they are. Philosophers have been grappling with this issue for many centuries :-)

A car can explode, and so can a piñata.

Therefore deciding whether they are interchangeable is a deep question that may take centuries to resolve.

I don't understand this analogy.

The analogy is meant to show that, while it’s possible to raise deep philosophical questions based on superficial or trivial observations, it can also also be quite silly to do that.

But in the case of chain-of-thought and that ReAct paper, the results did have a measured increase in accuracy.

Oh yeah, absolutely. Just not something I'd use for, e.g., mortgages where denying someone unfairly could lead to a lawsuit.

Yes. I have been using ChatGPT quite a bit with programming tasks. One of the things I've been trying to do is using chain-of-thought prompting to ask the model to review its own code, line by line, evaluating it for the presence of a certain type of bug or some other criterion.

This has been illuminating: as ChatGPT steps through the lines of code, its "analysis" discusses material that _is not present in the line of code_. It then reaches a "conclusion" that is either correct or incorrect, but having no real relationship to the actual code.

I try to ask chatGPT add comments to my code, everything is good until I found it modify code, too.

Yes, look up "chain of thought prompting". It's also been shown previously that asking a model to write down its chain of thought improves the accuracy of its answers.

See https://ai.googleblog.com/2022/05/language-models-perform-re...

That being said, I don't think ChatGPT is ready for high-risk applications like insurance/loan approvals yet. Maybe in a year or two. For now, treat ChatGPT like you would a mediocre intern.

A Large Language Model is a tool and people have to learn to use it, just like any other tool. I have used GPT as a machine learning model in a way similar to the grandparent comment, and I now have some understanding of when and why it hallucinates in certain situations [0]. I use that understanding to filter responses and tweak prompts. It took a lot of time and effort invested, but I was able to get a handle on it.

I know that a lot of people won't do this, and will just accept whatever an LLM says at face value. And again, that's true for any tool. But if you invest the time to understand the parameters and limitations of the model, it can be incredibly valuable.

[0] in my experience, GPT davinci is much more likely to hallucinate in (non-programming) situations that would be difficult for a human to explain. using the above example, it can easily handle a standard credit application. But it will be more likely to hallucinate reasoning for a rare case like someone with a very high income but a low credit score. YMMV, just sharing what I've seen

Sadly for me, that's the only thing I want it to do- ingest the entire internet's worth of data and make a well reasoned assessment about an edge case or subtle trends based on that vast trove.

Yes, this is the same for me. The things ChatGTP gets right are often the "common" situations. It's possible you aren't aware of these so surfacing these more "common" facts can be helpful. But ask it a question about an edge case and it will hallucinate a plausible answer, and even double down when you point out a mistake.

By that point it's faster to find an answer by using a few "traditional" search keywords and reading the content yourself.

We can assume it will improve though, it's just not there yet for "edge cases" in it's training data.

I'll admit most people have played more with these models than I have, but I was really struck by how it would helpfully explain the structure of a LOC call number to me in ways that were just... incoherent within the span of two sentences – claiming that the first two digits of a Cutter number were from the author's name, but also that all the digits together represented the count of works under that author initial. Either sentence might have been plausible in an explanation of call numbers within some system - but together they were self-evidently wrong – and yet the kind of thing that someone less familiar with the material explained might have been confused by, might have assumed was their own failure to understand.

Suffice to say: hoo boy I hope all y'all commenters aren't tacking these APIs directly onto systems where the results have real consequences.

The solution is to break the explanation/decision into two or more steps.

Have it summarize the pros and cons of the loan. Have it score the issues. Have it explain why those issues and scores should receive a loan or not. Have it decide if those reasons are legitimate and the reasoning sound. Have it pronounce the final answer.

Get the rationalization out first and then examine it. This way even if you're wrong you have multiple steps of fairly clear reasoning to adjust. If it's racially biased you can literally see it and can see where to explain not to use that reasoning.

I don't think the person you're responding to necessarily disagrees with anything you've written, which all seems quite right to me, too. But it's about the marketability of the app.

A vast amount of the world is built on “close enough” and this is no different

Reply on news.ycombinator.com