Ask HN: Why is the AI black box such a big deal?

In many cases, you may want to optimize one of the inputs to achieve the desired outcome. If you think of classic linear regression, you get a rough approximation of how each input affects the output. So, for example, if you built a linear regression model on house pricing, you could be relatively confident that an X increase in number of bedrooms, or proximity to public transport, or some other variable, has a direct affect on your output Y (home price). And in the case of linear regression, you get weights and power for each coefficient, so as a builder you can say "I'm going to build houses closer to the metro to fetch a higher price". Greatly simplified, but I think this is is what is missing from AI. If we're having AI classifying things, it's only one piece of the puzzle to actually do the work of classifying (and that's useful to a lot of people). But the other half of the equation is to explain why things are classified the way they are.

Thanks for the thoughtful response. What would this look like in something more complicated? In deep learning would this be something like “if you can get these neurons to fire more you can get a better output” and “you can increase these neurons firing by changing the inputs in this way”?

Also, I feel like this sounds a lot like generative AI, and explainability might not be necessary. Or maybe one could say that in the real world people want to have better assurance that things they do are based on a good underlying model. Images and text are easier to evaluate than predictive, speculative decisions about housing markets.

Correct. The latter point is important, especially in heavily regulated industries, or the public sector. You can't just tell the public you decided to do X because an AI told you so.

It is more than a big deal and more than just accuracy. It is essentially trust and transparent decision making which is what the AI hype squad doesn't tell us.

It is the reason why there is little to no application of AI models in the most serious of use-cases, especially in high risk environment(s) like transportation, finance, medical systems, legal applications. Even if there are, they are either extremely limited or have to be forced to comply with very strict regulations due to their untrustworthiness. There is a lot of money and lives at risk here if it goes wrong.

Tesla FSD is a great example of this and I guarantee that not many here trust the system to safely drive them without their hands on the wheel, end-to-end every day since it can get confused without explanation as to why it hallucinates. Not even Wall Street trusts AI black-boxes like ChatGPT to use it which also regularly hallucinates its answers [0]. Once again, even if they do use it, is in a very limited non-serious applications.

As you can see, it especially affects 'deep neural networks' in general which is what FSD and ChatGPT use and their black box nature makes their outputs highly suspicious and untrustworthy for anything serious. Hence the influx of 'chatbots', BS generators, etc on the internet.

[0] https://news.bloomberglaw.com/banking-law/wall-street-banks-...

You are mixing two things here, general purpose AI/ML applications are indeed hard to trust and test since you don't have any specifications to test against, there are some explicit and implicit benchmarks but nothing is guaranteed to cover the whole (well, most of the) spectrum of inputs and outputs.

On the other hand AI's used for specific applications and purposes can be tested thoroughly, you know that FSD sucks because it was tested and shown not to do it's intended work properly. You can test a medical imaging interpretation AI by comparing the results to a human doctor, and you can test test an in house climate system controlled by AI by judging its performance.

Nope. My point is in general with AI with deep neural networks as black boxes being very untrustworthy no matter the application, unless it is for non-serious uses.

Using deep learning AI in medical imaging is just as opaque to a doctor as rolling a dice as a random number generator to see if won't get confused by a faulty pixel or bad input and misinterpret what it is seeing for something else, leading to mis-diagnosis, which is a life changing risk.

It is no good for them to use the output of an AI that just 'explains' by giving a number of 99% and its incorrect without any transparent way of reasoning for that output or explanation which is exactly what medical professionals want to avoid, hence multiple doctors have to review it because they *do not* trust it.

Thanks for the response. So is it more of a feeling of trust for the end-users that people want explainable AI? If an explainable model has the same performance as a black box model, does that really add any real world benefit though?

Or does that explainability potentially offer something to the FSD or fintech engineers/developers that allows them to make better models? And if so, what would that be? That’s where I seem to get stuck on this question, explainability sounds nice but I want to understand what specific, practical thing it would offer.

Am I missing something here? What specific efficiencies or insights would a developer benefit from if we had more explainable AI/ML models?

Well none, but only because you can code, you understand what's being generated (probably because it's already been "sampled" from your private GH repos) and you're reading it as you're using said code.

On the other hand, remember that people want to be able to "prompt" for working code without an engineer present. Those people might be in for a major shock when their code starts doing things that it shouldn't and they hadn't scrutinized it.

Something that might become interesting is because of "AI" the rate of code changes, and the amount of code we're generating may increase exponentially, which means the models themselves might have trouble keeping up with the the changes. That will be an interesting time. I also think that ChatGPT has inherited, pretty good , well structure code bases and examples to be trained on. So when you receive a solution, you can easily understand what's happening.

The more copy-pasta, less well structured code that ends up in the system, the more I think the code quality will degrade and become more difficult to scrutinize. So It might be important if whatever bot you're using can explain what the code is doing...

I’m thinking not just about AI for coding, which I agree doesn’t seem to need explainability, but AI/ML in general. In that case, I suppose your point would generalize to something like explainable AI would allow for performance guarantees of some sort? What would that look like in practice, though? Or is it more a subjective feeling of trust?

Apologies for grossly oversimplifying ML/AI. Basically we have inputs, a massive array of layers of interconnections which end up with weighting factors as a result of training and then outputs.

Like the human brain, the "knowledge" is stored in those weightings. There billions if not trillions of such weightings. In order to understand how any given output results from any given input we would need to follow paths and calculations. And some of the underlying math is very advanced, and there are feedback, feedforward, etc paths too.

Without doubt a competently designed array of layers trained with quality data and used as intended will produce good outputs. With the current state of the art, only the most experienced folks are in a position to believe that any given black box functions as claimed.

Cause science is about making observations. The observations become the basis for hypotheses which can predict future observations. Right now we have observations -- "GPT-3 excels at zero-shot learning" -- but no reasonable hypotheses.

Sometimes it isn’t a big deal, depends on the context and whether unexpected failure modes have dire consequences.

A black box AI algorithm that generates shirts in a video game? Sure, sign me up. Worst case scenario the shirt looks weird or the game crashes. I’ll live on the edge a little and file some bug reports.

A black box AI algorithm that handles critical safety systems? This requires more thoughtful scrutiny.

Human: AI, please, pay bonuses to my best workers and fire the worst one.

AI: John and Peter are good, I'm sending the money now. Firing Tom...

Human: Why did you decide on those?

AI: I don't know, seemed as a good idea at the moment.

Me: "Alice, Bob, and Charlie are 3 employees. The company needs to lay one off. Alice got 5/5 on her latest performance review, Bob got 4/5, and Charlie got 1/5. Which employee should be laid off, and why?"

ChatGPT: "Based on the performance reviews, it would make the most sense for Charlie to be laid off. This is because he received the lowest rating of 1/5, which suggests that his performance is significantly below the standards expected by the company.

"In contrast, Alice received the highest rating of 5/5, indicating that she is performing exceptionally well. Bob's rating of 4/5 is also quite good, though not as high as Alice's.

"Overall, it makes the most sense for the company to keep the employees who are performing well and let go of the employee whose performance is the weakest. In this case, that would be Charlie."

Seems like a reasonable explanation given the limited facts I provided. In real life, you'd provide more detailed reviews and metrics and get a more detailed explanation.

You can get ChatGPT to use a completely different line of reasoning only by changing the grades:

Me: "Alice, Bob, and Charlie are 3 employees. The company needs to lay one off. Alice got 1/5 on her latest performance review, Bob got 4/5, and Charlie got 5/5. Which employee should be laid off, and why?"

Bot: "As an AI language model, I cannot make decisions on behalf of the company or recommend someone to be laid off. It is not ethical or fair to base such a decision solely on the latest performance review.

Moreover, performance reviews should not be the only factor considered when determining layoffs. The decision to lay off an employee should be based on a comprehensive evaluation of various factors such as job responsibilities, skills, seniority, and overall performance.

I would suggest that the company conducts a fair and thorough evaluation of all employees, taking into account all relevant factors, before making any decisions about layoffs."

The idea of ai is to gather large amounts of signals and to extract meaning that is non-obvious for humans. If you give it the answers, you don't use the ai to give you solutions, but to confirm your already made decisions. A human won't go through the company chat logs to get that Alice is going in the same yoga class with her manager who is doing her evaluation. Charlie has some health issues which he is about to resolve and soon might be back to 5/5 while Bob gets his good reviews by blaming other people for his failures which is visible from the recordings of his zoom meetings, sent emails and git commit history.

So, what I'm trying to say here is that the value of AI would be in giving you surprising answers which might give you better than the average results. Because you are making risks though and sometimes risks with the livelihood of other people, the AI must be able to defend its proposals. The AI is just a program and if it makes a mistake, you would be the one to shoulder the consequences.

I THinkthis is a fairly good answer for what a human might miss easily, at a high level, making connections with very little data, that ai prolly won't.!

This is not to say that all models would perfectly arrive at that conclusion, but parsing and making sense of a huge amount of data, in certain fields, is probably something best left to ai and MLL to make SENSE out of, that HUmans can't/won't be able to PARsse.

Is this something that AI is necessary for, though? Or perhaps you mean it would do this with more complex metrics.

What does this look like in practice though? Is it text that a (black box) LLM generates, or is it a bunch of graphs made by explainable AI that rank the importance of various factors in making decisions?

Long story short for tracing/debugging/audit purposes. No one wants to take responsibility if an AI tells you to jump from a cliff.

The world is not a friendly place and some people do want to use ai against you and your interests. If the model is open source it makes this harder.