Hacker News Reader

Comment by wredue | Hacker News Reader

Dude on HN was describing the other day how they’re moving to use an AI to generate a list of cars with specific features in their inventory and how this was a great part of the AI future.

But as we see frequently even on the most sophisticated AIs, they get shit wrong… a lot.

So this company decided to replace an actually working, guaranteed to produce proper results filtering system with a guaranteed to not produce proper results an unknown amount of the time, and the feeling was that this was good business direction.

People want buzzwords, not working software.

View on HN

People don't want buzzwords, people want promotions, bonuses, and recognition. It turns out that producing working software is not the optimal way to obtain those things at a lot of companies.

And how do you get said promotions, bonuses and recognition? By introducing change. And what change is easiest? One that relies on established ground of buzzwords.

I'm detecting a lack of synergy here.

The helicopter view says that I need to implement my change and move on before the shit hits the fan. The next person to take over will blame everything on "the previous guy", quite rightly, but I won't care as I'll have taken my promotion and be to busy blaming everything at the new company on the previous guy.

Make sure to leave 3 envelopes for the next guy.

https://kevinkruse.com/the-ceo-and-the-three-envelopes/

The system is neither dynamic nor robust enough...

We haven't leveraged those for at least ten years.

careful who you make fun of, technical debt is technically leverage

It's also the investing/partnering side of things too though.

Some VCs and the like aren't interested if your company isn't hitting those buzzwords.

Right, it's just a means to an end, one of several.

This is the real “safety” problem with AI. Since it’s so good at generating convincingly wrong answers, eventually we are going to build it into all sorts of mission critical systems and and everything from finance to healthcare will start to accumulate sophisticated, non deterministic failures over time. Like a trojan horse or a perhaps a civilization-level virus.

It's not that dissimilar to the sophisticated, non-deterministic failures already introduce over time by humans.

No, it is dissimilar.

Humans learn over time.

Humans have social pressures.

Humans have explanatory rather than merely predictive models about what they do.

Deviations by humans from expected standard behavior often produce better rather than worse results.

The list goes on.

The philosophical question is whether "social pressure" and "reward function" are the same thing.

For one person, at one point in time, sure, even though it'd be a quite complex one. But with an AI used by a thousand companies to replace one team member each, they're replacing a thousand of different "reward functions" with just one across all of them. A huge advantage of humanity is that we have different people, from different cultural and social backgrounds, all with social and personal norms and values that vary over time for each and every person, and telling an AI "imitate a pirate" or "answer without any biases" is not going to replace that.

It will be interesting to see. People are already involved in business-critical processes, and they have a far-from-100% success rate. You can see this with self-driving cars; 0.00001% of the time, they decide to swerve into oncoming traffic for no reason. But they can't get drunk! So whether or not that's a win is something that only time can tell. (People will be outraged, because no person would ever do what the AI did, but what matters are the numbers in aggregate. One could also argue; why should I have to put up with this when I never drink and drive? It's a good point and involves philosophical discussions about free will and all that.)

I guarantee people will be mad when an AI chatbot denies their insurance or customer service claim. But I have a friend who works in customer service and it feels like half her job is correcting the mistakes other customer service agents made. If the first line support was AI, would it change much, other than not paying people to make a bunch of mistakes?

Again, it's going to be painful while we try it out and get the data. It feels like it's going to be worse, and individual anecdotes will make each fuckup seem even worse than it is. But it seems inevitable that we're going to get the data. The only way to get VC writing software right now seems to be by tying your company to AI. That means a lot of experiments are about to leak out into the world. We'll learn a lot.

That's the great part about AI, because computers don't make mistakes, you can launder whatever biases you want through an objective and rational computer.

This is such an excellent way of concisely expressing a major fallacy I see whenever I talk about AI/modeling/etc. (I.e. “it isn’t racist it’s just numbers” and other nonsensical takes). I’m borrowing this language for the future.

Meanwhile, human judges, juries, parole boards, etc are racist as hell and have no update algorithm...

That's a hopefully fixable organization problem. I think the closest we should let AI get to judicial decisions is suggested ranges. There still needs to be a person a decision traces back to, who's not just passing through what the computer says.

Ha. Human problems never get fixed, just moved around. There's nothing new to help get the racism or of the justice system - a core piece of the problem is that there are only disincentives for mercy and no checks whatsoever on individual biases. We still have a racist system in the US sixty years after the civil rights movement, and I don't see anything changing that anytime soon.

Tech problems sometimes get fixed, and occasionally either eliminate a class of human problems or create new ones. It's the only thing that changes... Rejecting tech wholesale means embracing the status quo.

There should absolutely be a human where the buck stops. But... "fixing an organisation" involves calling people racist. Then screwing their careers. That should carry a heavy emotional burden. It should involve anger and pushback.

"Some weights in an algorithm are wrong" seems a much less fraught problem.

and we have had racism algorithm in legal services.

Look up the term of art "mathwashing".

We already have this today when anti-theft alarms go off at stores as you exit with your paid purchase. You're presumed guilty because someone didn't disable the tag since the robot can't be wrong.

> People will be outraged, because no person would ever do what the AI did, but _what matters are the numbers in aggregate._

That is an opinion, not objective fact.

If Grandma June accidentally hits the accelerator, jumps the curb and kills a kindergartener, we can throw her in prison. She is held accountable for her actions.

https://www.upi.com/Top_News/US/2011/01/25/Woman-83-gets-3-y...

If Grandma June is sitting in her "self-driving" car and it spontaneously accelerates, jumps the curb and kills a kindergartener, what can we do? Who is held accountable?

Something something "fix the problem, not the blame", but accountability is part of fixing a problem. Improving numbers in the aggregate is good, but it is not (and should not be) the sole matter in discussion.

https://en.wikipedia.org/wiki/The_Ones_Who_Walk_Away_from_Om...

That is a strange example, because usually we don’t put people in prison for accidents unless there is specific type of gross negligence (e.g., drunk driving) or malicious disregard for life and safety.

> Anderson decided to plead guilty to vehicular homicide because she did not want the family of Karla Campos to endure a trial.

Maybe the woman knew that it would be revealed that she was drunk or had been told that she didn’t have the capability any more to drive safely, but she did it anyway. Without knowing all the facts, pleading guilty for an accident seems like a bizarre choice.

This is really not the dilemma it is often portrayed as. You can extend this problem to any instance where a man-made inanimate object kills someone, which we have centuries of precedent for. If a bridge collapses and crushes a schoolbus full of children it's not like we throw our hands up in confusion - there is a very developed legal process that will determine the whos and whats of accountability and punishment.

It isn't a new problem, it's something that every court handles on a daily basis. Rebar or autoregression, the core legal elements are the same.

We can try to provide decent transportation infrastructure with feasible alternatives to driving so that Grandma June doesn't have to operate a 2 ton death machine to live her life, but maybe that's getting offtopic.

Cherrypicked raw death counts are a lobbyist measuring scheme, not social science.

For example, if self-driving cars end up being prone to hitting children darting out on to city streets in new and unpredictable ways, then you have created impediments to the growth and development of children that did not previously exist. Would this be balanced by enhanced safety in other situations? Maybe.

These are complex topics like whether and how long schools should have stayed closed during COVID. Balancing the death and disability effects largely falling on adults vs. the educational interests of children.

"what matters are the numbers in aggregate" is the opposite of cherrypicking. One of those numbers would be the level of danger around children.

The unitless number for driving into oncoming traffic wasn't supposed to be a real stat.

Humans are not expected to be infallible. Computers are.

This expectation causes a lot of other problems, and will cause problems here too.

That's true but humans are, or at least should be, accountable. When the AIs start making/contributing to impactful decision it will be hard to determine who to consider accountable for them (if something bad happens as a result).

> It's not that dissimilar to the sophisticated, non-deterministic failures already introduce over time by humans.

It's not dissimilar at all, just significantly faster and now in the hands of anyone with a laptop.

Waiting for the AI judges and the AI executioners. End-to-end fully automated pipeline.

Or, more mundanely, automated customer support systems that just cannot handle lots of edge cases. And they'll have shut off all the "escalate to a human" paths since most of the mainstream paths work ok-ish.

What many people don't seem to want to hear is that most "ML" problems are simple ML problems. LLMs are a neat advancement and a great tool in a variety of applications but for most applications they introduce a great deal of uncertainty and often at significant cost.

They're a good complement to great design and infrastructure but you're fooling yourself if you think they're a replacement.

My accountant forced AI into software. It literally removes the labels I put and puts bullshit labels instead.

I’m part of their largest customers and I have only 300 invoices a year. I’m still stunned how people can mess up like this.

I'm quite surprised how we still don't have fixed quality data formats that anyone can easily transform into every possible thing. Accountants should be analyzing records not maintaining them?

We did have XML but, y'know, Microsoft.

> People want buzzwords, not working software.

I'm pretty sure this doesn't generalize that well. I would say that people who don't use every day the product they are buying prefer buzzwords, but it's less true for people who actually use it.

For the purposes of the company selling the software, only the opinions of the people doing the buying would matter.

The working list probably requires paying someone to keep it up to date. So the company is saving money by not needing that person with something customers may not notice is a degradation, at least not enough to switch companies.

In the overwhelming majority of companies, I’d venture to say that “AI and ML powered” actually means “sort this list and take the first item.”

Reply on news.ycombinator.com