Hacker News Reader

Comment by 0xKelsey | Hacker News Reader

> The scenario that I’m worried about, and that is playing out right now, is that they get good enough that we (or our leaders) become overconfident in their abilities and start integrating them into applications that they just aren’t ready for without a proper understanding of their limitations.

Very true.

View on HN

I would say this is true until we hit a "oh shit, they can really sue us, even if we warn them" moment. I'm imagining legislation will come into play that will make it less ideal for business critical solutions. I know Air Canada has been sued and probably others.

I still think we are in the honeymoon phase, and once that is over, LLM will become what it is meant to be, which is a power tool for domain experts.

This is going to happen like in many engineering industries when a cheaper, more likely to be faulty part has been used in replacement of a more expensive, sturdier part. And people will groan but be unable to stop it, unfortunately

It is happening in insurance right now and it is an unmitigated disaster that nobody wants to address.

There is real data used for insurance premiums and claims payouts but it's being swapped out for AI slop, and the sales folks are getting bonuses for selling hot garbage and the executives are getting bonuses for buying hot garbage.

Huh, if sales uses AI/LLMs and succeeds with it, as it seemed to me youve implied with their bonuses ... Isn't that actually positive for the company?

Or do you mean they succeed by promising lies via AI?

Not OP but I could see this being short term savings related to the cost of sourcing/generating risk data leading to bonuses prior to the deficiencies in that risk model being exposed in claims long term.

The industry is insurance. If you offload your due diligence to an AI that's wrong, the company will go underwater. What if the LLM tells you it's found a great market with little competition, high net worth individuals, etc. etc. and you don't check and end up making all your sales in the hills of California in wildfire country?

> succeeds with it

I'd worry about things which:

1. "Succeed" in the short-term, but sets the company up for long-term failure.

2. Outperforms the competition with a pattern of activity which is actually illegal.

Reply on news.ycombinator.com