Comment by pu_pe | Hacker News Reader

Comment by pu_pe | Hacker News Reader

> So you fire your expensive, grumpy human team and request larger and larger additions from your AI: a new guestroom, built-ins, and a walk-in closet.

> You feel great, until…you realize that your new powder room doesn’t have running water; it was never connected to the water main.

> You ask the AI to fix it. In doing so, it breaks the plumbing in the kitchen. It can’t tell you why because it doesn’t know. Neural systems are inherently black boxes; they can’t recognize their own hallucinations and gaps in logic.

I've seen plenty of humans causing problems where they didn't expect, so it's not like using using humans instead of AI prevents the problem being described here. Besides, even when AI hallucinates, when you interact with it again it is able to recognize its error and try to fix the mistake it made, just like a human would.

The article correctly describes tech debt as a tug-of-war between what developers see as important in the long-term versus immediate priorities dictated by business needs and decisions. It's hard to justify spending 40 man-hours chasing a bug which your customers hardly even notice. However, this equation fundamentally changes when you are able to put a semi-autonomous agent on that task. It might not be the case just yet but in the long run, AI will enable you to lower your tech debt because it dramatically reduces the cost of addressing it.

There's something that feels tremendously off to me about these defenses which are to the effect of "What's so wrong with this? In principle, there's nothing saying a human couldn't also make one of these errors."

I think about the AI vending machine business simulation. Yes, LLMs were able to keep it running for a long time, but they were also prone to nosediving failure spirals due to simple errors a human would spot immediately.

People point to an infinity in the distance where these LLMs cross some incredibly high threshold of accuracy that lets us thoughtlessly replace human workers and thinkers with them... I have not seen sufficient evidence to believe this.

> There's something that feels tremendously off to me about these defenses which are to the effect of "What's so wrong with this? In principle, there's nothing saying a human couldn't also make one of these errors."

The big difference is that (most) humans will learn from this mistake. An LLM, however, is always a blank slate so will make the same mistakes over and over again.

And more generally, because of how these models work, it would be very strange if they didn't make the same mistakes as humans.

Many humans not only could, but do and will keep doing so. The catch is that we have systems in place that keep these impacts from being harmful. Largely by preventing people without the ability to self reflect and improve their actions from being put in roles where they can cause disruption or harm. This is why we have some many different credentialing systems, from school diplomas to professional licenses to even something like good reviews of a business, to say nothing of our own personal filtering when we interact with someone before we do business with them.

AIs are a horrible fit for the current system. They aren't really better or worse, but instead completely alien. From one angle they look like a good fit and end up passing initial checks, and even doing a somewhat decent job, but then a request comes in from another angle that results in them making mistakes that we would normally only associate with a complete beginner.

The question becomes, can we modify the existing systems, or design new ones, where the alien AIs do fit in. I think so. Not enough to completely replace all humans, but 8 people with AI systems being able to do the job 10 people use to do still means 2 people were replaced, even if no one person's entire job was taken over entirely by the AI.

What remains unknown is how far this can extend, between getting better AI and getting better systems where the AI can be slotted in. (And then to grandparent's point, what new roles in these systems become available for AI that weren't financially viable to turn into full jobs for humans.)

> Besides, even when AI hallucinates, when you interact with it again it is able to recognize its error and try to fix the mistake it made, just like a human would.

This feels beyond generous. I'm sure I'm not the only one who has led my AI assistant into a corner it couldn't get itself out of

> I'm sure I'm not the only one who has led my AI assistant into a corner it couldn't get itself out of

My team recently did a vibe coding exercise to get a sense of the boundaries of current LLMs. I kept running into a problem where the LLM would code a function that worked perfectly fine. I'd request tweak and it would sometimes create a new, independent function, and then continually update the new completely orphaned function. Naturally, the LLM was very confident that it was making the changes, but I was seeing no result. I was able to manually edit the needed files to get unstuck, but it kept happening.

Yeah, the following has happened to me multiple times when I ask copoilot to fix something. It updates my code to version A, which is still broken. Then I share the error messages asking for a fix, resulting in version B, which is also broken. Then I share the error messages and copilot confidently changes things back to version A, which is still broken.

It will confidently explain version A is broken because it isn't version B, and version B is broken because it isn't version A. There's no "awareness" that this is cycle is happening and it could go on indefinitely.

I use LLMs all the time and if they make a mistake it's basically over. I'll start a new session because in my experience there's no recovery.

> I use LLMs all the time and if they make a mistake it's basically over. I'll start a new session because in my experience there's no recovery.

Yup, because the error becomes part of the prompt. Even if you tell them to do something different it's basically impossible for the model to recover.

Humans would not forget to connect water to a room and if they did would know how to diagnose and fix it by reasoning about it and eliminating possible causes, something that lllm’s are not capable of.

Many many people report experiences that directly contradict what you say next. The type of large language model people currently label as AI does not learn and when asked to fix a problem it takes another guess, usually also wrong, sometimes creating other problems.

Yes. AI usage is a solution slot machine not a constantly improving process

The slot machine is an excellent metaphor. Feeding the LLM prompts over and over, seeing incorrect results most of the time but once in a while hitting the correct response jackpot. Feeding the exact prompt again and getting a different result.

https://softwarecrisis.dev/letters/llmentalist/

> even when AI hallucinates, when you interact with it again it is able to recognize its error

"recognize" is a strong claim

> and try to fix the mistake it made,

or double down on it

>just like a human would.

probably not, because the human is also responding to emotional dynamics (short and long term) which the AI only pretends to mimic.

Reply on news.ycombinator.com