Hacker News Reader

Comment by ZaoLahma | Hacker News Reader

What irks me is how LLMs won't just say "no, it won't work" or "it's beyond my capabilities" and instead just give you "solutions" that are wrong.

Codeium for example will absolutely bend over backwards to provide you with solutions to requests that can't be satisfied, producing more and more garbage for every attempt. I don't think I've ever seen it just say no.

ChatGPT is marginally better and will sometimes tell you straight up that an algorithm can't be rewritten as you suggest, because of ... But sometimes it too will produce garbage in its attempts at doing something impossible that you ask it to do.

View on HN

Two notes: I've never had any say no for code related stuff, but I have it disagree that something exists all the time. In fact I just one deny a Subaru brat exists, twice.

Secondly, if an llm is giving you the runaround it does not have a solution for the prompt you asked and you need either another prompt or another model or another approach to using the model (for vendor lock in like openai)

>What irks me is how LLMs won't just say "no, it won't work" or "it's beyond my capabilities" and instead just give you "solutions" that are wrong.

This is one of the clearest ways to demonstrate that an LLM doesn't "know" anything, and isn't "intelligence." Until an LLM can determine whether its own output is based on something or completely made up, it's not intelligent. I find them downright infuriating to use because of this property.

I'm glad to see other people are waking up

That’s an easily solvable problem for programming. Today ChatGPT has an embedded Python runtime that it can use to verify its own code and I have seen times that it will try different techniques if the code doesn’t give the expected answer. The one time I can remember is with generating regex.

I don’t see any reason that an IDE especially with a statically typed language can’t have an AI integrated that at least will never hallucinate classes/functions that don’t exist.

Modern IDEs can already give you real time errors across large solutions for code that won’t compile.

Tools need to mature.

Yeah, but it would have to reason about the thing it just halucinated. Or it would have to be somehow hard prompted. There will be more tools and code around LLM, to make it behave like a human then people can imagine. They are trying to solve everything with LLMs. They have 0 agency.

Intelligence doesn't imply knowing when you're wrong though.

Hackernews has Intelligent people...

Q. E. D.

% LLMs can RAG incorrect PDF citations too

> ChatGPT is marginally better and will sometimes tell you straight up that an algorithm can't be rewritten as you suggest

Unfortunately this very often it gets wrong, especially if it involves some multistep process.

Reply on news.ycombinator.com