Comment by duxup | Hacker News Reader

Comment by duxup | Hacker News Reader

Parent Full thread

I find LLMs WANT TO ANSWER TOO MUCH. If I give them too little data, they're not curious and they'll try to craft an answer when it's nearly impossible for them to be right.

I'll type and hit enter too early and I get an answer and think "This could never be right because I gave you broken sentences and too little." but there it goes answering away, dead wrong.

I would rather the LLM say "yo I don't know what you're talking I need more" but of course they're not really thinking so they don't do that / likely can't.

The LLM nature to run that word math and string SOMETHING together seems like an very serious footgun. Reminds me of the movie 2010 when they discuss how the HAL 9000 couldn't function correctly because it was told to lie despite its core programming to tell the truth. HAVING to answer seems like a serious impediment for AI. I see similar-ish things on google's gemini AI when I ask a question and it says the answer is "no" but then gives all the reasons the answer is clearly "yes".

The most annoying thing I've found is that they always assume I'm right. If I ask a question, they assume the answer is yes, and will bend over backwards in an obsequious manner to ensure that I'm correct.

"Why of course, sir, we should absolutely be trying to compile python to assembly in order to run our tests. Why didn't I think of that? I'll redesign our testing strategy immediately."

Ugh , I agree.

I would imagine this all comes from fine tuning, or RLHF, whatever is used.

I’d bet LLMs trained on the internet without the final “tweaking” steps would roast most of my questions … which is exactly what I want when I’m wrong without realizing it.

>> The most annoying thing I've found is that they always assume I'm right.

Not always. The other day I described the architecture of a file upload feature I have on my website. I then told Claude that I want to change it. The response stunned me: it said "actually, the current architecture is the most common method, and it has these strengths over the other [also well-known] method you're describing..."

The question I asked it wasn't "explain the pros and cons of each approach" or even "should I change it". I had more or less made my decision and was just providing Claude with context. I really didn't expect a "what you have is the better way" type of answer.

Similarly, with Claude in Cursor I've found that it will assume it's wrong when I even suggest that it might be: "Are you sure that's right? I've not seen that method before" will be followed by "I need to apologize, let me correct myself" and a wrong answer and this'll loop until eventually arriving at a worse version of what it suggested first even if I tell it "Nevermind, you were right in the first place. Let's go with that one".

Yeah I get this. Often I'll prompt it like "my intern looked at this and said maybe you should x. What do you think?"

Seems to help.

h/t to @knurlknurl on Reddit today shared these methods:

- “I need you to be my red team”(works really well with, Claude seems to understand the term)

“analyze the plan and highlight any weaknesses, counter arguments and blind spots critically review”

> you can't just say "disagree with me", you have to prompt it into adding a "counter check".

It’s funny AI will happily follow my lead and “bounce too close to a supernova” and I really have to push it to offer something new.

I’ve been prefacing every code related question with “Do not write code. Ask me clarifying questions and let’s talk this out first”. Seems to help especially with planning and organizing a design rather than monkeying with code fixing it later.

I incorporate this into system prompts at the start of conversations and still find I have to emphasize it again over course of convos.

Yeah, they forget as the chat context gets too large. A good example I’ve had is where I’ve been using chartkick to create a lot of charts, and suddenly they want to use another ruby gem. I have to remind them. We’re using chart kick.

Thank you.

It's possible to mitigate this with a conservative system prompt.

Do you have an example? I’m curious.

> I find LLMs WANT TO ANSWER TOO MUCH.

That's easy to fix. You need to add something like "give a succinct answer in one phrase" to your prompts.

I can’t tell if this was intentional, but it’s a hilarious joke. OP was referring to the decision to provide an “answer”, not the length of the response.

LLMs can't think. They're just fancy autocomplete with a lot of context.

This means you need to prompt them with a text that increases the probability of getting back what you want. Adding something about the length of the response will do that.

Reply on news.ycombinator.com