Hacker News Reader

Comment by htrp | Hacker News Reader

the fact that you have to encourage these models and tell them that they can solve these problems and warm up on easier problems seems to indicate that there's something to AI pairing above and beyond prompt

View on HN

They are relying on training data of humans talking about how hard these problems are. Same way an un-reminded Claude gives estimates for work that are as if a human is doing it by hand, but then will drop them by 20x if you remind them it’s going to do the work.

Yeah it s because they have a strong prior on unsolved problems being unsolvable.

Once the idea of AI routinely solving conjectures enters the training data this encouragement will disappear like 2023-era prompt engineering did.

I would think this type of behavior (ugh, or dare I say default mindset) by consumer-facing LLMs will always be desirable for ‘hard’ problems (things previously unsolved) because it’s a bit like having saftey mechanisms in place to prevent hallucinations for users incapable of verifying correctness of the output. You’ve got to do a little work to prove you understand that it’s hard but it’s still something the LLM might be able to accomplish.

I think this is also a healthy mindset for a person to have in many cases. If my boss asks me "go solve P = NP", as extreme example, I would also give some pushback.

It will likely burn a lot of tokens, take a long time, and might not work, so hopefully they'll still ask if you really want to spend the money on the attempt.

I spent about a month walking through proving something with Claude a few months ago and it _constantly_ told me that it was impossible and I should stop working on it, right up until it proved it.

One way they improved the hallucination problem was basically training the models to refuse to do or say something if they are not very sure they can do it. As a side effect, they refuse to work on problems they know are extremely hard.

Reply on news.ycombinator.com