Hacker News Reader

Comment by NitpickLawyer | Hacker News Reader

Jeez. While I obviously can't talk at all about the math, I've noticed a few things:

a) The model thinks on some questions while straight answers on others. (I wish I'd knew from the questions if this is somehow correlated to hard tasks or "inventive" tasks, but that's way out of my league).

b) The model sometimes pushes back. Again, I'd wish I knew if it was warranted, but I counted 2 instances where it said "yes, but with caveats", one where it said "mostly yes but with this correction" and one where it said "careful here, because x y z".

c) The model did q&a + pdf ingestion + code writing + more q&a + thinking + more q&a, for a looong while, while seemingly staying on topic (at least Terrence Tao seems to think they're still productive, so I'll trust that).

This is what model progress is, not number goes up on xBency or yBencher. Damn.

View on HN

The "yes, with caveats" thing is boilerplate for both Codex and Claude since this current generation.

It's actually a bit annoying because it primes you to think that the caveats are real, but most of the time it's just something terribly obvious and not a real caveat, but the model probably has some system prompt that tells it to always consider caveats or something like that.

Same as the model starting every reply with a commitment to be "honest". LLMism are fun but I tend to just suppress them via AGENTS.md because they distract me

A lot of things are not black and white and it’s nuanced so someone with higher intelligence will point it out. It’s an emergent behavior not a defect

At least Opus up to 4.7 or so, my experience is that Claude often uses "yes, with caveats" in place of "no, you numbskull".

"Is a meter the same as a foot?"

"Yes, exactly, you have it now, except they're different distances."

Maybe it's because I ask it to quiz me, and it really doesn't like to tell me I'm wrong. I also got a fair amount of

Claude: Ok, I will test your understanding. Question A? Question B? Question C? Question D?

Me: A=10. B=2. C=121. D is not solvable.

Claude: You got most of them right! You're very astute in saying that A=10, but actually it's 7. B=2 is exactly right! C could be 121 if we were talking base 4, but we're actually in base 10 so it's 25. D is trivially solvable and is 0.

Me: ...isn't that like 1 out of 4? How is that "most of them right"?

Claude: You're absolutely right! ...blah blah blah

This has to be one of the more annoying characteristics of the recent models.

Every answer is met with "But did you think about X", or "One more thing to consider" and it almost starts to feel manic as you dig down that hole.

Multiple times I've had to literally say "Stop with the follow-ups and suggestions. Just answer the questions clearly".

Reply on news.ycombinator.com