Hacker News Reader

Comment by kgeist | Hacker News Reader

Tried the 1B model with the "think step by step" prompt.

It gets "which is larger: 9.11 or 9.9?" right if it manages to mention that decimals need to be compared first in its step-by-step thinking. If it skips mentioning decimals, then it says 9.11 is larger.

It gets the strawberry question wrong even after enumerating all the letters correctly, probably because it can't properly count.

View on HN

My understanding is the way the tokenization works prevents the LMM from being able to count occurrences of words or individual characters.

Of course, in many contexts, it is correct to put 9.11 after 9.9--software versioning does it that way, for example.

That's why it's an interesting question and why it struggles so hard.

A good answer would explain that and state both results if the context is not hundred percent clear.

I'm not sure how useful that question is in exploring capabilities--"ask the user clarifying questions if the answer is ambiguous" is more of a rlhf or fine tune thing than a base model thing.

What is the "think step by step" prompt? An example would be great, Is this part of the system prompt?

It's appending "think step-by-step" to the end of the prompt to elicit a chain-of-thought response. See: https://arxiv.org/abs/2205.11916

Does anyone know of a CoT dataset somewhere for finetuning? I would think exposing it to that type of modality during a finetune/lora would help.

Reply on news.ycombinator.com