Hacker News Reader

Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suck at counting. On the other hand they're perfectly capable of writing code based on instructions, and writing a program that will count a specific letter occurrences in a string is trivial.

View on HN

I mean, given that o1-preview takes sometimes a minute to answer, I'd imagine that they could append the prompt "Write a program and run it as well" to double check itself. It seems like they just don't trust themselves enough to run code that they generate, even sandboxed.

o1 gets this correct, through pure reasoning without a program. OP was likely using GPT-4(o|o-mini)

The example for "strawnberrystrawberry" (so the word concatenated with itself) was counted by O1 to have 4 r's.

https://chatgpt.com/share/66eb38c0-22cc-8004-9d29-024de2e39d...

You're right, sorry, I mixed up the models while testing.

yeah because now that we've all been asking about it, that answer is in its training data. the trick with LLMs is always "is the answer in the training data".

I think it'd be just too expensive to incorporate code-writing in CoT. Maybe once they implement having a cluster of different model sizes in one answer it'll work out.

Reply on news.ycombinator.com